2024-09-20T20:00:12.381 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T20:00:12.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T20:00:12.484 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912878 branch: wip-adk-testing-2024-09-20-0910 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} email: adking@redhat.com first_in_suite: false job_id: '7912878' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-09-20-0910 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 workunit: branch: wip-adk-testing-2024-09-20-0910 sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 8031 sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-adk-testing-2024-09-20-0910 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvYF2VfVw1Q9GlSmecms5y342x7pVQdTvzkgGwnm4sMk/fLH6tz4we9rq5MjpY1l+2ZyeGV+z4NTZhO19o6jGo= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGfkT3KIIm1kKf9uz3g30SG4FzHvE31OS/Xws32Hy9rtS3S8AezjehGVRd3k/5RWceIADP7NYp910X/demVKYS4= tasks: - cephadm: compiled_cephadm_branch: reef image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_18:25:04 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-20T20:00:12.485 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa; will attempt to use it 2024-09-20T20:00:12.485 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/qa/tasks 2024-09-20T20:00:12.486 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T20:00:12.487 INFO:teuthology.task.internal:Checking packages... 2024-09-20T20:00:12.509 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7' 2024-09-20T20:00:12.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T20:00:12.509 INFO:teuthology.packaging:ref: None 2024-09-20T20:00:12.509 INFO:teuthology.packaging:tag: None 2024-09-20T20:00:12.509 INFO:teuthology.packaging:branch: wip-adk-testing-2024-09-20-0910 2024-09-20T20:00:12.509 INFO:teuthology.packaging:sha1: e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:00:12.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-09-20-0910 2024-09-20T20:00:12.651 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5127.ge8baca1e 2024-09-20T20:00:12.653 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T20:00:12.659 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T20:00:12.659 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T20:00:12.666 INFO:teuthology.task.internal:Saving configuration 2024-09-20T20:00:12.678 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T20:00:12.684 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T20:00:12.710 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912878', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 19:55:03.836691', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvYF2VfVw1Q9GlSmecms5y342x7pVQdTvzkgGwnm4sMk/fLH6tz4we9rq5MjpY1l+2ZyeGV+z4NTZhO19o6jGo='} 2024-09-20T20:00:12.730 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-20_18:25:04-orch:cephadm-wip-adk-testing-2024-09-20-0910-distro-default-smithi/7912878', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 19:55:03.834723', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGfkT3KIIm1kKf9uz3g30SG4FzHvE31OS/Xws32Hy9rtS3S8AezjehGVRd3k/5RWceIADP7NYp910X/demVKYS4='} 2024-09-20T20:00:12.731 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T20:00:12.737 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-20T20:00:12.737 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-20T20:00:12.737 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T20:00:12.802 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68d94d0280>, signals=[15]) 2024-09-20T20:00:12.802 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T20:00:12.809 INFO:teuthology.task.internal:Opening connections... 2024-09-20T20:00:12.809 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-09-20T20:00:12.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:00:12.888 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-09-20T20:00:12.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:00:12.965 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T20:00:12.972 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-09-20T20:00:12.990 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-09-20T20:00:12.991 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-09-20T20:00:13.048 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T20:00:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T20:00:13.050 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-09-20T20:00:13.079 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-09-20T20:00:13.097 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-09-20T20:00:13.098 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-09-20T20:00:13.155 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T20:00:13.156 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T20:00:13.157 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T20:00:13.157 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-09-20T20:00:13.178 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T20:00:13.190 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T20:00:13.195 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T20:00:13.195 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-09-20T20:00:13.198 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-09-20T20:00:13.214 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T20:00:13.222 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T20:00:13.223 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-09-20T20:00:13.255 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-09-20T20:00:13.370 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T20:00:13.428 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T20:00:13.442 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T20:00:13.443 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T20:00:13.443 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T20:00:13.443 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T20:00:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:00:13.460 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-20T20:00:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:00:13.461 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-20T20:00:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-503.el9.x86_64 2024-09-20T20:00:13.518 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-503.el9.x86_64 2024-09-20T20:00:13.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-09-20T20:00:13.519 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-503.el9.x86_64 2024-09-20T20:00:13.519 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-503.el9.x86_64 2024-09-20T20:00:13.519 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-09-20T20:00:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-20T20:00:16.538 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.3 MB 00:01 2024-09-20T20:00:24.591 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-20T20:00:25.296 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-20T20:00:29.258 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.5 MB 00:00 2024-09-20T20:00:32.788 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 8.3 kB/s | 19 kB 00:02 2024-09-20T20:00:33.213 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.5 MB 00:04 2024-09-20T20:00:33.606 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-20T20:00:35.190 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 19 kB 00:00 2024-09-20T20:00:36.001 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-20T20:00:39.079 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-20T20:00:40.977 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T20:00:40.977 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T20:00:41.070 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-20T20:00:41.080 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-09-20T20:00:41.081 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:00:41.082 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 79 M 2024-09-20T20:00:41.082 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 126 M 2024-09-20T20:00:41.082 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-20T20:00:41.390 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-20T20:00:41.414 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 503 kB/s | 36 kB 00:00 2024-09-20T20:00:43.288 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T20:00:43.289 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T20:00:43.385 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T20:00:43.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 79 M 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 126 M 2024-09-20T20:00:43.396 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-09-20T20:00:43.533 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 402 kB/s | 36 kB 00:00 2024-09-20T20:00:43.917 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-20T20:00:44.375 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-20T20:00:44.742 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-20T20:00:44.743 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-09-20T20:00:44.743 INFO:teuthology.orchestra.run.smithi146.stdout:Total 59 MB/s | 79 MB 00:01 2024-09-20T20:00:45.322 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:03 2024-09-20T20:00:45.427 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-09-20T20:00:45.794 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-09-20T20:00:45.795 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-09-20T20:00:45.947 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 7.5 MB/s | 35 MB 00:04 2024-09-20T20:00:46.767 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-09-20T20:00:46.768 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-09-20T20:00:47.897 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.4 MB/s | 29 MB 00:06 2024-09-20T20:00:47.897 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-20T20:00:47.897 INFO:teuthology.orchestra.run.smithi078.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-20T20:00:48.578 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-20T20:00:48.921 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-20T20:00:48.921 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-20T20:00:49.210 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-09-20T20:00:49.879 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-20T20:00:49.879 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-20T20:00:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T20:00:49.967 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T20:00:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-20T20:00:52.749 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T20:00:52.887 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T20:00:53.018 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T20:00:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T20:00:55.854 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T20:00:55.994 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T20:00:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T20:00:57.820 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:00.884 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T20:01:01.086 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:05.961 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:41.320 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:41.321 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T20:01:41.321 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T20:01:41.321 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T20:01:42.176 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:01:42.177 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-09-20T20:01:42.516 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T20:01:42.596 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T20:01:42.597 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T20:01:42.597 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T20:01:42.597 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T20:01:42.597 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T20:01:42.597 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-20T20:01:42.597 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T20:01:42.597 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T20:01:42.656 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T20:01:42.657 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T20:01:42.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-09-20T20:01:42.688 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-09-20T20:01:42.688 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 82.el9 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8274670 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-09-20T20:01:42.689 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:01:42.690 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T20:01:42.691 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T20:01:42.691 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-508.el9.x86_64 2024-09-20T20:01:42.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T20:01:43.346 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-09-20T20:01:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T20:01:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T20:01:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T20:01:44.672 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T20:01:44.707 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-09-20T20:01:44.709 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-09-20T20:01:44.724 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.6g15ahlGAD 2024-09-20T20:01:44.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6g15ahlGAD 2024-09-20T20:01:44.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.6g15ahlGAD 2024-09-20T20:01:44.944 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.6g15ahlGAD is 6KB 2024-09-20T20:01:44.992 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.6g15ahlGAD 2024-09-20T20:01:45.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T20:01:45.070 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T20:01:45.070 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T20:01:45.070 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T20:01:45.070 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T20:01:45.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.235 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-09-20T20:01:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:01:45.338 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-20T20:01:45.702 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T20:01:45.782 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.782 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T20:01:45.782 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T20:01:45.782 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T20:01:45.782 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T20:01:45.783 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-20T20:01:45.783 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T20:01:45.783 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T20:01:45.840 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T20:01:45.841 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T20:01:45.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-09-20T20:01:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-09-20T20:01:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-09-20T20:01:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-09-20T20:01:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 82.el9 2024-09-20T20:01:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-09-20T20:01:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8274670 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-09-20T20:01:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:01:45.875 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T20:01:45.875 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T20:01:45.875 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T20:01:45.875 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-09-20T20:01:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:01:45.875 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T20:01:45.876 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T20:01:45.876 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-508.el9.x86_64 2024-09-20T20:01:45.876 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T20:01:46.533 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-09-20T20:01:47.852 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T20:01:47.884 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-09-20T20:01:47.896 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-20T20:01:47.912 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.4aaImChoby 2024-09-20T20:01:47.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4aaImChoby 2024-09-20T20:01:47.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.4aaImChoby 2024-09-20T20:01:48.130 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.4aaImChoby is 6KB 2024-09-20T20:01:48.180 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.4aaImChoby 2024-09-20T20:01:48.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T20:01:48.267 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T20:01:48.267 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T20:01:48.267 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T20:01:48.267 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T20:01:48.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T20:01:48.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-09-20T20:02:15.263 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-20T20:02:15.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:02:15.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:02:18.430 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-20T20:02:18.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-20T20:02:18.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:02:33.583 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-20T20:02:36.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-09-20T20:02:42.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:02:42.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:02:45.647 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-20T20:02:45.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-20T20:02:45.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:02:48.975 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-09-20T20:02:57.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:02:57.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:03:00.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-20T20:03:00.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:03:16.079 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-20T20:03:31.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:03:31.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:03:31.463 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-20T20:03:31.888 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:03:31.888 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T20:03:31.888 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-20T20:03:31.903 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-508.el9.x86_64 2024-09-20T20:03:31.904 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T20:03:31.904 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T20:03:31.904 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T20:03:32.744 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-20T20:03:32.905 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-20T20:03:32.905 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T20:03:32.985 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-09-20T20:03:33.018 DEBUG:teuthology.parallel:result is None 2024-09-20T20:03:33.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-20T20:03:33.204 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T20:03:33.204 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-20T20:03:33.220 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-508.el9.x86_64 2024-09-20T20:03:33.220 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T20:03:33.220 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T20:03:33.220 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T20:03:34.221 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-20T20:03:34.222 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T20:03:34.279 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-09-20T20:03:34.298 DEBUG:teuthology.parallel:result is None 2024-09-20T20:03:34.298 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T20:03:34.304 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T20:03:34.304 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T20:03:34.306 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T20:03:34.323 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T20:03:34.330 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T20:03:34.339 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T20:03:34.339 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T20:03:34.361 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T20:03:34.392 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T20:03:34.399 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T20:03:34.399 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T20:03:34.430 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T20:03:34.462 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T20:03:34.463 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T20:03:34.471 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T20:03:34.474 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T20:03:34.475 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T20:03:34.481 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T20:03:34.482 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T20:03:34.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T20:03:34.542 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T20:03:34.551 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T20:03:34.551 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T20:03:34.581 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T20:03:34.596 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T20:03:34.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T20:03:34.729 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T20:03:34.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T20:03:34.849 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:03:34.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T20:03:34.914 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T20:03:34.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T20:03:35.004 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T20:03:35.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T20:03:35.120 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:03:35.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T20:03:35.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-09-20T20:03:35.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-09-20T20:03:35.234 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T20:03:35.264 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T20:03:35.735 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T20:03:35.742 INFO:teuthology.task.internal:Starting timer... 2024-09-20T20:03:35.742 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T20:03:35.769 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T20:03:35.797 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-09-20T20:03:35.841 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-09-20T20:03:35.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-09-20T20:03:35.896 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-09-20T20:03:35.898 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T20:03:35.898 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-09-20T20:03:35.927 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-09-20T20:03:35.927 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-09-20T20:03:35.958 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-09-20T20:03:35.958 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-09-20T20:03:35.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T20:03:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:03:35.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T20:03:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T20:03:36.020 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T20:03:36.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-09-20T20:03:36.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-09-20T20:03:36.186 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T20:03:36.195 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T20:03:36.206 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T20:03:36.207 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T20:07:22.448 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-09-20T20:07:22.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-20T20:07:22.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:07:22.528 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-20T20:07:22.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-20T20:07:22.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:07:22.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T20:07:22.682 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-20T20:07:22.770 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-20T20:07:22.770 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T20:07:22.784 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T20:07:22.784 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T20:07:22.785 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T20:07:22.787 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T20:07:22.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T20:07:22.814 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T20:07:22.828 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T20:07:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T20:07:22.853 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T20:07:22.853 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-09-20T20:07:22.864 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-09-20T20:07:22.878 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-09-20T20:07:22.878 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T20:07:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-09-20T20:07:22.893 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T20:07:22.904 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T20:07:22.918 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T20:07:22.946 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-09-20T20:07:22.949 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T20:07:22.949 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-09-20T20:07:22.949 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.950 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.950 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.950 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.972 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-09-20T20:07:22.975 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T20:07:22.975 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-09-20T20:07:22.976 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.976 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.976 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.976 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T20:07:22.976 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T20:07:23.065 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7'} 2024-09-20T20:07:23.066 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T20:07:23.066 INFO:tasks.cephadm:Cluster fsid is f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:07:23.066 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T20:07:23.066 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.c': '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]', 'mon.b': '172.21.15.146'} 2024-09-20T20:07:23.066 INFO:tasks.cephadm:First mon is mon.a on smithi078 2024-09-20T20:07:23.066 INFO:tasks.cephadm:First mgr is y 2024-09-20T20:07:23.067 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T20:07:23.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2024-09-20T20:07:23.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-09-20T20:07:23.119 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-20T20:07:23.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:07:23.267 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8baca1eac6c3c9c88bf2ea4493e448bd54936a7', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83188/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5127-ge8baca1e', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5127.ge8baca1e'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-20-0910/e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 14:53:51.724057', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-09-20-0910', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-09-20-0910/e8baca1eac6c3c9c88bf2ea4493e448bd54936a7/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-20T20:07:23.512 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-20T20:07:23.514 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm 2024-09-20T20:07:23.514 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm 2024-09-20T20:07:23.514 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T20:07:24.054 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-r--r--. 1 ubuntu ubuntu 217226 Sep 20 20:07 /home/ubuntu/cephtest/cephadm 2024-09-20T20:07:24.054 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T20:07:24.608 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-r--r--. 1 ubuntu ubuntu 217226 Sep 20 20:07 /home/ubuntu/cephtest/cephadm 2024-09-20T20:07:24.608 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T20:07:24.627 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T20:07:24.658 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-20T20:07:24.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-20T20:07:24.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-20T20:07:26.222 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T20:07:26.368 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T20:08:07.111 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-20T20:08:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)", 2024-09-20T20:08:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd", 2024-09-20T20:08:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-09-20T20:08:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce" 2024-09-20T20:08:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-20T20:08:07.112 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)", 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd", 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce" 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-20T20:08:11.154 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:08:11.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2024-09-20T20:08:11.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-09-20T20:08:11.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2024-09-20T20:08:11.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-09-20T20:08:11.325 INFO:tasks.cephadm:Writing seed config... 2024-09-20T20:08:11.326 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T20:08:11.327 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T20:08:11.328 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:08:11.328 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T20:08:11.343 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f2df6e16-778b-11ef-baf5-efdc52797490 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T20:08:11.343 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service 2024-09-20T20:08:11.388 DEBUG:teuthology.orchestra.run.smithi078:mgr.y> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service 2024-09-20T20:08:11.432 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T20:08:11.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid f2df6e16-778b-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T20:08:11.585 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-20T20:08:11.585 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', 'f2df6e16-778b-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2024-09-20T20:08:11.608 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 5.2.2 2024-09-20T20:08:11.609 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-20T20:08:11.609 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2024-09-20T20:08:11.631 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 5.2.2 2024-09-20T20:08:11.631 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2024-09-20T20:08:11.631 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2024-09-20T20:08:11.638 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T20:08:11.639 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T20:08:11.644 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T20:08:11.644 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-20T20:08:11.651 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-09-20T20:08:11.657 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-09-20T20:08:11.657 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-09-20T20:08:11.657 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2024-09-20T20:08:11.680 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 5.2.2 2024-09-20T20:08:11.680 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-20T20:08:11.680 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2024-09-20T20:08:11.680 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2024-09-20T20:08:11.686 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T20:08:11.686 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T20:08:11.692 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T20:08:11.692 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-20T20:08:11.699 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-09-20T20:08:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-09-20T20:08:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 140569044766880 on /run/cephadm/f2df6e16-778b-11ef-baf5-efdc52797490.lock 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 140569044766880 acquired on /run/cephadm/f2df6e16-778b-11ef-baf5-efdc52797490.lock 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2024-09-20T20:08:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-09-20T20:08:11.709 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.78 metric 100 2024-09-20T20:08:11.710 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2024-09-20T20:08:11.712 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T20:08:11.712 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T20:08:11.712 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T20:08:11.714 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T20:08:11.714 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link noprefixroute 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-09-20T20:08:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-20T20:08:11.716 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T20:08:11.716 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T20:08:12.198 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd 2024-09-20T20:08:12.198 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-20T20:08:12.198 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2024-09-20T20:08:12.199 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:1f571018c56d8512357839d2ed445827ec316c3aed0fbfc68aff9046d4aceb99 2024-09-20T20:08:12.199 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-20T20:08:12.199 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd 2024-09-20T20:08:12.199 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-20T20:08:13.144 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable) 2024-09-20T20:08:13.144 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable) 2024-09-20T20:08:13.144 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2024-09-20T20:08:14.134 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2024-09-20T20:08:14.134 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2024-09-20T20:08:15.056 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQAu1u1mCNWrGhAA4FzkriCHA8Z3YA1fw8ME5g== 2024-09-20T20:08:15.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQAv1u1m34qKFxAAQpO8B1lSdBeiMlhtRViIgA== 2024-09-20T20:08:16.885 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQAw1u1mPOFiEhAAG+AjLuZ6zEDdmDGhPYiP7A== 2024-09-20T20:08:16.886 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2024-09-20T20:08:17.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T20:08:17.941 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = pacific 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:08:17.942 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2024-09-20T20:08:18.880 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2024-09-20T20:08:19.274 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T20:08:19.474 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f2df6e16-778b-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-f2df6e16-778b-11ef-baf5-efdc52797490.target. 2024-09-20T20:08:19.474 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f2df6e16-778b-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-f2df6e16-778b-11ef-baf5-efdc52797490.target. 2024-09-20T20:08:19.699 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a 2024-09-20T20:08:19.699 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-20T20:08:19.877 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f2df6e16-778b-11ef-baf5-efdc52797490.target.wants/ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-f2df6e16-778b-11ef-baf5-efdc52797490@.service. 2024-09-20T20:08:20.406 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T20:08:20.406 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-20T20:08:20.413 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T20:08:20.413 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-20T20:08:20.413 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-20T20:08:20.413 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T20:08:20.414 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2024-09-20T20:08:20.414 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2024-09-20T20:08:21.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:21 smithi078 ceph-mon[23317]: from='client.? 172.21.15.78:0/3917649875' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.545207s) 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-20T20:08:21.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2024-09-20T20:08:21.453 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2024-09-20T20:08:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:22 smithi078 ceph-mon[23317]: from='client.? 172.21.15.78:0/437189032' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T20:08:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:22 smithi078 ceph-mon[23317]: from='client.? 172.21.15.78:0/437189032' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-09-20T20:08:27.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T20:08:27.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:27.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T20:08:27.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T20:08:27.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T20:08:27.901 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2024-09-20T20:08:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:30 smithi078 ceph-mon[23317]: from='client.? 172.21.15.78:0/3383349148' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:08:32.465 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2024-09-20T20:08:32.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:32 smithi078 systemd[1]: Stopping Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[23313]: 2024-09-20T20:08:32.763+0000 7ffba729e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[23313]: 2024-09-20T20:08:32.763+0000 7ffba729e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T20:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:32 smithi078 podman[23560]: 2024-09-20 20:08:32.773908436 +0000 UTC m=+0.138058114 container died c5d32ec2184387ef8af02fc7da1b814595942ab9ccccb1375f4698f79f5f0e43 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-20T20:08:33.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 podman[23560]: 2024-09-20 20:08:33.138350326 +0000 UTC m=+0.502500006 container cleanup c5d32ec2184387ef8af02fc7da1b814595942ab9ccccb1375f4698f79f5f0e43 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T20:08:33.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 bash[23560]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a 2024-09-20T20:08:33.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 podman[23573]: 2024-09-20 20:08:33.397548403 +0000 UTC m=+0.621500221 container remove c5d32ec2184387ef8af02fc7da1b814595942ab9ccccb1375f4698f79f5f0e43 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T20:08:33.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T20:08:33.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 systemd[1]: Stopped Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:08:33.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 systemd[1]: Starting Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:08:34.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 podman[23666]: 2024-09-20 20:08:33.715045765 +0000 UTC m=+0.109341102 container create e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T20:08:34.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 podman[23666]: 2024-09-20 20:08:33.621385154 +0000 UTC m=+0.015680504 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph:reef 2024-09-20T20:08:34.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 podman[23666]: 2024-09-20 20:08:33.917153991 +0000 UTC m=+0.311449329 container init e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-09-20T20:08:34.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 podman[23666]: 2024-09-20 20:08:33.920177741 +0000 UTC m=+0.314473079 container start e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T20:08:34.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:08:34.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T20:08:34.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: pidfile_write: ignore empty --pid-file 2024-09-20T20:08:34.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: load: jerasure load: lrc 2024-09-20T20:08:34.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:08:34.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Git sha 0 2024-09-20T20:08:34.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T20:08:34.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: DB SUMMARY 2024-09-20T20:08:34.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: DB Session ID: XEAQL3WXSAHIK7AY1YPW 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 83640 ; 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:08:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:08:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:08:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:08:34.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:08:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:08:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.env: 0x563fb9fafbc0 2024-09-20T20:08:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:08:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.info_log: 0x563fbc6e0be0 2024-09-20T20:08:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:08:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.statistics: (nil) 2024-09-20T20:08:34.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.use_fsync: 0 2024-09-20T20:08:34.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:08:34.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:08:34.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:08:34.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:08:34.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:08:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:08:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:08:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:08:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:08:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:08:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.db_log_dir: 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.wal_dir: 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:08:34.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.write_buffer_manager: 0x563fbc6ee320 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:08:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:08:34.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:08:34.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:08:34.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:08:34.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:08:34.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.unordered_write: 0 2024-09-20T20:08:34.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.row_cache: None 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.wal_filter: None 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.wal_compression: 0 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:08:34.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:08:34.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_open_files: -1 2024-09-20T20:08:34.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Compression algorithms supported: 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kZSTD supported: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:08:34.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.merge_operator: 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_filter: None 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:08:34.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563fbc6e07e0) 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x563fbc6d5350 2024-09-20T20:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-09-20T20:08:34.029 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:08:34.030 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression: NoCompression 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.num_levels: 7 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:08:34.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:08:34.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:08:34.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:08:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:08:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.ttl: 2592000 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:08:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-20T20:08:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d6e19c1c-7286-43f7-95ba-2c0353c411d2 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862913945979, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862913946803, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80289, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 78450, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 72761, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862913, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d6e19c1c-7286-43f7-95ba-2c0353c411d2", "db_session_id": "XEAQL3WXSAHIK7AY1YPW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862913946906, "job": 1, "event": "recovery_finished"} 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563fbc700e00 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: DB pointer 0x563fbc7f4000 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:08:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 80.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 80.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:08:34.039 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 13.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 13.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:08:34.040 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x563fbc6d5350#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???) e1 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).mds e1 new map 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).mds e1 print_map 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T20:08:34.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:33 smithi078 ceph-mon[23685]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 bash[23666]: e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 ceph-mon[23685]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 ceph-mon[23685]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 ceph-mon[23685]: fsmap 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 ceph-mon[23685]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 ceph-mon[23685]: mgrmap e1: no daemons active 2024-09-20T20:08:34.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:34 smithi078 systemd[1]: Started Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:08:34.043 INFO:teuthology.orchestra.run.smithi078.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-20T20:08:35.052 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-20T20:08:35.074 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T20:08:35.074 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2024-09-20T20:08:35.075 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-20T20:08:35.075 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-20T20:08:35.308 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y 2024-09-20T20:08:35.308 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Unit ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service not loaded. 2024-09-20T20:08:35.452 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f2df6e16-778b-11ef-baf5-efdc52797490.target.wants/ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-f2df6e16-778b-11ef-baf5-efdc52797490@.service. 2024-09-20T20:08:35.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:35 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/510347278' entity='client.admin' 2024-09-20T20:08:35.995 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:35 smithi078 podman[23899]: 2024-09-20 20:08:35.897014787 +0000 UTC m=+0.324868500 container start a768f0dec912601b0a4753818b3f433637c3f8ab030dc98fd8cbc9daad32da42 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, RELEASE=reef-0b2cc4d, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T20:08:35.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:35 smithi078 bash[23899]: a768f0dec912601b0a4753818b3f433637c3f8ab030dc98fd8cbc9daad32da42 2024-09-20T20:08:36.009 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T20:08:36.009 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-20T20:08:36.016 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T20:08:36.016 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-20T20:08:36.016 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-20T20:08:36.016 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T20:08:36.025 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T20:08:36.025 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-20T20:08:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T20:08:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-20T20:08:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-20T20:08:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-20T20:08:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2024-09-20T20:08:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2024-09-20T20:08:36.278 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:35 smithi078 systemd[1]: Started Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:08:36.278 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.015+0000 7f3e1d93a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:08:36.278 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.081+0000 7f3e1d93a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:08:36.278 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.230+0000 7f3e1d93a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:08:36.528 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.300+0000 7f3e1d93a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:08:36.528 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.450+0000 7f3e1d93a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:08:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/955955515' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T20:08:36.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.598+0000 7f3e1d93a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:08:37.093 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.912+0000 7f3e1d93a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:08:37.094 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:36.972+0000 7f3e1d93a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:08:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:08:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "f2df6e16-778b-11ef-baf5-efdc52797490", 2024-09-20T20:08:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:37.104 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.105 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T20:08:37.106 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T20:08:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T20:08:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T20:08:20.356855+0000", 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T20:08:37.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:08:37.110 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2024-09-20T20:08:37.535 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:37 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:37.156+0000 7f3e1d93a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:08:38.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:38.025+0000 7f3e1d93a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:08:38.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:38.090+0000 7f3e1d93a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:08:39.113 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:38.798+0000 7f3e1d93a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:08:39.113 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:38.910+0000 7f3e1d93a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:08:39.113 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.047+0000 7f3e1d93a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:08:39.113 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.111+0000 7f3e1d93a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:08:39.396 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.264+0000 7f3e1d93a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:08:39.396 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.333+0000 7f3e1d93a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:08:39.396 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.393+0000 7f3e1d93a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:08:39.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1224079537' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T20:08:39.692 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.553+0000 7f3e1d93a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:08:39.692 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.622+0000 7f3e1d93a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:08:39.692 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.689+0000 7f3e1d93a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:08:40.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.772+0000 7f3e1d93a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:08:40.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.830+0000 7f3e1d93a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:08:40.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:39.891+0000 7f3e1d93a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:08:40.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:40.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:08:40.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "f2df6e16-778b-11ef-baf5-efdc52797490", 2024-09-20T20:08:40.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T20:08:40.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T20:08:40.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T20:08:40.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T20:08:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T20:08:40.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T20:08:20.356855+0000", 2024-09-20T20:08:40.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T20:08:40.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:40.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T20:08:40.160 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:08:40.160 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:40.666+0000 7f3e1d93a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: Activating manager daemon y 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: mgrmap e2: y(active, starting, since 0.0031251s) 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:08:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: Manager daemon y is now available 2024-09-20T20:08:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:08:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:08:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' 2024-09-20T20:08:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' 2024-09-20T20:08:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:40 smithi078 ceph-mon[23685]: from='mgr.14100 172.21.15.78:0/3920189060' entity='mgr.y' 2024-09-20T20:08:42.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:42 smithi078 ceph-mon[23685]: mgrmap e3: y(active, since 1.00632s) 2024-09-20T20:08:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:08:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "f2df6e16-778b-11ef-baf5-efdc52797490", 2024-09-20T20:08:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T20:08:43.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T20:08:43.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T20:08:43.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T20:08:43.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T20:08:43.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T20:08:43.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T20:08:20.356855+0000", 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T20:08:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-20T20:08:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T20:08:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:08:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2024-09-20T20:08:43.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:43 smithi078 ceph-mon[23685]: mgrmap e4: y(active, since 2s) 2024-09-20T20:08:43.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:43 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1812439979' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T20:08:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:44 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/150779455' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T20:08:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:44 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/150779455' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:44.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T20:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T20:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-20T20:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T20:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T20:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T20:08:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2024-09-20T20:08:45.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:45 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/83171565' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T20:08:45.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setuser ceph since I am not root 2024-09-20T20:08:45.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setgroup ceph since I am not root 2024-09-20T20:08:45.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:45.820+0000 7f9020a2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:08:45.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:45.875+0000 7f9020a2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:08:46.256 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.023+0000 7f9020a2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:08:46.257 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.087+0000 7f9020a2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:08:46.257 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.231+0000 7f9020a2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:08:46.511 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.372+0000 7f9020a2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:08:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/83171565' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T20:08:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-mon[23685]: mgrmap e5: y(active, since 5s) 2024-09-20T20:08:46.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.684+0000 7f9020a2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:08:46.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.746+0000 7f9020a2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:08:47.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:46.931+0000 7f9020a2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:08:47.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:08:47.524 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-20T20:08:47.525 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T20:08:47.525 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T20:08:47.525 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T20:08:47.525 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:08:47.525 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-09-20T20:08:47.525 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 5... 2024-09-20T20:08:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:47 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/28078643' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T20:08:48.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:47 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:47.813+0000 7f9020a2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:08:48.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:47 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:47.882+0000 7f9020a2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:08:48.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:48 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:48.578+0000 7f9020a2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:08:48.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:48 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:48.690+0000 7f9020a2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:08:48.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:48 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:48.827+0000 7f9020a2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:08:48.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:48 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:48.887+0000 7f9020a2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:08:49.157 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.030+0000 7f9020a2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:08:49.157 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.096+0000 7f9020a2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:08:49.157 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.154+0000 7f9020a2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:08:49.438 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.307+0000 7f9020a2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:08:49.438 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.371+0000 7f9020a2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:08:49.438 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.436+0000 7f9020a2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:08:49.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.516+0000 7f9020a2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:08:49.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.575+0000 7f9020a2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:08:49.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:49.635+0000 7f9020a2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:08:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:50.397+0000 7f9020a2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: Active manager daemon y restarted 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: Activating manager daemon y 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: mgrmap e6: y(active, starting, since 0.00360936s) 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: Manager daemon y is now available 2024-09-20T20:08:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:50.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:50.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:50.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:08:50.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:08:50.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:08:50.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:50 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:08:51.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:51 smithi078 ceph-mon[23685]: Found migration_current of "None". Setting to last migration. 2024-09-20T20:08:51.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:51 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:51.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:51 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:51.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:51 smithi078 ceph-mon[23685]: mgrmap e7: y(active, since 1.00503s) 2024-09-20T20:08:52.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:08:52.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-20T20:08:52.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T20:08:52.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:08:52.152 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 5 is available 2024-09-20T20:08:52.152 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2024-09-20T20:08:52.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-mon[23685]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T20:08:52.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-mon[23685]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T20:08:52.777 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:52.488+0000 7f8f8870b640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:20:08:52] ENGINE Error in HTTPServer.serve 2024-09-20T20:08:52.777 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: Traceback (most recent call last): 2024-09-20T20:08:52.777 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:08:52.777 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._connections.run(self.expiration_interval) 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._run(expiration_interval) 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: s = self.context.wrap_socket( 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: return self.sslsocket_class._create( 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self.do_handshake() 2024-09-20T20:08:52.778 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:08:52.779 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._sslobj.do_handshake() 2024-09-20T20:08:52.779 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:08:52.779 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:08:53.596 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: [20/Sep/2024:20:08:52] ENGINE Bus STARTING 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: [20/Sep/2024:20:08:52] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: mgrmap e8: y(active, since 2s) 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: [20/Sep/2024:20:08:52] ENGINE Serving on https://172.21.15.78:7150 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: [20/Sep/2024:20:08:52] ENGINE Bus STARTED 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: [20/Sep/2024:20:08:52] ENGINE Error in HTTPServer.serve 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:08:53.597 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:53.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:53 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:08:54.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:54 smithi078 ceph-mon[23685]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:54.517 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-20T20:08:54.517 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-mon[23685]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: Generating public/private rsa key pair. 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: Your identification has been saved in /tmp/tmp_l0k98yq/key 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: Your public key has been saved in /tmp/tmp_l0k98yq/key.pub 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: The key fingerprint is: 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: SHA256:sJJgJ0QCNkh4Df9Bgw5ALEFz1ix+hIFE4/nEB1x1n50 ceph-f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: The key's randomart image is: 2024-09-20T20:08:55.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: +---[RSA 3072]----+ 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: |/#*B*o+. . | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: |*=@=+= .. . o . | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: |.+===.o o E | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: | .o=o+ + | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: | .+ o S | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: | . | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: | | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: | | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: | | 2024-09-20T20:08:55.537 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:08:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: +----[SHA256]-----+ 2024-09-20T20:08:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:56 smithi078 ceph-mon[23685]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:56 smithi078 ceph-mon[23685]: Generating ssh key... 2024-09-20T20:08:57.006 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgVjSsV5TMLmk/yYL66nd3sSKcfGfNKeam1BcRRwaQ7MwOLbmUbAqjWjk/XtQIK/9ejo7K1ajxJ2YXXEI2NeoOHUtelWtueHxgKPAV4vGlpaMuoUSgS+zkIamW+j+uMGh1jGJXAgI8oy6cFDzf6MmLqXpsya3TqZL4BBKzzMhh6d8qUm7BEI6VDokkt6Vuk1AABz3tcbNx4x+zCtHVP2+QeE4Doz8k3jwSD1Y0e104Hle4QThksY2HqrgHiXlB0Ucye2eDNHT17FrndExtAws1FO/Xy9gFwtN4C9dgmM5ffNXKEDGsCeHM4zEwpCxMmjesA4qn/oe90F9x3yEJjpH/GWlLMyjOQ0HsY2vgs/dNTcAD1RLO7nrX64hUJ3f1MX/arPidryVnlVg0hZ5pwMLKe/Rl3RPLAsOXo3lqD6kj0mAGhq+QR//GZqCOf68E6nRpsokgPmtScwLFzhctyThKb7AVTr6LFtyhvxyR4UDVHdtki6NqpMCtgcWO6m7ReRM= ceph-f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:08:57.006 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T20:08:57.006 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2024-09-20T20:08:57.008 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2024-09-20T20:08:57.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:57 smithi078 ceph-mon[23685]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:58.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:58 smithi078 ceph-mon[23685]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:08:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:08:59 smithi078 ceph-mon[23685]: Deploying cephadm binary to smithi078 2024-09-20T20:09:00.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2024-09-20T20:09:00.646 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2024-09-20T20:09:01.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:01 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:01.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:01 smithi078 ceph-mon[23685]: Added host smithi078 2024-09-20T20:09:01.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:01 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:01.912 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-20T20:09:01.912 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2024-09-20T20:09:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:02 smithi078 ceph-mon[23685]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:02 smithi078 ceph-mon[23685]: Saving service mon spec with placement count:5 2024-09-20T20:09:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:02 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:03 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:03 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:03 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.378 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: Saving service mgr spec with placement count:2 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' 2024-09-20T20:09:04.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:09:04.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:04 smithi078 ceph-mon[23685]: from='mgr.14118 172.21.15.78:0/2949935646' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]': finished 2024-09-20T20:09:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:05 smithi078 ceph-mon[23685]: Deploying daemon agent.smithi078 on smithi078 2024-09-20T20:09:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:05 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2245827312' entity='client.admin' 2024-09-20T20:09:06.671 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2024-09-20T20:09:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:07 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2548241449' entity='client.admin' 2024-09-20T20:09:08.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2878640717' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T20:09:08.484 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setuser ceph since I am not root 2024-09-20T20:09:08.485 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setgroup ceph since I am not root 2024-09-20T20:09:08.485 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:08.316+0000 7f9a246ad040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:09:08.485 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:08.371+0000 7f9a246ad040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:09:08.734 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:08.521+0000 7f9a246ad040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:09:08.734 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:08.587+0000 7f9a246ad040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:09:08.734 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:08.732+0000 7f9a246ad040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:09:09.002 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:08.876+0000 7f9a246ad040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:09:09.255 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:09.189+0000 7f9a246ad040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:09:09.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:09 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2878640717' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T20:09:09.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:09 smithi078 ceph-mon[23685]: mgrmap e9: y(active, since 17s) 2024-09-20T20:09:09.535 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:09.253+0000 7f9a246ad040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:09:09.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:09.439+0000 7f9a246ad040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:09:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-09-20T20:09:09.994 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2024-09-20T20:09:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:10 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/917115000' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T20:09:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:10.326+0000 7f9a246ad040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:09:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:10.394+0000 7f9a246ad040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:09:11.363 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.102+0000 7f9a246ad040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:09:11.363 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.217+0000 7f9a246ad040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:09:11.648 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.360+0000 7f9a246ad040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:09:11.648 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.423+0000 7f9a246ad040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:09:11.649 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.576+0000 7f9a246ad040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:09:11.649 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.646+0000 7f9a246ad040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:09:11.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.709+0000 7f9a246ad040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:09:11.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.872+0000 7f9a246ad040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:09:11.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:11.939+0000 7f9a246ad040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:09:12.214 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:12.007+0000 7f9a246ad040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:09:12.214 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:12.092+0000 7f9a246ad040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:09:12.214 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:12.152+0000 7f9a246ad040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:09:12.215 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:12.212+0000 7f9a246ad040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:09:13.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:12.993+0000 7f9a246ad040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: Active manager daemon y restarted 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: Activating manager daemon y 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: mgrmap e10: y(active, starting, since 0.00305442s) 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:09:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: Manager daemon y is now available 2024-09-20T20:09:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:09:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:09:14.450 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:14.363+0000 7f9990342640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:20:09:14] ENGINE Error in HTTPServer.serve 2024-09-20T20:09:14.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: Traceback (most recent call last): 2024-09-20T20:09:14.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:09:14.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._connections.run(self.expiration_interval) 2024-09-20T20:09:14.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:09:14.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._run(expiration_interval) 2024-09-20T20:09:14.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: s = self.context.wrap_socket( 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: return self.sslsocket_class._create( 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self.do_handshake() 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._sslobj.do_handshake() 2024-09-20T20:09:14.452 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:09:14.453 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:09:14.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-20T20:09:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-20T20:09:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T20:09:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-20T20:09:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2024-09-20T20:09:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2024-09-20T20:09:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-mon[23685]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T20:09:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-mon[23685]: mgrmap e11: y(active, since 1.00553s) 2024-09-20T20:09:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:14 smithi078 ceph-mon[23685]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T20:09:15.909 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-20T20:09:15.909 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2024-09-20T20:09:16.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: [20/Sep/2024:20:09:14] ENGINE Bus STARTING 2024-09-20T20:09:16.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: [20/Sep/2024:20:09:14] ENGINE Serving on https://172.21.15.78:7150 2024-09-20T20:09:16.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: [20/Sep/2024:20:09:14] ENGINE Error in HTTPServer.serve 2024-09-20T20:09:16.147 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:09:16.148 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: [20/Sep/2024:20:09:14] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: [20/Sep/2024:20:09:14] ENGINE Bus STARTED 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: mgrmap e12: y(active, since 2s) 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:09:16.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:16.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:16.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:17.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:17 smithi078 ceph-mon[23685]: Deploying daemon agent.smithi078 on smithi078 2024-09-20T20:09:17.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:17.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$EHnku2/oh.qzgevzy/Y7XuHd1IB0MkWtibuqoVLiAIfKQUjOIJg2y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726862956, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T20:09:17.273 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2024-09-20T20:09:18.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:18 smithi078 ceph-mon[23685]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:18.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:18 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3321410799' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T20:09:18.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2024-09-20T20:09:18.447 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T20:09:18.447 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-20T20:09:18.454 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T20:09:18.454 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-20T20:09:18.454 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-20T20:09:18.454 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T20:09:18.456 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2024-09-20T20:09:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078.front.sepia.ceph.com:8443/ 2024-09-20T20:09:18.456 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2024-09-20T20:09:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: Password: b3ma8zykgb 2024-09-20T20:09:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:18.457 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config directory 2024-09-20T20:09:18.457 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling autotune for osd_memory_target 2024-09-20T20:09:20.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:20 smithi078 ceph-mon[23685]: mgrmap e13: y(active, since 6s) 2024-09-20T20:09:20.842 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-20T20:09:20.842 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-20T20:09:20.842 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.842 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:20.843 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2024-09-20T20:09:20.875 INFO:tasks.cephadm:Fetching config... 2024-09-20T20:09:20.875 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:09:20.876 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T20:09:20.891 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T20:09:20.891 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:09:20.891 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T20:09:20.945 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T20:09:20.945 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:09:20.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-20T20:09:21.075 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T20:09:21.076 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:09:21.076 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T20:09:21.136 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T20:09:21.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgVjSsV5TMLmk/yYL66nd3sSKcfGfNKeam1BcRRwaQ7MwOLbmUbAqjWjk/XtQIK/9ejo7K1ajxJ2YXXEI2NeoOHUtelWtueHxgKPAV4vGlpaMuoUSgS+zkIamW+j+uMGh1jGJXAgI8oy6cFDzf6MmLqXpsya3TqZL4BBKzzMhh6d8qUm7BEI6VDokkt6Vuk1AABz3tcbNx4x+zCtHVP2+QeE4Doz8k3jwSD1Y0e104Hle4QThksY2HqrgHiXlB0Ucye2eDNHT17FrndExtAws1FO/Xy9gFwtN4C9dgmM5ffNXKEDGsCeHM4zEwpCxMmjesA4qn/oe90F9x3yEJjpH/GWlLMyjOQ0HsY2vgs/dNTcAD1RLO7nrX64hUJ3f1MX/arPidryVnlVg0hZ5pwMLKe/Rl3RPLAsOXo3lqD6kj0mAGhq+QR//GZqCOf68E6nRpsokgPmtScwLFzhctyThKb7AVTr6LFtyhvxyR4UDVHdtki6NqpMCtgcWO6m7ReRM= ceph-f2df6e16-778b-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T20:09:21.280 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgVjSsV5TMLmk/yYL66nd3sSKcfGfNKeam1BcRRwaQ7MwOLbmUbAqjWjk/XtQIK/9ejo7K1ajxJ2YXXEI2NeoOHUtelWtueHxgKPAV4vGlpaMuoUSgS+zkIamW+j+uMGh1jGJXAgI8oy6cFDzf6MmLqXpsya3TqZL4BBKzzMhh6d8qUm7BEI6VDokkt6Vuk1AABz3tcbNx4x+zCtHVP2+QeE4Doz8k3jwSD1Y0e104Hle4QThksY2HqrgHiXlB0Ucye2eDNHT17FrndExtAws1FO/Xy9gFwtN4C9dgmM5ffNXKEDGsCeHM4zEwpCxMmjesA4qn/oe90F9x3yEJjpH/GWlLMyjOQ0HsY2vgs/dNTcAD1RLO7nrX64hUJ3f1MX/arPidryVnlVg0hZ5pwMLKe/Rl3RPLAsOXo3lqD6kj0mAGhq+QR//GZqCOf68E6nRpsokgPmtScwLFzhctyThKb7AVTr6LFtyhvxyR4UDVHdtki6NqpMCtgcWO6m7ReRM= ceph-f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:09:21.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgVjSsV5TMLmk/yYL66nd3sSKcfGfNKeam1BcRRwaQ7MwOLbmUbAqjWjk/XtQIK/9ejo7K1ajxJ2YXXEI2NeoOHUtelWtueHxgKPAV4vGlpaMuoUSgS+zkIamW+j+uMGh1jGJXAgI8oy6cFDzf6MmLqXpsya3TqZL4BBKzzMhh6d8qUm7BEI6VDokkt6Vuk1AABz3tcbNx4x+zCtHVP2+QeE4Doz8k3jwSD1Y0e104Hle4QThksY2HqrgHiXlB0Ucye2eDNHT17FrndExtAws1FO/Xy9gFwtN4C9dgmM5ffNXKEDGsCeHM4zEwpCxMmjesA4qn/oe90F9x3yEJjpH/GWlLMyjOQ0HsY2vgs/dNTcAD1RLO7nrX64hUJ3f1MX/arPidryVnlVg0hZ5pwMLKe/Rl3RPLAsOXo3lqD6kj0mAGhq+QR//GZqCOf68E6nRpsokgPmtScwLFzhctyThKb7AVTr6LFtyhvxyR4UDVHdtki6NqpMCtgcWO6m7ReRM= ceph-f2df6e16-778b-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T20:09:21.348 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgVjSsV5TMLmk/yYL66nd3sSKcfGfNKeam1BcRRwaQ7MwOLbmUbAqjWjk/XtQIK/9ejo7K1ajxJ2YXXEI2NeoOHUtelWtueHxgKPAV4vGlpaMuoUSgS+zkIamW+j+uMGh1jGJXAgI8oy6cFDzf6MmLqXpsya3TqZL4BBKzzMhh6d8qUm7BEI6VDokkt6Vuk1AABz3tcbNx4x+zCtHVP2+QeE4Doz8k3jwSD1Y0e104Hle4QThksY2HqrgHiXlB0Ucye2eDNHT17FrndExtAws1FO/Xy9gFwtN4C9dgmM5ffNXKEDGsCeHM4zEwpCxMmjesA4qn/oe90F9x3yEJjpH/GWlLMyjOQ0HsY2vgs/dNTcAD1RLO7nrX64hUJ3f1MX/arPidryVnlVg0hZ5pwMLKe/Rl3RPLAsOXo3lqD6kj0mAGhq+QR//GZqCOf68E6nRpsokgPmtScwLFzhctyThKb7AVTr6LFtyhvxyR4UDVHdtki6NqpMCtgcWO6m7ReRM= ceph-f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:09:21.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:21 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3981985622' entity='client.admin' 2024-09-20T20:09:21.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T20:09:21.735 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:09:22.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:24 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:25.649 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T20:09:25.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T20:09:25.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:25 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1898613826' entity='client.admin' 2024-09-20T20:09:25.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:25.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:25.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:25.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:26.114 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:09:29.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:29.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:29.141 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-09-20T20:09:29.141 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:09:29.141 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-09-20T20:09:29.160 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:09:29.161 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:09:29.218 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-09-20T20:09:29.219 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch host add smithi146 2024-09-20T20:09:29.409 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:09:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:29 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:09:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:29 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:31 smithi078 ceph-mon[23685]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:32 smithi078 ceph-mon[23685]: Deploying cephadm binary to smithi146 2024-09-20T20:09:33.845 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi146' with addr '172.21.15.146' 2024-09-20T20:09:34.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:34 smithi078 ceph-mon[23685]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:34.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:34.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:34.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T20:09:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:09:35.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: Added host smithi146 2024-09-20T20:09:35.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:35.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:35.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:35.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:09:35.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:35.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:09:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]': finished 2024-09-20T20:09:36.360 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:09:36.360 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-09-20T20:09:36.959 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T20:09:36.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T20:09:37.157 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:09:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: Deploying daemon agent.smithi146 on smithi146 2024-09-20T20:09:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:38.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:38 smithi078 ceph-mon[23685]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:09:38.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:38 smithi078 ceph-mon[23685]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:38.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:38 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:38.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:38 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:38.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:38 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:39.177 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-09-20T20:09:39.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:39 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1455908183' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T20:09:39.753 INFO:tasks.cephadm:Adding mon.a on smithi078 2024-09-20T20:09:39.753 INFO:tasks.cephadm:Adding mon.c on smithi078 2024-09-20T20:09:39.753 INFO:tasks.cephadm:Adding mon.b on smithi146 2024-09-20T20:09:39.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b' 2024-09-20T20:09:39.946 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:09:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:40 smithi078 ceph-mon[23685]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:40 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1455908183' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T20:09:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:40 smithi078 ceph-mon[23685]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:41.602 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-09-20T20:09:42.201 DEBUG:teuthology.orchestra.run.smithi078:mon.c> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.c.service 2024-09-20T20:09:42.204 DEBUG:teuthology.orchestra.run.smithi146:mon.b> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.b.service 2024-09-20T20:09:42.206 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T20:09:42.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T20:09:42.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:42.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:42.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:42.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:42.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:42.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:42.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:09:42.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:43.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:43 smithi078 ceph-mon[23685]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:43.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:43 smithi078 ceph-mon[23685]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b;count:3 2024-09-20T20:09:43.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:43 smithi078 ceph-mon[23685]: Deploying daemon mon.b on smithi146 2024-09-20T20:09:44.223 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:09:44.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:44 smithi078 ceph-mon[23685]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:44.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:44 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:44.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:44 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:44.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:44 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:45.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:45 smithi146 systemd[1]: Starting Ceph mon.b for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:09:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:45 smithi146 podman[25549]: 2024-09-20 20:09:45.825043875 +0000 UTC m=+0.095070737 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T20:09:46.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 podman[25549]: 2024-09-20 20:09:46.009220184 +0000 UTC m=+0.279247035 container create 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:09:46.151 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:09:46.151 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"f2df6e16-778b-11ef-baf5-efdc52797490","modified":"2024-09-20T20:08:17.222812Z","created":"2024-09-20T20:08:17.222812Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T20:09:46.151 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-09-20T20:09:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 podman[25549]: 2024-09-20 20:09:46.245480649 +0000 UTC m=+0.515507515 container init 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True) 2024-09-20T20:09:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 podman[25549]: 2024-09-20 20:09:46.248867621 +0000 UTC m=+0.518894475 container start 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T20:09:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:46 smithi078 ceph-mon[23685]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:46 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/3713885782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:09:46.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:09:46.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T20:09:46.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pidfile_write: ignore empty --pid-file 2024-09-20T20:09:46.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: load: jerasure load: lrc 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Git sha 0 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: DB SUMMARY 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: DB Session ID: V5IYVEIZ0A677VRS4TM3 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:09:46.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.env: 0x556ba3193bc0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.info_log: 0x556ba5304be0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.statistics: (nil) 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.use_fsync: 0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:09:46.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.db_log_dir: 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.wal_dir: 2024-09-20T20:09:46.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.write_buffer_manager: 0x556ba5312320 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:09:46.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.unordered_write: 0 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.row_cache: None 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.wal_filter: None 2024-09-20T20:09:46.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.wal_compression: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:09:46.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_open_files: -1 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:09:46.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Compression algorithms supported: 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kZSTD supported: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:09:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.merge_operator: 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_filter: None 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ba53047e0) 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:09:46.668 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_type: 0 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_index_type: 0 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_shortening: 1 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: checksum: 4 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: no_block_cache: 0 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache: 0x556ba52f9350 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_options: 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: capacity : 536870912 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_shard_bits : 4 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: strict_capacity_limit : 0 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_compressed: (nil) 2024-09-20T20:09:46.669 INFO:journalctl@ceph.mon.b.smithi146.stdout: persistent_cache: (nil) 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size: 4096 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size_deviation: 10 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_restart_interval: 16 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_block_restart_interval: 1 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: metadata_block_size: 4096 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: partition_filters: 0 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_delta_encoding: 1 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: filter_policy: bloomfilter 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: whole_key_filtering: 1 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: verify_compression: 0 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: format_version: 5 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_index_compression: 1 2024-09-20T20:09:46.670 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_align: 0 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout: max_auto_readahead_size: 262144 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout: prepopulate_block_cache: 0 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression: NoCompression 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.num_levels: 7 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:09:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:09:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:09:46.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:09:46.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:09:46.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.ttl: 2592000 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:09:46.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b6608849-9ec2-4625-b45b-41932e88bcb9 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862986275602, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862986275999, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862986, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b6608849-9ec2-4625-b45b-41932e88bcb9", "db_session_id": "V5IYVEIZ0A677VRS4TM3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862986276096, "job": 1, "event": "recovery_finished"} 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556ba5324e00 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: DB pointer 0x556ba5418000 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:09:46.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** DB Stats ** 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:46.678 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:09:46.679 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache BinnedLRUCache@0x556ba52f9350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: using public_addr v2:172.21.15.146:0/0 -> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: starting mon.b rank -1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(???) e0 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).mds e1 new map 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: e1 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:09:46.680 INFO:journalctl@ceph.mon.b.smithi146.stdout: legacy client fscid: -1 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout: No filesystems configured 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:09:46.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]': finished 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Deploying daemon agent.smithi146 on smithi146 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:46.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1455908183' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1455908183' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b;count:3 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: Deploying daemon mon.b on smithi146 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/3713885782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:09:46.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 ceph-mon[25603]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-20T20:09:46.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 bash[25549]: 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 2024-09-20T20:09:46.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:46 smithi146 systemd[1]: Started Ceph mon.b for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:09:48.048 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T20:09:48.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T20:09:48.259 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: mon.a calling monitor election 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: mon.b calling monitor election 2024-09-20T20:09:51.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: fsmap 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: mgrmap e13: y(active, since 38s) 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: overall HEALTH_OK 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:09:51.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: mon.a calling monitor election 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: mon.b calling monitor election 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: fsmap 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: mgrmap e13: y(active, since 38s) 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: overall HEALTH_OK 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:09:51.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:52 smithi146 ceph-mon[25603]: Deploying daemon mon.c on smithi078 2024-09-20T20:09:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:52 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:52 smithi078 ceph-mon[23685]: Deploying daemon mon.c on smithi078 2024-09-20T20:09:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:52 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:09:53.008 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"fsid":"f2df6e16-778b-11ef-baf5-efdc52797490","modified":"2024-09-20T20:09:46.303267Z","created":"2024-09-20T20:08:17.222812Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T20:09:53.009 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 2 2024-09-20T20:09:53.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:53 smithi078 ceph-mon[23685]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:53.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:53 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/3127635547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:09:53.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:53 smithi146 ceph-mon[25603]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:53.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:53 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/3127635547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:09:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 podman[30010]: 2024-09-20 20:09:54.233441708 +0000 UTC m=+0.129750305 container create 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-20T20:09:54.727 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T20:09:54.727 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T20:09:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 podman[30010]: 2024-09-20 20:09:54.435665167 +0000 UTC m=+0.331973773 container init 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T20:09:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 podman[30010]: 2024-09-20 20:09:54.438712808 +0000 UTC m=+0.335021408 container start 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T20:09:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable), process ceph-mon, pid 2 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pidfile_write: ignore empty --pid-file 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: load: jerasure load: lrc 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Git sha 0 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Compile date 2024-09-20 15:42:07 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: DB SUMMARY 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: DB Session ID: 4KGNLJ94PZKF6TT7D3PH 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T20:09:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.env: 0x55c482cf3bc0 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.info_log: 0x55c483df6be0 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:09:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.statistics: (nil) 2024-09-20T20:09:54.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.use_fsync: 0 2024-09-20T20:09:54.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:09:54.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.db_log_dir: 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.wal_dir: 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:09:54.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.write_buffer_manager: 0x55c483e04320 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.unordered_write: 0 2024-09-20T20:09:54.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.row_cache: None 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.wal_filter: None 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.wal_compression: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:09:54.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:09:54.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_open_files: -1 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Compression algorithms supported: 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kZSTD supported: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:09:54.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.merge_operator: 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_filter: None 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:09:54.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c483df67e0) 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_type: 0 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: data_block_index_type: 0 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_shortening: 1 2024-09-20T20:09:54.797 INFO:journalctl@ceph.mon.c.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: checksum: 4 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: no_block_cache: 0 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache: 0x55c483deb1f0 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_options: 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: capacity : 536870912 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: num_shard_bits : 4 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: strict_capacity_limit : 0 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_compressed: (nil) 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: persistent_cache: (nil) 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_size: 4096 2024-09-20T20:09:54.798 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_size_deviation: 10 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_restart_interval: 16 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_block_restart_interval: 1 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: metadata_block_size: 4096 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: partition_filters: 0 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: use_delta_encoding: 1 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: filter_policy: bloomfilter 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: whole_key_filtering: 1 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: verify_compression: 0 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: format_version: 5 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: enable_index_compression: 1 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_align: 0 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-20T20:09:54.799 INFO:journalctl@ceph.mon.c.smithi078.stdout: prepopulate_block_cache: 0 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression: NoCompression 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.num_levels: 7 2024-09-20T20:09:54.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:09:54.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:09:54.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:09:54.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:09:54.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.ttl: 2592000 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:09:54.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T20:09:54.806 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 02a0691f-2c78-405d-ad9e-04f6291722b7 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862994465488, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862994465879, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726862994, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "02a0691f-2c78-405d-ad9e-04f6291722b7", "db_session_id": "4KGNLJ94PZKF6TT7D3PH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726862994465978, "job": 1, "event": "recovery_finished"} 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c483e16e00 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: DB pointer 0x55c483f0a000 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:09:54.807 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** DB Stats ** 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:09:54.808 INFO:journalctl@ceph.mon.c.smithi078.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:09:54.809 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: Block cache BinnedLRUCache@0x55c483deb1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: using public_addrv [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] 2024-09-20T20:09:54.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: starting mon.c rank -1 at public addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] at bind addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(???) e0 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).mds e1 new map 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout: e1 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout: legacy client fscid: -1 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout: No filesystems configured 2024-09-20T20:09:54.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:09:54.812 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]': finished 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Deploying daemon agent.smithi146 on smithi146 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:54.813 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1455908183' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1455908183' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi078:[v2:172.21.15.78:3301,v1:172.21.15.78:6790]=c;smithi146:172.21.15.146=b;count:3 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Deploying daemon mon.b on smithi146 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/3713885782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.a calling monitor election 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.b calling monitor election 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.816 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: fsmap 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mgrmap e13: y(active, since 38s) 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: overall HEALTH_OK 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: Deploying daemon mon.c on smithi078 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:54.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/3127635547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:09:54.818 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 ceph-mon[30028]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-20T20:09:54.818 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 bash[30010]: 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac 2024-09-20T20:09:54.818 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:54 smithi078 systemd[1]: Started Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:09:54.932 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: mon.a calling monitor election 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: mon.b calling monitor election 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:59.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: mon.c calling monitor election 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: fsmap 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: mgrmap e13: y(active, since 46s) 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: overall HEALTH_OK 2024-09-20T20:09:59.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:09:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:09:59.868 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:09:59.868 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"fsid":"f2df6e16-778b-11ef-baf5-efdc52797490","modified":"2024-09-20T20:09:54.489834Z","created":"2024-09-20T20:08:17.222812Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3301","nonce":0},{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-20T20:09:59.869 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 3 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: mon.a calling monitor election 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: mon.b calling monitor election 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: mon.c calling monitor election 2024-09-20T20:10:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: fsmap 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: mgrmap e13: y(active, since 46s) 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: overall HEALTH_OK 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: mon.a calling monitor election 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: mon.b calling monitor election 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:00.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: mon.c calling monitor election 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: fsmap 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: mgrmap e13: y(active, since 46s) 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: overall HEALTH_OK 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:09:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:00.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:00.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:00.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/2408626886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: overall HEALTH_OK 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:00.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T20:10:00.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:00.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:00.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:00.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: Reconfiguring daemon mon.a on smithi078 2024-09-20T20:10:00.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:10:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/2408626886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: overall HEALTH_OK 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: Reconfiguring daemon mon.a on smithi078 2024-09-20T20:10:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/2408626886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: overall HEALTH_OK 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: Reconfiguring daemon mon.a on smithi078 2024-09-20T20:10:00.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:10:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:01 smithi146 ceph-mon[25603]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[23685]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:02.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:01 smithi078 ceph-mon[30028]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:02.421 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T20:10:02.421 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T20:10:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:10:03.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: Reconfiguring mon.c (monmap changed)... 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: Reconfiguring daemon mon.c on smithi078 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: Reconfiguring mon.c (monmap changed)... 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: Reconfiguring daemon mon.c on smithi078 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:03 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: Reconfiguring mon.c (monmap changed)... 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: Reconfiguring daemon mon.c on smithi078 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:03 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:04.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:04.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:04.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:04.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:05 smithi078 ceph-mon[23685]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:05 smithi078 ceph-mon[30028]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:06.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:05 smithi146 ceph-mon[25603]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:07.038 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:10:07.038 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-09-20T20:10:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:10:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: Reconfiguring mon.b (monmap changed)... 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:07.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: Reconfiguring daemon mon.b on smithi146 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2853479818' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: Reconfiguring mon.b (monmap changed)... 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:07.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: Reconfiguring daemon mon.b on smithi146 2024-09-20T20:10:07.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:07.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:07 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2853479818' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:07.624 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T20:10:07.624 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:10:07.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T20:10:07.654 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:10:07.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:10:07.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: Reconfiguring mon.b (monmap changed)... 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: Reconfiguring daemon mon.b on smithi146 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:07.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:07 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2853479818' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:07.719 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:10:07.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T20:10:07.746 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:10:07.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:10:07.811 INFO:tasks.cephadm:Adding mgr.y on smithi078 2024-09-20T20:10:07.812 INFO:tasks.cephadm:Adding mgr.x on smithi146 2024-09-20T20:10:07.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch apply mgr '2;smithi078=y;smithi146=x' 2024-09-20T20:10:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:10:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:08 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:08 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:08 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:08 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:08 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:08 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:08 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:09.636 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mgr update... 2024-09-20T20:10:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:09 smithi146 ceph-mon[25603]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:09 smithi078 ceph-mon[23685]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:09 smithi078 ceph-mon[30028]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:10.267 DEBUG:teuthology.orchestra.run.smithi146:mgr.x> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service 2024-09-20T20:10:10.269 DEBUG:tasks.cephadm:set 0 configs 2024-09-20T20:10:10.269 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T20:10:10.269 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:10:10.269 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T20:10:10.288 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T20:10:10.288 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-09-20T20:10:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T20:10:10.347 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:10.347 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-20T20:10:10.347 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:10.347 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:10.348 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-20 20:10:03.043934517 +0000 2024-09-20T20:10:10.348 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-20 20:07:04.466931953 +0000 2024-09-20T20:10:10.348 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-20 20:07:04.466931953 +0000 2024-09-20T20:10:10.348 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-20 20:07:04.466931953 +0000 2024-09-20T20:10:10.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T20:10:10.416 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-20T20:10:10.416 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-20T20:10:10.416 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000301675 s, 1.7 MB/s 2024-09-20T20:10:10.417 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T20:10:10.477 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-09-20T20:10:10.536 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T20:10:10.536 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:10.536 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-20T20:10:10.536 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:10.536 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:10.536 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-20 20:10:03.043934517 +0000 2024-09-20T20:10:10.537 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-20 20:07:04.788927041 +0000 2024-09-20T20:10:10.537 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-20 20:07:04.788927041 +0000 2024-09-20T20:10:10.537 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-20 20:07:04.788927041 +0000 2024-09-20T20:10:10.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T20:10:10.603 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-20T20:10:10.604 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-20T20:10:10.604 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000203436 s, 2.5 MB/s 2024-09-20T20:10:10.605 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T20:10:10.665 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-09-20T20:10:10.723 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T20:10:10.723 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:10.723 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-20T20:10:10.723 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:10.723 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:10.724 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-20 20:10:03.043934517 +0000 2024-09-20T20:10:10.724 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-20 20:07:05.103922235 +0000 2024-09-20T20:10:10.724 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-20 20:07:05.103922235 +0000 2024-09-20T20:10:10.724 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-20 20:07:05.103922235 +0000 2024-09-20T20:10:10.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi078=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: Saving service mgr spec with placement smithi078=y;smithi146=x;count:2 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:10:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[23685]: Deploying daemon mgr.x on smithi146 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi078=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: Saving service mgr spec with placement smithi078=y;smithi146=x;count:2 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:10.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:10 smithi078 ceph-mon[30028]: Deploying daemon mgr.x on smithi146 2024-09-20T20:10:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-20T20:10:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-20T20:10:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000208836 s, 2.5 MB/s 2024-09-20T20:10:10.802 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T20:10:10.860 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-09-20T20:10:10.915 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T20:10:10.915 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:10.915 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-20T20:10:10.915 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:10.915 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:10.916 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-20 20:10:03.043934517 +0000 2024-09-20T20:10:10.916 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-20 20:07:05.411917537 +0000 2024-09-20T20:10:10.916 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-20 20:07:05.411917537 +0000 2024-09-20T20:10:10.916 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-20 20:07:05.411917537 +0000 2024-09-20T20:10:10.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T20:10:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-20T20:10:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-20T20:10:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000192351 s, 2.7 MB/s 2024-09-20T20:10:10.980 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T20:10:10.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi078=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:10.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: Saving service mgr spec with placement smithi078=y;smithi146=x;count:2 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:10.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:10 smithi146 ceph-mon[25603]: Deploying daemon mgr.x on smithi146 2024-09-20T20:10:11.037 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:10:11.037 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T20:10:11.053 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T20:10:11.053 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-20 20:10:02.265730263 +0000 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-20 20:06:59.809785037 +0000 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-20 20:06:59.809785037 +0000 2024-09-20T20:10:11.108 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-20 20:06:59.809785037 +0000 2024-09-20T20:10:11.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T20:10:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-20T20:10:11.173 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-20T20:10:11.173 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000195586 s, 2.6 MB/s 2024-09-20T20:10:11.174 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T20:10:11.230 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-20 20:10:02.265730263 +0000 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-20 20:07:00.134780206 +0000 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-20 20:07:00.134780206 +0000 2024-09-20T20:10:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-20 20:07:00.134780206 +0000 2024-09-20T20:10:11.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T20:10:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-20T20:10:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-20T20:10:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000166935 s, 3.1 MB/s 2024-09-20T20:10:11.349 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T20:10:11.406 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-20 20:10:02.265730263 +0000 2024-09-20T20:10:11.464 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-20 20:07:00.448775538 +0000 2024-09-20T20:10:11.465 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-20 20:07:00.448775538 +0000 2024-09-20T20:10:11.465 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-20 20:07:00.448775538 +0000 2024-09-20T20:10:11.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T20:10:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-20T20:10:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-20T20:10:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000182848 s, 2.8 MB/s 2024-09-20T20:10:11.543 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T20:10:11.599 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-20 20:10:11.493594396 +0000 2024-09-20T20:10:11.654 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-20 20:07:00.778770632 +0000 2024-09-20T20:10:11.655 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-20 20:07:00.778770632 +0000 2024-09-20T20:10:11.655 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-20 20:07:00.778770632 +0000 2024-09-20T20:10:11.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T20:10:11.724 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-20T20:10:11.724 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-20T20:10:11.725 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000223427 s, 2.3 MB/s 2024-09-20T20:10:11.726 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T20:10:11.784 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2024-09-20T20:10:11.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T20:10:11.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:11 smithi146 ceph-mon[25603]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:11.954 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:11 smithi146 systemd[1]: Starting Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:10:11.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:11 smithi078 ceph-mon[23685]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:11.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:11 smithi078 ceph-mon[30028]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:11.983 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:10:12.246 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:11 smithi146 podman[27794]: 2024-09-20 20:10:11.85457268 +0000 UTC m=+0.015453466 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T20:10:12.247 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:11 smithi146 podman[27794]: 2024-09-20 20:10:11.959522165 +0000 UTC m=+0.120402982 container create b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:10:12.247 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 podman[27794]: 2024-09-20 20:10:12.161927626 +0000 UTC m=+0.322808411 container init b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T20:10:12.247 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 podman[27794]: 2024-09-20 20:10:12.16457012 +0000 UTC m=+0.325450895 container start b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:10:12.247 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-mgr[27813]: -- 172.21.15.146:0/3604260741 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a879b76d00 con 0x55a879b4b000 2024-09-20T20:10:12.247 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 bash[27794]: b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b 2024-09-20T20:10:12.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 systemd[1]: Started Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:10:12.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:12.286+0000 7f4369055040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:10:12.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:12.345+0000 7f4369055040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:10:12.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:12.498+0000 7f4369055040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:10:12.766 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:12.563+0000 7f4369055040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:10:12.766 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:12.705+0000 7f4369055040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:10:13.147 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:12.842+0000 7f4369055040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:10:13.147 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:13.145+0000 7f4369055040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: Reconfiguring daemon mgr.y on smithi078 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:13.203+0000 7f4369055040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:10:13.518 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:13 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:13.382+0000 7f4369055040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: Reconfiguring daemon mgr.y on smithi078 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: Reconfiguring daemon mgr.y on smithi078 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:14.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:14 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:14.295+0000 7f4369055040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:10:14.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:14 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:14.362+0000 7f4369055040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:10:15.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:15.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:15.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:15.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:15.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:15.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:15.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:10:15.401 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.145+0000 7f4369055040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:10:15.401 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.286+0000 7f4369055040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:10:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:10:15.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2024-09-20T20:10:15.654 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.440+0000 7f4369055040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:10:15.654 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.502+0000 7f4369055040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:10:15.759 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:10:15.917 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.658+0000 7f4369055040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:10:15.918 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.725+0000 7f4369055040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:10:15.918 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.784+0000 7f4369055040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:10:16.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:16 smithi146 ceph-mon[25603]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:16.224 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:15.940+0000 7f4369055040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:10:16.224 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:16.005+0000 7f4369055040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:10:16.224 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:16.071+0000 7f4369055040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:10:16.224 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:16.158+0000 7f4369055040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:10:16.224 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:16.221+0000 7f4369055040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:10:16.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:16.285+0000 7f4369055040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:10:16.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:16 smithi078 ceph-mon[23685]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:16.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:16 smithi078 ceph-mon[30028]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:17.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:10:17.070+0000 7f4369055040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:17.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:10:17.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:10:17.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:10:17.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:10:17.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:17 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: Standby manager daemon x started 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:10:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:17 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/309526212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:10:18.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[23685]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:18.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:10:18.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[23685]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[30028]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[30028]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-20T20:10:18.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:10:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:18 smithi146 ceph-mon[25603]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:10:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:10:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:18 smithi146 ceph-mon[25603]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-20T20:10:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:10:19.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:19 smithi078 ceph-mon[23685]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:19.347 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:19 smithi078 ceph-mon[30028]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:19 smithi146 ceph-mon[25603]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2751704684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]: dispatch 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]: dispatch 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]': finished 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[23685]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/381593958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2751704684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]: dispatch 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]: dispatch 2024-09-20T20:10:20.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]': finished 2024-09-20T20:10:20.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[30028]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T20:10:20.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:20.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:20 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/381593958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:10:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:20 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2751704684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]: dispatch 2024-09-20T20:10:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:20 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]: dispatch 2024-09-20T20:10:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:20 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d0d6fc3-31fc-4889-b9e8-b088a65da5bb"}]': finished 2024-09-20T20:10:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:20 smithi146 ceph-mon[25603]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T20:10:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:20 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:20 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/381593958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:10:21.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:21 smithi078 ceph-mon[23685]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:21.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:21 smithi078 ceph-mon[30028]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:21 smithi146 ceph-mon[25603]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:24 smithi078 ceph-mon[23685]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:24 smithi078 ceph-mon[30028]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:24 smithi146 ceph-mon[25603]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[23685]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[30028]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:26 smithi146 ceph-mon[25603]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:27.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:27 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi078 2024-09-20T20:10:27.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:27 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T20:10:27.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:27 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:27.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:27 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi078 2024-09-20T20:10:27.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:27 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T20:10:27.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:27 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:27 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi078 2024-09-20T20:10:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:27 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T20:10:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:27 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:28.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:28 smithi078 ceph-mon[30028]: Deploying daemon osd.0 on smithi078 2024-09-20T20:10:28.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:28 smithi078 ceph-mon[30028]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:28.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:28 smithi078 ceph-mon[23685]: Deploying daemon osd.0 on smithi078 2024-09-20T20:10:28.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:28 smithi078 ceph-mon[23685]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:28 smithi146 ceph-mon[25603]: Deploying daemon osd.0 on smithi078 2024-09-20T20:10:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:28 smithi146 ceph-mon[25603]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:30 smithi146 ceph-mon[25603]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:30.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:30 smithi078 ceph-mon[23685]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:30.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:30 smithi078 ceph-mon[30028]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:32.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:32.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:32.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:32.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:32.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:32.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:32 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:32 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.057 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:34.707 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T20:10:34.710 INFO:tasks.cephadm:Deploying osd.1 on smithi078 with /dev/vg_nvme/lv_3... 2024-09-20T20:10:34.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T20:10:34.941 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:10:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:35 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi078 to 15040M 2024-09-20T20:10:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:35 smithi146 ceph-mon[25603]: from='osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T20:10:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:35 smithi146 ceph-mon[25603]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T20:10:35.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:35 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi078 to 15040M 2024-09-20T20:10:35.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:35 smithi078 ceph-mon[23685]: from='osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T20:10:35.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:35 smithi078 ceph-mon[23685]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T20:10:35.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:35 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi078 to 15040M 2024-09-20T20:10:35.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:35 smithi078 ceph-mon[30028]: from='osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T20:10:35.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:35 smithi078 ceph-mon[30028]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T20:10:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:36.345 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:36.346 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[34319]: 2024-09-20T20:10:36.103+0000 7f444fcce640 -1 osd.0 0 waiting for initial osdmap 2024-09-20T20:10:36.346 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:10:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[34319]: 2024-09-20T20:10:36.105+0000 7f444baf7640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:10:37.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[23685]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[23685]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[30028]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[30028]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:37.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:37 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:37.380 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:10:37.398 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2024-09-20T20:10:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:37 smithi146 ceph-mon[25603]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:10:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:37 smithi146 ceph-mon[25603]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T20:10:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:37 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:37 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:37.589 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:10:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:10:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:10:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775] boot 2024-09-20T20:10:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775] boot 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T20:10:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:38 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:10:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:10:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T20:10:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: osd.0 [v2:172.21.15.78:6802/1463358775,v1:172.21.15.78:6803/1463358775] boot 2024-09-20T20:10:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T20:10:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:38 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:10:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:39 smithi078 ceph-mon[23685]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T20:10:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:39 smithi078 ceph-mon[30028]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T20:10:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:39 smithi146 ceph-mon[25603]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[23685]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[30028]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:10:40.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:40 smithi146 ceph-mon[25603]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:10:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:10:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:41 smithi078 ceph-mon[23685]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:41 smithi078 ceph-mon[30028]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:41 smithi146 ceph-mon[25603]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:10:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:42 smithi146 ceph-mon[25603]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/782847522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24879284-6ef9-4428-a3f8-94aedad8a73d"}]: dispatch 2024-09-20T20:10:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/782847522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24879284-6ef9-4428-a3f8-94aedad8a73d"}]': finished 2024-09-20T20:10:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:42 smithi146 ceph-mon[25603]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T20:10:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:42 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[30028]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/782847522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24879284-6ef9-4428-a3f8-94aedad8a73d"}]: dispatch 2024-09-20T20:10:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/782847522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24879284-6ef9-4428-a3f8-94aedad8a73d"}]': finished 2024-09-20T20:10:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[30028]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T20:10:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:42.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[23685]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:42.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/782847522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24879284-6ef9-4428-a3f8-94aedad8a73d"}]: dispatch 2024-09-20T20:10:42.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/782847522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24879284-6ef9-4428-a3f8-94aedad8a73d"}]': finished 2024-09-20T20:10:42.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[23685]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T20:10:42.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:43.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:43 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3894351113' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:10:43.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:43 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3894351113' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:10:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:43 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3894351113' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:10:44.376 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[30028]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[23685]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:44 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:44 smithi146 ceph-mon[25603]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:44.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:44 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:44 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:44.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:44 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:46 smithi146 ceph-mon[25603]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:46 smithi078 ceph-mon[23685]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:46 smithi078 ceph-mon[30028]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:48 smithi078 ceph-mon[23685]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:48 smithi078 ceph-mon[30028]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:48 smithi146 ceph-mon[25603]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:49 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T20:10:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:49 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:49 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T20:10:49.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:49 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:49 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T20:10:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:49 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:50.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:50 smithi078 ceph-mon[30028]: Deploying daemon osd.1 on smithi078 2024-09-20T20:10:50.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:50 smithi078 ceph-mon[30028]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:50.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:50 smithi078 ceph-mon[23685]: Deploying daemon osd.1 on smithi078 2024-09-20T20:10:50.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:50 smithi078 ceph-mon[23685]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:50 smithi146 ceph-mon[25603]: Deploying daemon osd.1 on smithi078 2024-09-20T20:10:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:50 smithi146 ceph-mon[25603]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:52 smithi078 ceph-mon[23685]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:52 smithi078 ceph-mon[30028]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:52 smithi146 ceph-mon[25603]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:54.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:54.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:54.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:54.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:54.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.181 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 1 on host 'smithi078' 2024-09-20T20:10:56.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:56 smithi146 ceph-mon[25603]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:56.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[23685]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:56.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.412 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[30028]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:56.412 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.413 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.413 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:56.844 DEBUG:teuthology.orchestra.run.smithi078:osd.1> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T20:10:56.846 INFO:tasks.cephadm:Deploying osd.2 on smithi078 with /dev/vg_nvme/lv_2... 2024-09-20T20:10:56.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T20:10:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:10:57.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:57.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:57.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi078 to 7520M 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T20:10:57.301 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:57 smithi146 ceph-mon[25603]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T20:10:57.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:57.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:57.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi078 to 7520M 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[30028]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.488 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi078 to 7520M 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T20:10:57.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:57 smithi078 ceph-mon[23685]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T20:10:58.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[23685]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:58.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[23685]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:10:58.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[23685]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[23685]: from='osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[23685]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[30028]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[30028]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[30028]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[30028]: from='osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:58.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-mon[30028]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:58 smithi146 ceph-mon[25603]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:10:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:58 smithi146 ceph-mon[25603]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:10:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:58 smithi146 ceph-mon[25603]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T20:10:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:58 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:58 smithi146 ceph-mon[25603]: from='osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:58 smithi146 ceph-mon[25603]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:10:58.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[37698]: 2024-09-20T20:10:58.649+0000 7f79ad66e640 -1 osd.1 0 waiting for initial osdmap 2024-09-20T20:10:58.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:10:58 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[37698]: 2024-09-20T20:10:58.652+0000 7f79a4c78640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:10:59.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[30028]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:10:59.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[30028]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T20:10:59.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[30028]: from='osd.1 ' entity='osd.1' 2024-09-20T20:10:59.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[23685]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:10:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[23685]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T20:10:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[23685]: from='osd.1 ' entity='osd.1' 2024-09-20T20:10:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:10:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:10:59.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2024-09-20T20:10:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:59 smithi146 ceph-mon[25603]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:10:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:59 smithi146 ceph-mon[25603]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T20:10:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:59 smithi146 ceph-mon[25603]: from='osd.1 ' entity='osd.1' 2024-09-20T20:10:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:10:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:10:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:11:00.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:11:00.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[23685]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[23685]: osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671] boot 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[23685]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[30028]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[30028]: osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671] boot 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[30028]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T20:11:00.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:11:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:00 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:11:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:00 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:11:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:00 smithi146 ceph-mon[25603]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T20:11:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:00 smithi146 ceph-mon[25603]: osd.1 [v2:172.21.15.78:6810/1479333671,v1:172.21.15.78:6811/1479333671] boot 2024-09-20T20:11:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:00 smithi146 ceph-mon[25603]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T20:11:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:11:01.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[30028]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T20:11:01.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[30028]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:01.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:01.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:01.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:01.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[23685]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T20:11:01.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[23685]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:01.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:01.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:01.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:01 smithi146 ceph-mon[25603]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T20:11:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:01 smithi146 ceph-mon[25603]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:02 smithi146 ceph-mon[25603]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:02 smithi078 ceph-mon[30028]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:02 smithi078 ceph-mon[23685]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:03.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:03 smithi078 ceph-mon[23685]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:03.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:03 smithi078 ceph-mon[30028]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:03 smithi146 ceph-mon[25603]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi078 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi078 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:05.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:05.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi078 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:05.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:05.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:06.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:06 smithi078 ceph-mon[23685]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:06.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:06 smithi078 ceph-mon[30028]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:06 smithi146 ceph-mon[25603]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:07 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1076245623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ca1720b-6579-46ec-9813-ad56a23aa8e2"}]: dispatch 2024-09-20T20:11:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:07 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1076245623' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ca1720b-6579-46ec-9813-ad56a23aa8e2"}]': finished 2024-09-20T20:11:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:07 smithi146 ceph-mon[25603]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T20:11:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1076245623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ca1720b-6579-46ec-9813-ad56a23aa8e2"}]: dispatch 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1076245623' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ca1720b-6579-46ec-9813-ad56a23aa8e2"}]': finished 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[30028]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1076245623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ca1720b-6579-46ec-9813-ad56a23aa8e2"}]: dispatch 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1076245623' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ca1720b-6579-46ec-9813-ad56a23aa8e2"}]': finished 2024-09-20T20:11:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[23685]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T20:11:07.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:08 smithi146 ceph-mon[25603]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:08 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/298455841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:08 smithi078 ceph-mon[30028]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:08 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/298455841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:08 smithi078 ceph-mon[23685]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:08 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/298455841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:10.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:10 smithi078 ceph-mon[23685]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:10.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:10 smithi078 ceph-mon[30028]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:10 smithi146 ceph-mon[25603]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:12.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:12 smithi078 ceph-mon[23685]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:12.354 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:12 smithi078 ceph-mon[30028]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:12 smithi146 ceph-mon[25603]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:13.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T20:11:13.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:13.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T20:11:13.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T20:11:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:14.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:14 smithi078 ceph-mon[23685]: Deploying daemon osd.2 on smithi078 2024-09-20T20:11:14.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:14 smithi078 ceph-mon[23685]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:14.365 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:14 smithi078 ceph-mon[30028]: Deploying daemon osd.2 on smithi078 2024-09-20T20:11:14.365 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:14 smithi078 ceph-mon[30028]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:14 smithi146 ceph-mon[25603]: Deploying daemon osd.2 on smithi078 2024-09-20T20:11:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:14 smithi146 ceph-mon[25603]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[30028]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[23685]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:16 smithi146 ceph-mon[25603]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:16.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:16.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:18.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:18.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:18.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:18.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:20.319 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:20 smithi078 ceph-mon[23685]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:20.319 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:20 smithi078 ceph-mon[30028]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:20.351 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 2 on host 'smithi078' 2024-09-20T20:11:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:20 smithi146 ceph-mon[25603]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:20.977 DEBUG:teuthology.orchestra.run.smithi078:osd.2> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T20:11:20.979 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_1... 2024-09-20T20:11:20.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T20:11:21.213 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:11:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:21 smithi146 ceph-mon[25603]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T20:11:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[23685]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:21.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:21 smithi078 ceph-mon[30028]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T20:11:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:22 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi078 to 5013M 2024-09-20T20:11:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:22 smithi146 ceph-mon[25603]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:22 smithi146 ceph-mon[25603]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T20:11:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:22 smithi146 ceph-mon[25603]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T20:11:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:22 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:22 smithi146 ceph-mon[25603]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi078 to 5013M 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[30028]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[30028]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[30028]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[30028]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi078 to 5013M 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[23685]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[23685]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[23685]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T20:11:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:22.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-mon[23685]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:11:22.842 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[41209]: 2024-09-20T20:11:22.827+0000 7f4e9237c640 -1 osd.2 0 waiting for initial osdmap 2024-09-20T20:11:22.843 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[41209]: 2024-09-20T20:11:22.831+0000 7f4e8a19d640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:11:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[23685]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:11:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[23685]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T20:11:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[23685]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' 2024-09-20T20:11:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[23685]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:23.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[30028]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:11:23.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[30028]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T20:11:23.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:23.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:23.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[30028]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' 2024-09-20T20:11:23.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:23 smithi078 ceph-mon[30028]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:23 smithi146 ceph-mon[25603]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:11:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:23 smithi146 ceph-mon[25603]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T20:11:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:23 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:23 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:23 smithi146 ceph-mon[25603]: from='osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420]' entity='osd.2' 2024-09-20T20:11:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:23 smithi146 ceph-mon[25603]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T20:11:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:24 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:11:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:24 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:11:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:24 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:24 smithi146 ceph-mon[25603]: osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420] boot 2024-09-20T20:11:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:24 smithi146 ceph-mon[25603]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T20:11:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:24 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:11:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:11:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[23685]: osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420] boot 2024-09-20T20:11:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[23685]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T20:11:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:24.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:11:24.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:11:24.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:24.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[30028]: osd.2 [v2:172.21.15.78:6818/1690980420,v1:172.21.15.78:6819/1690980420] boot 2024-09-20T20:11:24.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[30028]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T20:11:24.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:24 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:11:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:11:25.185 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2024-09-20T20:11:25.723 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:25 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:25 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[23685]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[23685]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T20:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[23685]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T20:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T20:11:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[30028]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[30028]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:11:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T20:11:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[30028]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T20:11:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T20:11:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:26 smithi146 ceph-mon[25603]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:26 smithi146 ceph-mon[25603]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:11:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T20:11:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:26 smithi146 ceph-mon[25603]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T20:11:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T20:11:27.773 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T20:11:27.773 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:27.773 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42768]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:27.773 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42768]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:27 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T20:11:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:27 smithi078 ceph-mon[23685]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T20:11:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:27 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T20:11:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:27 smithi078 ceph-mon[30028]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T20:11:28.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T20:11:28.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:28.037 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42775]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:28.037 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:11:27 smithi078 sudo[42775]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:27 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T20:11:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:27 smithi146 ceph-mon[25603]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T20:11:28.448 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42827]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T20:11:28.448 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42827]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:28.448 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42827]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:28.448 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42827]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:28.448 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42982]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T20:11:28.703 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42982]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:28.704 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42982]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:28.704 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[42982]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:28.966 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43001]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T20:11:28.966 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43001]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:28.967 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43001]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:28.967 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43001]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:28.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi078 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi078 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:28 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43024]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T20:11:29.216 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T20:11:29.216 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:29.217 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43020]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:29.217 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:11:28 smithi078 sudo[43020]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi078 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:28 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:29.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43024]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:29.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43024]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:29.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43024]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:29.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43030]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T20:11:29.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43030]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:29.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43030]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:29.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 sudo[43030]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:29.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29188]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T20:11:29.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:29.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29188]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:29.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29188]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: Cluster is now healthy 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: Cluster is now healthy 2024-09-20T20:11:30.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:30.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:30.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:30.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:30.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:30.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29191]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29191]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29191]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 sudo[29191]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: Cluster is now healthy 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:30.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43101]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T20:11:30.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43101]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:30.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43101]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:30.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43101]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43105]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T20:11:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43105]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T20:11:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43105]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T20:11:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 sudo[43105]: pam_unix(sudo:session): session closed for user root 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3792643887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6c960e5-7b68-4454-b8a9-0534dd820ea1"}]: dispatch 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3792643887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6c960e5-7b68-4454-b8a9-0534dd820ea1"}]': finished 2024-09-20T20:11:31.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T20:11:31.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:31.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:30 smithi146 ceph-mon[25603]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3792643887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6c960e5-7b68-4454-b8a9-0534dd820ea1"}]: dispatch 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3792643887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6c960e5-7b68-4454-b8a9-0534dd820ea1"}]': finished 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[30028]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3792643887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6c960e5-7b68-4454-b8a9-0534dd820ea1"}]: dispatch 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3792643887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6c960e5-7b68-4454-b8a9-0534dd820ea1"}]': finished 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:31.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:30 smithi078 ceph-mon[23685]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T20:11:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:31 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3719539104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:31 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3719539104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:31 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3719539104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:33.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:33 smithi078 ceph-mon[23685]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:33.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:33 smithi078 ceph-mon[30028]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:33 smithi146 ceph-mon[25603]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:34.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:34 smithi078 ceph-mon[23685]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:34.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:34 smithi078 ceph-mon[30028]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:34.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:34 smithi146 ceph-mon[25603]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[23685]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:36.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:36.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:36.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:36.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[30028]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-20T20:11:36.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T20:11:36.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:36 smithi146 ceph-mon[25603]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-20T20:11:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T20:11:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:37.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:37 smithi078 ceph-mon[23685]: Deploying daemon osd.3 on smithi078 2024-09-20T20:11:37.866 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:37 smithi078 ceph-mon[30028]: Deploying daemon osd.3 on smithi078 2024-09-20T20:11:37.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:37 smithi146 ceph-mon[25603]: Deploying daemon osd.3 on smithi078 2024-09-20T20:11:38.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:38 smithi078 ceph-mon[23685]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-20T20:11:38.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:38 smithi078 ceph-mon[30028]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-20T20:11:39.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:38 smithi146 ceph-mon[25603]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-20T20:11:40.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:40 smithi078 ceph-mon[23685]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-20T20:11:40.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:40 smithi078 ceph-mon[30028]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-20T20:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:40 smithi146 ceph-mon[25603]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:41 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:41 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:41 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:41 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:41 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:41 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:41 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:42.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:42 smithi078 ceph-mon[30028]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 786 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-20T20:11:42.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:42 smithi078 ceph-mon[23685]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 786 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-20T20:11:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:42 smithi146 ceph-mon[25603]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 786 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-20T20:11:43.685 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2024-09-20T20:11:44.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:44.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:44.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:44.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:43 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:11:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:11:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:43 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:44.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:44 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi078 to 3760M 2024-09-20T20:11:44.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:44 smithi078 ceph-mon[30028]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T20:11:44.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:44 smithi078 ceph-mon[30028]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-20T20:11:44.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:44 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi078 to 3760M 2024-09-20T20:11:44.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:44 smithi078 ceph-mon[23685]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T20:11:44.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:44 smithi078 ceph-mon[23685]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-20T20:11:44.927 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T20:11:44.929 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_4... 2024-09-20T20:11:44.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T20:11:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:44 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi078 to 3760M 2024-09-20T20:11:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:44 smithi146 ceph-mon[25603]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T20:11:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:44 smithi146 ceph-mon[25603]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-20T20:11:45.137 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:11:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:45 smithi146 ceph-mon[25603]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T20:11:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:45 smithi146 ceph-mon[25603]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T20:11:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:45 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:45 smithi146 ceph-mon[25603]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:11:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[23685]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T20:11:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[23685]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T20:11:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[23685]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:11:46.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[30028]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T20:11:46.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[30028]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T20:11:46.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:45 smithi078 ceph-mon[30028]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-20T20:11:46.507 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[44763]: 2024-09-20T20:11:46.195+0000 7fb32d32e640 -1 osd.3 0 waiting for initial osdmap 2024-09-20T20:11:46.507 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[44763]: 2024-09-20T20:11:46.199+0000 7fb324938640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:11:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[23685]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:11:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[23685]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T20:11:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[23685]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' 2024-09-20T20:11:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[23685]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[30028]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:11:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[30028]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T20:11:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[30028]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' 2024-09-20T20:11:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:46 smithi078 ceph-mon[30028]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:46 smithi146 ceph-mon[25603]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-20T20:11:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:46 smithi146 ceph-mon[25603]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T20:11:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:46 smithi146 ceph-mon[25603]: from='osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434]' entity='osd.3' 2024-09-20T20:11:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:46 smithi146 ceph-mon[25603]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T20:11:47.377 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:11:47.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-09-20T20:11:47.596 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434] boot 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:11:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:11:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434] boot 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: osd.3 [v2:172.21.15.78:6826/847381434,v1:172.21.15.78:6827/847381434] boot 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:11:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:48.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:48 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi078 2024-09-20T20:11:48.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:48 smithi146 ceph-mon[25603]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:48 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi078 2024-09-20T20:11:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:48 smithi078 ceph-mon[23685]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:48 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi078 2024-09-20T20:11:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:48 smithi078 ceph-mon[30028]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:49 smithi146 ceph-mon[25603]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T20:11:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:49 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:50.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:49 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:50.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:49 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[23685]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[30028]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:11:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:49 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:50.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:50 smithi146 ceph-mon[25603]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:50.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:50 smithi146 ceph-mon[25603]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:50 smithi078 ceph-mon[23685]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:50 smithi078 ceph-mon[23685]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:50 smithi078 ceph-mon[30028]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:11:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:50 smithi078 ceph-mon[30028]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:51 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/4030155048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]: dispatch 2024-09-20T20:11:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:51 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]: dispatch 2024-09-20T20:11:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:51 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]': finished 2024-09-20T20:11:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:51 smithi146 ceph-mon[25603]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T20:11:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/4030155048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]: dispatch 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]: dispatch 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]': finished 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[23685]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/4030155048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]: dispatch 2024-09-20T20:11:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]: dispatch 2024-09-20T20:11:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57f45a84-12c0-41ff-8a83-53eb19266ea7"}]': finished 2024-09-20T20:11:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[30028]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T20:11:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:11:53.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:52 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/3038679900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:53.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:52 smithi146 ceph-mon[25603]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:52 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/3038679900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:52 smithi078 ceph-mon[23685]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:52 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/3038679900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:11:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:52 smithi078 ceph-mon[30028]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:54 smithi146 ceph-mon[25603]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:54 smithi078 ceph-mon[23685]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:54 smithi078 ceph-mon[30028]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:56.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:56.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:56.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:56.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:56 smithi146 ceph-mon[25603]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[23685]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:11:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:56 smithi078 ceph-mon[30028]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:58 smithi146 ceph-mon[25603]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:58 smithi078 ceph-mon[23685]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:58 smithi078 ceph-mon[30028]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:11:59.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T20:11:59.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:59 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:59.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:11:59 smithi146 ceph-mon[25603]: Deploying daemon osd.4 on smithi146 2024-09-20T20:11:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T20:11:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:59 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:11:59 smithi078 ceph-mon[23685]: Deploying daemon osd.4 on smithi146 2024-09-20T20:11:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T20:11:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:59 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:11:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:11:59 smithi078 ceph-mon[30028]: Deploying daemon osd.4 on smithi146 2024-09-20T20:12:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:00 smithi146 ceph-mon[25603]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:00 smithi078 ceph-mon[23685]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:00 smithi078 ceph-mon[30028]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:02.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:02 smithi146 ceph-mon[25603]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:02 smithi078 ceph-mon[23685]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:02 smithi078 ceph-mon[30028]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:04.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:04 smithi146 ceph-mon[25603]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:04 smithi078 ceph-mon[23685]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:04 smithi078 ceph-mon[30028]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:05.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:05.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:05.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:05.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:05.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:05.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:05.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:05 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:05 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:06 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:06 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:06 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:06 smithi146 ceph-mon[25603]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[23685]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:06.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:06 smithi078 ceph-mon[30028]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:07.221 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2024-09-20T20:12:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi146 to 16064M 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:07 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.807 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T20:12:07.809 INFO:tasks.cephadm:Deploying osd.5 on smithi146 with /dev/vg_nvme/lv_3... 2024-09-20T20:12:07.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi146 to 16064M 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:07.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi146 to 16064M 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:07.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:07 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:08.039 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:12:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:08 smithi146 ceph-mon[25603]: from='osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T20:12:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:08 smithi146 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T20:12:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:08 smithi146 ceph-mon[25603]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:08 smithi078 ceph-mon[30028]: from='osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T20:12:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:08 smithi078 ceph-mon[30028]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T20:12:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:08 smithi078 ceph-mon[30028]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:08 smithi078 ceph-mon[23685]: from='osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T20:12:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:08 smithi078 ceph-mon[23685]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T20:12:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:08 smithi078 ceph-mon[23685]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:09.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:09 smithi146 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:12:09.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:09 smithi146 ceph-mon[25603]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T20:12:09.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:09 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:09.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:09 smithi146 ceph-mon[25603]: from='osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:09.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:09 smithi146 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[23685]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[23685]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[23685]: from='osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[23685]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[30028]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:12:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[30028]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T20:12:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[30028]: from='osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:09 smithi078 ceph-mon[30028]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:10.261 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[31825]: 2024-09-20T20:12:10.055+0000 7fe29c77e640 -1 osd.4 0 waiting for initial osdmap 2024-09-20T20:12:10.261 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[31825]: 2024-09-20T20:12:10.059+0000 7fe29459f640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:12:10.634 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:12:10.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-09-20T20:12:10.854 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:12:10.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[23685]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:10.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[23685]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[23685]: from='osd.4 ' entity='osd.4' 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[23685]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[30028]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[30028]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[30028]: from='osd.4 ' entity='osd.4' 2024-09-20T20:12:10.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:10 smithi078 ceph-mon[30028]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:10.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:10.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-mon[25603]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T20:12:10.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:10.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-mon[25603]: from='osd.4 ' entity='osd.4' 2024-09-20T20:12:10.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:10 smithi146 ceph-mon[25603]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T20:12:11.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:11 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:12:11.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:11 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:12:11.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:11 smithi146 ceph-mon[25603]: osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729] boot 2024-09-20T20:12:11.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:11 smithi146 ceph-mon[25603]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T20:12:11.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:11 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[23685]: osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729] boot 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[23685]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:12:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:12:12.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[30028]: osd.4 [v2:172.21.15.146:6800/1760773729,v1:172.21.15.146:6801/1760773729] boot 2024-09-20T20:12:12.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[30028]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T20:12:12.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:11 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:12:12.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:12 smithi146 ceph-mon[25603]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T20:12:12.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:12 smithi146 ceph-mon[25603]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:12 smithi078 ceph-mon[23685]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T20:12:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:12 smithi078 ceph-mon[23685]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:12 smithi078 ceph-mon[30028]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T20:12:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:12 smithi078 ceph-mon[30028]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:13.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:13 smithi146 ceph-mon[25603]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T20:12:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:13 smithi146 ceph-mon[25603]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:12:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:12:13.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:13 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[23685]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[23685]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[30028]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T20:12:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[30028]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:12:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:12:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:13 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:14 smithi146 ceph-mon[25603]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T20:12:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:14 smithi146 ceph-mon[25603]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:14 smithi078 ceph-mon[23685]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T20:12:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:14 smithi078 ceph-mon[23685]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:14 smithi078 ceph-mon[30028]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T20:12:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:14 smithi078 ceph-mon[30028]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/998187375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]: dispatch 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]: dispatch 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]': finished 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:15 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/1125718484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/998187375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]: dispatch 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]: dispatch 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]': finished 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/1125718484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/998187375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]: dispatch 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]: dispatch 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "880306b7-38a4-4e4b-9019-cd6e2b24c069"}]': finished 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:16.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:15 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/1125718484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi146 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:16 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi146 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi146 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:17.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:16 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:17 smithi146 ceph-mon[25603]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-09-20T20:12:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:17 smithi078 ceph-mon[23685]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-09-20T20:12:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:17 smithi078 ceph-mon[30028]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-09-20T20:12:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:19 smithi146 ceph-mon[25603]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T20:12:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:19 smithi078 ceph-mon[23685]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T20:12:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:19 smithi078 ceph-mon[30028]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T20:12:22.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:21 smithi146 ceph-mon[25603]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T20:12:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:21 smithi078 ceph-mon[23685]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T20:12:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:21 smithi078 ceph-mon[30028]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T20:12:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:23 smithi146 ceph-mon[25603]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T20:12:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:23 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T20:12:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:23 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:23 smithi146 ceph-mon[25603]: Deploying daemon osd.5 on smithi146 2024-09-20T20:12:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[23685]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T20:12:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T20:12:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[23685]: Deploying daemon osd.5 on smithi146 2024-09-20T20:12:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[30028]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T20:12:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T20:12:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:23 smithi078 ceph-mon[30028]: Deploying daemon osd.5 on smithi146 2024-09-20T20:12:26.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:26 smithi146 ceph-mon[25603]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-20T20:12:26.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[23685]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-20T20:12:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[30028]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-20T20:12:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:28 smithi146 ceph-mon[25603]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:28 smithi078 ceph-mon[23685]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:28 smithi078 ceph-mon[30028]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:29.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:29.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:29.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:29.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:29 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:29 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:30 smithi146 ceph-mon[25603]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:30 smithi078 ceph-mon[23685]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:30 smithi078 ceph-mon[30028]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:30.933 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 5 on host 'smithi146' 2024-09-20T20:12:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:31 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:31 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:31.627 DEBUG:teuthology.orchestra.run.smithi146:osd.5> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T20:12:31.629 INFO:tasks.cephadm:Deploying osd.6 on smithi146 with /dev/vg_nvme/lv_2... 2024-09-20T20:12:31.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T20:12:31.861 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:12:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:32 smithi146 ceph-mon[25603]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:32 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi146 to 8032M 2024-09-20T20:12:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:32 smithi146 ceph-mon[25603]: from='osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T20:12:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:32 smithi146 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T20:12:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[23685]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi146 to 8032M 2024-09-20T20:12:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[23685]: from='osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T20:12:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[23685]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T20:12:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[30028]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi146 to 8032M 2024-09-20T20:12:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[30028]: from='osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T20:12:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:32 smithi078 ceph-mon[30028]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[23685]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[23685]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[23685]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[23685]: from='osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[23685]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[30028]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[30028]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:12:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[30028]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T20:12:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[30028]: from='osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:33 smithi078 ceph-mon[30028]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:33.557 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-mon[25603]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:33.557 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:12:33.557 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-mon[25603]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T20:12:33.557 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:33.557 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-mon[25603]: from='osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:33.557 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:33.846 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[35277]: 2024-09-20T20:12:33.723+0000 7f485eb28640 -1 osd.5 0 waiting for initial osdmap 2024-09-20T20:12:33.846 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:12:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[35277]: 2024-09-20T20:12:33.729+0000 7f4856949640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:12:34.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:34 smithi146 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:34.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:34 smithi146 ceph-mon[25603]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T20:12:34.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:34.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:34.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:34 smithi146 ceph-mon[25603]: from='osd.5 ' entity='osd.5' 2024-09-20T20:12:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:12:34.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[23685]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[23685]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[23685]: from='osd.5 ' entity='osd.5' 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[30028]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[30028]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T20:12:34.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:34.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:34.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:34 smithi078 ceph-mon[30028]: from='osd.5 ' entity='osd.5' 2024-09-20T20:12:34.695 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261] boot 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:35 smithi146 ceph-mon[25603]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261] boot 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[23685]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T20:12:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:12:35.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:12:35.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T20:12:35.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: osd.5 [v2:172.21.15.146:6808/4165836261,v1:172.21.15.146:6809/4165836261] boot 2024-09-20T20:12:35.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T20:12:35.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:12:35.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:35 smithi078 ceph-mon[30028]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T20:12:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T20:12:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:12:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:12:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:36 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:12:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:12:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:36 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:37 smithi146 ceph-mon[25603]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:37 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi146 2024-09-20T20:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:37 smithi146 ceph-mon[25603]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:37 smithi146 ceph-mon[25603]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:37 smithi146 ceph-mon[25603]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:37 smithi146 ceph-mon[25603]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T20:12:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[23685]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi146 2024-09-20T20:12:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[23685]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[23685]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[23685]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:12:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[23685]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T20:12:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[30028]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi146 2024-09-20T20:12:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[30028]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[30028]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:12:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[30028]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:12:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:37 smithi078 ceph-mon[30028]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T20:12:38.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:38 smithi146 ceph-mon[25603]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T20:12:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:38 smithi078 ceph-mon[23685]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T20:12:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:38 smithi078 ceph-mon[30028]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T20:12:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:39 smithi146 ceph-mon[25603]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:39 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/3949237698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]: dispatch 2024-09-20T20:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:39 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]: dispatch 2024-09-20T20:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:39 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]': finished 2024-09-20T20:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:39 smithi146 ceph-mon[25603]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T20:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:39 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[23685]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/3949237698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]: dispatch 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]: dispatch 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]': finished 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[23685]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[30028]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/3949237698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]: dispatch 2024-09-20T20:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]: dispatch 2024-09-20T20:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a5dd93d-ae84-41fd-836a-3c3dd96da843"}]': finished 2024-09-20T20:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[30028]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T20:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:39 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/892760496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:12:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/892760496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:12:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:40 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/892760496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:40 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:41 smithi078 ceph-mon[30028]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:41 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:12:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:41 smithi078 ceph-mon[30028]: Cluster is now healthy 2024-09-20T20:12:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:41 smithi078 ceph-mon[23685]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:41 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:12:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:41 smithi078 ceph-mon[23685]: Cluster is now healthy 2024-09-20T20:12:41.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:41 smithi146 ceph-mon[25603]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:41.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:41 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:12:41.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:41 smithi146 ceph-mon[25603]: Cluster is now healthy 2024-09-20T20:12:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:43 smithi078 ceph-mon[23685]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:43 smithi078 ceph-mon[30028]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:43.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:43 smithi146 ceph-mon[25603]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:45.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:45 smithi146 ceph-mon[25603]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:45 smithi078 ceph-mon[23685]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:45 smithi078 ceph-mon[30028]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:46.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:46.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:46.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:46.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T20:12:46.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:46 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T20:12:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T20:12:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:46 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:48.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:47 smithi146 ceph-mon[25603]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:48.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:47 smithi146 ceph-mon[25603]: Deploying daemon osd.6 on smithi146 2024-09-20T20:12:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:47 smithi078 ceph-mon[23685]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:47 smithi078 ceph-mon[23685]: Deploying daemon osd.6 on smithi146 2024-09-20T20:12:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:47 smithi078 ceph-mon[30028]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:47 smithi078 ceph-mon[30028]: Deploying daemon osd.6 on smithi146 2024-09-20T20:12:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:49 smithi146 ceph-mon[25603]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:49 smithi078 ceph-mon[23685]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:49 smithi078 ceph-mon[30028]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:51.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:51.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:51.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:51.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:51.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:51.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:51.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:51 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:51 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:53 smithi146 ceph-mon[25603]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:54.051 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 6 on host 'smithi146' 2024-09-20T20:12:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:53 smithi078 ceph-mon[23685]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:53 smithi078 ceph-mon[30028]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:54.783 DEBUG:teuthology.orchestra.run.smithi146:osd.6> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T20:12:54.785 INFO:tasks.cephadm:Deploying osd.7 on smithi146 with /dev/vg_nvme/lv_1... 2024-09-20T20:12:54.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi146 to 5354M 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T20:12:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:54 smithi146 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T20:12:55.020 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:12:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi146 to 5354M 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[23685]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:12:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi146 to 5354M 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T20:12:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:54 smithi078 ceph-mon[30028]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T20:12:56.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-mon[25603]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:56.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:12:56.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-mon[25603]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T20:12:56.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:56.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-mon[25603]: from='osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:56.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:56.177 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[38746]: 2024-09-20T20:12:56.084+0000 7fe14bfcb640 -1 osd.6 0 waiting for initial osdmap 2024-09-20T20:12:56.177 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:12:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[38746]: 2024-09-20T20:12:56.090+0000 7fe147df4640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:12:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[23685]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[23685]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:12:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[23685]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T20:12:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[23685]: from='osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[23685]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[30028]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[30028]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[30028]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[30028]: from='osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:56 smithi078 ceph-mon[30028]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:12:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:57 smithi146 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:57 smithi146 ceph-mon[25603]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T20:12:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:57 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:57 smithi146 ceph-mon[25603]: osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825] boot 2024-09-20T20:12:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:57 smithi146 ceph-mon[25603]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T20:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[23685]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[23685]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T20:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[23685]: osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825] boot 2024-09-20T20:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[23685]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T20:12:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[30028]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:12:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[30028]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T20:12:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[30028]: osd.6 [v2:172.21.15.146:6816/1470150825,v1:172.21.15.146:6817/1470150825] boot 2024-09-20T20:12:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:57 smithi078 ceph-mon[30028]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T20:12:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-20T20:12:57.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2024-09-20T20:12:57.834 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:12:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:58 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:12:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:58 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:12:58.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:58 smithi146 ceph-mon[25603]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:58.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:58 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:58.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:12:58 smithi146 ceph-mon[25603]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T20:12:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:12:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:12:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[23685]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:58.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:58.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[23685]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T20:12:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:12:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:12:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[30028]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T20:12:58.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:12:58.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:12:58 smithi078 ceph-mon[30028]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T20:13:00.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:00 smithi146 ceph-mon[25603]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:00.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:00 smithi146 ceph-mon[25603]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T20:13:00.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:00 smithi146 ceph-mon[25603]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T20:13:00.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:13:00.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:13:00.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:00 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[23685]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[23685]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T20:13:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[23685]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T20:13:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:13:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:13:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[30028]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[30028]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T20:13:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[30028]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T20:13:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T20:13:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T20:13:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:00 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:01.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:01.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:01.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:01.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:01.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:01 smithi146 ceph-mon[25603]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[30028]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:01.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:01 smithi078 ceph-mon[23685]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T20:13:02.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:02 smithi146 ceph-mon[25603]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:02.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:02 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi146 2024-09-20T20:13:02.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:02 smithi146 ceph-mon[25603]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:02.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:02 smithi146 ceph-mon[25603]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:13:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[30028]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi146 2024-09-20T20:13:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[30028]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[30028]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:13:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[23685]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:02.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi146 2024-09-20T20:13:02.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[23685]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:02.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:02 smithi078 ceph-mon[23685]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/2838727937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]: dispatch 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]: dispatch 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]': finished 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:04.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/3999921645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/2838727937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]: dispatch 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]: dispatch 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]': finished 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:04.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:04.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:04 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/3999921645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:13:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/2838727937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]: dispatch 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]: dispatch 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f13969f-4da8-4014-bad0-360aae8a5600"}]': finished 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:04 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/3999921645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T20:13:05.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:05 smithi078 ceph-mon[23685]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-09-20T20:13:05.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:05 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:13:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:05 smithi078 ceph-mon[23685]: Cluster is now healthy 2024-09-20T20:13:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:05 smithi078 ceph-mon[30028]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-09-20T20:13:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:05 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:13:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:05 smithi078 ceph-mon[30028]: Cluster is now healthy 2024-09-20T20:13:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:05 smithi146 ceph-mon[25603]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 150 KiB/s, 0 objects/s recovering 2024-09-20T20:13:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:05 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-20T20:13:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:05 smithi146 ceph-mon[25603]: Cluster is now healthy 2024-09-20T20:13:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:06 smithi146 ceph-mon[25603]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-09-20T20:13:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:06 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:06 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:06 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[23685]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-09-20T20:13:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[30028]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-09-20T20:13:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:06 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:09.158 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:08 smithi146 ceph-mon[25603]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T20:13:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:08 smithi078 ceph-mon[30028]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T20:13:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:08 smithi078 ceph-mon[23685]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T20:13:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:10 smithi146 ceph-mon[25603]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T20:13:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T20:13:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:10 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:10 smithi146 ceph-mon[25603]: Deploying daemon osd.7 on smithi146 2024-09-20T20:13:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[23685]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[23685]: Deploying daemon osd.7 on smithi146 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[30028]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:10 smithi078 ceph-mon[30028]: Deploying daemon osd.7 on smithi146 2024-09-20T20:13:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:12 smithi146 ceph-mon[25603]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:12 smithi078 ceph-mon[23685]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:12 smithi078 ceph-mon[30028]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:15.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:14 smithi146 ceph-mon[25603]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:14 smithi078 ceph-mon[23685]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:14 smithi078 ceph-mon[30028]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:15 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:15 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:17.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:16 smithi146 ceph-mon[25603]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:16 smithi078 ceph-mon[23685]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:16 smithi078 ceph-mon[30028]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 7 on host 'smithi146' 2024-09-20T20:13:18.718 DEBUG:teuthology.orchestra.run.smithi146:osd.7> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T20:13:18.743 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T20:13:18.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T20:13:18.953 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi146 to 4016M 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T20:13:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:18 smithi146 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi146 to 4016M 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[23685]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi146 to 4016M 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T20:13:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:18 smithi078 ceph-mon[30028]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T20:13:20.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[23685]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:13:20.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[23685]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T20:13:20.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:20.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[23685]: from='osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:13:20.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[23685]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:13:20.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[30028]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:13:20.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[30028]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T20:13:20.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:20.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[30028]: from='osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:13:20.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:20 smithi078 ceph-mon[30028]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:13:20.386 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:13:20.386 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-mon[25603]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T20:13:20.386 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:20.386 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-mon[25603]: from='osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:13:20.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-09-20T20:13:20.745 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:13:20.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[42208]: 2024-09-20T20:13:20.508+0000 7f50b00c4640 -1 osd.7 0 waiting for initial osdmap 2024-09-20T20:13:20.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:13:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[42208]: 2024-09-20T20:13:20.516+0000 7f50a7ee5640 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: from='osd.7 ' entity='osd.7' 2024-09-20T20:13:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:21 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3662733156' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T20:13:21.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:21.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:13:21.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T20:13:21.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: from='osd.7 ' entity='osd.7' 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3662733156' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: from='osd.7 ' entity='osd.7' 2024-09-20T20:13:21.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:21 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3662733156' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T20:13:21.419 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1726863177,"num_in_osds":8,"osd_in_since":1726863183,"num_remapped_pgs":0} 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: purged_snaps scrub starts 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: purged_snaps scrub ok 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546] boot 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:22 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T20:13:22.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: purged_snaps scrub starts 2024-09-20T20:13:22.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: purged_snaps scrub ok 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546] boot 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: purged_snaps scrub starts 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: purged_snaps scrub ok 2024-09-20T20:13:22.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: osd.7 [v2:172.21.15.146:6824/241106546,v1:172.21.15.146:6825/241106546] boot 2024-09-20T20:13:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T20:13:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:22 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:22.632 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:23.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:23 smithi078 ceph-mon[30028]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:23.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:23 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi146 2024-09-20T20:13:23.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:23 smithi078 ceph-mon[30028]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T20:13:23.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:23 smithi078 ceph-mon[23685]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:23.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:23 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi146 2024-09-20T20:13:23.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:23 smithi078 ceph-mon[23685]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T20:13:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:23 smithi146 ceph-mon[25603]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T20:13:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:23 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi146 2024-09-20T20:13:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:23 smithi146 ceph-mon[25603]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T20:13:24.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:24 smithi078 ceph-mon[23685]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T20:13:24.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:24 smithi078 ceph-mon[30028]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T20:13:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:24 smithi146 ceph-mon[25603]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T20:13:24.378 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:13:24.941 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1726863201,"num_in_osds":8,"osd_in_since":1726863183,"num_remapped_pgs":0} 2024-09-20T20:13:24.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T20:13:25.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:25 smithi078 ceph-mon[30028]: pgmap v178: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T20:13:25.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:25 smithi078 ceph-mon[30028]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T20:13:25.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:25 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2811138688' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T20:13:25.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:25 smithi078 ceph-mon[23685]: pgmap v178: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T20:13:25.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:25 smithi078 ceph-mon[23685]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T20:13:25.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:25 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2811138688' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T20:13:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:25 smithi146 ceph-mon[25603]: pgmap v178: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T20:13:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:25 smithi146 ceph-mon[25603]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T20:13:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:25 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2811138688' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T20:13:26.051 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:26.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:26 smithi078 ceph-mon[23685]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T20:13:26.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:26.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:26 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:26.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:26 smithi078 ceph-mon[30028]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T20:13:26.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:26.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:26 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:26 smithi146 ceph-mon[25603]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T20:13:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:26 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:27.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:27 smithi078 ceph-mon[30028]: pgmap v181: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T20:13:27.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:27 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:27.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:27 smithi078 ceph-mon[23685]: pgmap v181: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T20:13:27.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:27 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:27 smithi146 ceph-mon[25603]: pgmap v181: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-20T20:13:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:27 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:28.688 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:13:28.688 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":59,"fsid":"f2df6e16-778b-11ef-baf5-efdc52797490","created":"2024-09-20T20:08:20.356399+0000","modified":"2024-09-20T20:13:25.068965+0000","last_up_change":"2024-09-20T20:13:21.045468+0000","last_in_change":"2024-09-20T20:13:03.105710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T20:11:25.307285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9d0d6fc3-31fc-4889-b9e8-b088a65da5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6803","nonce":1463358775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6805","nonce":1463358775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6809","nonce":1463358775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6807","nonce":1463358775}]},"public_addr":"172.21.15.78:6803/1463358775","cluster_addr":"172.21.15.78:6805/1463358775","heartbeat_back_addr":"172.21.15.78:6809/1463358775","heartbeat_front_addr":"172.21.15.78:6807/1463358775","state":["exists","up"]},{"osd":1,"uuid":"24879284-6ef9-4428-a3f8-94aedad8a73d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6811","nonce":1479333671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6813","nonce":1479333671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6817","nonce":1479333671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6815","nonce":1479333671}]},"public_addr":"172.21.15.78:6811/1479333671","cluster_addr":"172.21.15.78:6813/1479333671","heartbeat_back_addr":"172.21.15.78:6817/1479333671","heartbeat_front_addr":"172.21.15.78:6815/1479333671","state":["exists","up"]},{"osd":2,"uuid":"7ca1720b-6579-46ec-9813-ad56a23aa8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6819","nonce":1690980420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6821","nonce":1690980420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6825","nonce":1690980420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6823","nonce":1690980420}]},"public_addr":"172.21.15.78:6819/1690980420","cluster_addr":"172.21.15.78:6821/1690980420","heartbeat_back_addr":"172.21.15.78:6825/1690980420","heartbeat_front_addr":"172.21.15.78:6823/1690980420","state":["exists","up"]},{"osd":3,"uuid":"a6c960e5-7b68-4454-b8a9-0534dd820ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6827","nonce":847381434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6829","nonce":847381434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6833","nonce":847381434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6831","nonce":847381434}]},"public_addr":"172.21.15.78:6827/847381434","cluster_addr":"172.21.15.78:6829/847381434","heartbeat_back_addr":"172.21.15.78:6833/847381434","heartbeat_front_addr":"172.21.15.78:6831/847381434","state":["exists","up"]},{"osd":4,"uuid":"57f45a84-12c0-41ff-8a83-53eb19266ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6801","nonce":1760773729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6803","nonce":1760773729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6807","nonce":1760773729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6805","nonce":1760773729}]},"public_addr":"172.21.15.146:6801/1760773729","cluster_addr":"172.21.15.146:6803/1760773729","heartbeat_back_addr":"172.21.15.146:6807/1760773729","heartbeat_front_addr":"172.21.15.146:6805/1760773729","state":["exists","up"]},{"osd":5,"uuid":"880306b7-38a4-4e4b-9019-cd6e2b24c069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6809","nonce":4165836261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6811","nonce":4165836261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6815","nonce":4165836261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6813","nonce":4165836261}]},"public_addr":"172.21.15.146:6809/4165836261","cluster_addr":"172.21.15.146:6811/4165836261","heartbeat_back_addr":"172.21.15.146:6815/4165836261","heartbeat_front_addr":"172.21.15.146:6813/4165836261","state":["exists","up"]},{"osd":6,"uuid":"9a5dd93d-ae84-41fd-836a-3c3dd96da843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6817","nonce":1470150825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6819","nonce":1470150825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6823","nonce":1470150825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6821","nonce":1470150825}]},"public_addr":"172.21.15.146:6817/1470150825","cluster_addr":"172.21.15.146:6819/1470150825","heartbeat_back_addr":"172.21.15.146:6823/1470150825","heartbeat_front_addr":"172.21.15.146:6821/1470150825","state":["exists","up"]},{"osd":7,"uuid":"0f13969f-4da8-4014-bad0-360aae8a5600","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6825","nonce":241106546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6827","nonce":241106546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6831","nonce":241106546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6829","nonce":241106546}]},"public_addr":"172.21.15.146:6825/241106546","cluster_addr":"172.21.15.146:6827/241106546","heartbeat_back_addr":"172.21.15.146:6831/241106546","heartbeat_front_addr":"172.21.15.146:6829/241106546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:10:35.573954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:10:57.762262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:11:21.846854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:11:45.190059+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:08.674507+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:32.376924+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:55.376155+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:13:19.360557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2330055755":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6801/674566149":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/3702369749":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/3374519687":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/1372816629":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/4123000022":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6801/2188634654":"2024-09-21T20:08:50.399277+0000","172.21.15.78:6800/2188634654":"2024-09-21T20:08:50.399277+0000","172.21.15.78:6800/674566149":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/2835130118":"2024-09-21T20:08:50.399277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T20:13:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:29 smithi078 ceph-mon[23685]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-20T20:13:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:29 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2108790476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:13:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:29 smithi078 ceph-mon[30028]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-20T20:13:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:29 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2108790476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:13:29.344 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T20:11:25.307285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-20T20:13:29.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T20:13:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:29 smithi146 ceph-mon[25603]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-09-20T20:13:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:29 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2108790476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:13:29.539 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:31.213 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-09-20T20:13:31.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:31 smithi078 ceph-mon[23685]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-20T20:13:31.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:31 smithi078 ceph-mon[30028]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-20T20:13:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:31 smithi146 ceph-mon[25603]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-20T20:13:31.859 INFO:tasks.cephadm:Adding prometheus.a on smithi146 2024-09-20T20:13:31.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch apply prometheus '1;smithi146=a' 2024-09-20T20:13:32.070 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:13:32.191 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:32 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2633023515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T20:13:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:32 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2633023515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T20:13:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:32 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2633023515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T20:13:33.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:33 smithi146 ceph-mon[25603]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:33 smithi078 ceph-mon[23685]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:33 smithi078 ceph-mon[30028]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:33.842 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled prometheus update... 2024-09-20T20:13:34.428 DEBUG:teuthology.orchestra.run.smithi146:prometheus.a> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service 2024-09-20T20:13:34.431 INFO:tasks.cephadm:Adding node-exporter.a on smithi078 2024-09-20T20:13:34.431 INFO:tasks.cephadm:Adding node-exporter.b on smithi146 2024-09-20T20:13:34.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch apply node-exporter '2;smithi078=a;smithi146=b' 2024-09-20T20:13:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: Saving service prometheus spec with placement smithi146=a;count:1 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:35.128 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:35.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:34 smithi146 ceph-mon[25603]: Deploying daemon prometheus.a on smithi146 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: Saving service prometheus spec with placement smithi146=a;count:1 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[23685]: Deploying daemon prometheus.a on smithi146 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: Saving service prometheus spec with placement smithi146=a;count:1 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:34 smithi078 ceph-mon[30028]: Deploying daemon prometheus.a on smithi146 2024-09-20T20:13:36.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:36 smithi146 ceph-mon[25603]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:37.116 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled node-exporter update... 2024-09-20T20:13:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:36 smithi078 ceph-mon[23685]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:36 smithi078 ceph-mon[30028]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:38 smithi146 ceph-mon[25603]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi078=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:38 smithi146 ceph-mon[25603]: Saving service node-exporter spec with placement smithi078=a;smithi146=b;count:2 2024-09-20T20:13:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:38 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:38 smithi078 ceph-mon[23685]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi078=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:38 smithi078 ceph-mon[23685]: Saving service node-exporter spec with placement smithi078=a;smithi146=b;count:2 2024-09-20T20:13:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:38 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:38 smithi078 ceph-mon[30028]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi078=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:38 smithi078 ceph-mon[30028]: Saving service node-exporter spec with placement smithi078=a;smithi146=b;count:2 2024-09-20T20:13:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:38 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:38.567 DEBUG:teuthology.orchestra.run.smithi078:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@node-exporter.a.service 2024-09-20T20:13:38.571 DEBUG:teuthology.orchestra.run.smithi146:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@node-exporter.b.service 2024-09-20T20:13:38.573 INFO:tasks.cephadm:Adding alertmanager.a on smithi078 2024-09-20T20:13:38.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch apply alertmanager '1;smithi078=a' 2024-09-20T20:13:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:13:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:39 smithi146 ceph-mon[25603]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:39 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:39 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:39 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[23685]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[30028]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:39.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:39 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:41.189 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled alertmanager update... 2024-09-20T20:13:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:41 smithi146 ceph-mon[25603]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:41 smithi078 ceph-mon[30028]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:41 smithi078 ceph-mon[23685]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:42.353 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:42 smithi146 ceph-mon[25603]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi078=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:42.353 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:42 smithi146 ceph-mon[25603]: Saving service alertmanager spec with placement smithi078=a;count:1 2024-09-20T20:13:42.353 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:42 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:42 smithi078 ceph-mon[23685]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi078=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:42 smithi078 ceph-mon[23685]: Saving service alertmanager spec with placement smithi078=a;count:1 2024-09-20T20:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:42 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:42 smithi078 ceph-mon[30028]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi078=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:42 smithi078 ceph-mon[30028]: Saving service alertmanager spec with placement smithi078=a;count:1 2024-09-20T20:13:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:42 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:43.262 DEBUG:teuthology.orchestra.run.smithi078:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@alertmanager.a.service 2024-09-20T20:13:43.265 INFO:tasks.cephadm:Adding grafana.a on smithi146 2024-09-20T20:13:43.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph orch apply grafana '1;smithi146=a' 2024-09-20T20:13:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:43 smithi146 ceph-mon[25603]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:43 smithi078 ceph-mon[23685]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:43 smithi078 ceph-mon[30028]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:44.367 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:13:45.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:45 smithi146 ceph-mon[25603]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:45 smithi078 ceph-mon[23685]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:45 smithi078 ceph-mon[30028]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:47 smithi146 ceph-mon[25603]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:47 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[23685]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[30028]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:47.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:47 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:49.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:49 smithi146 ceph-mon[25603]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:49.480 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled grafana update... 2024-09-20T20:13:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:49 smithi078 ceph-mon[23685]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:49 smithi078 ceph-mon[30028]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:50.145 DEBUG:teuthology.orchestra.run.smithi146:grafana.a> sudo journalctl -f -n 0 -u ceph-f2df6e16-778b-11ef-baf5-efdc52797490@grafana.a.service 2024-09-20T20:13:50.148 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T20:13:50.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T20:13:50.331 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:13:50.299Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T20:13:50.331 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:13:50 smithi146 bash[44678]: f480cbdaafe9bbea9067ffe5d842bfbee64f7928777fd0d1e611becaa8f53b9d 2024-09-20T20:13:50.358 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:50.612 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:13:50 smithi146 systemd[1]: Started Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:13:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: Saving service grafana spec with placement smithi146=a;count:1 2024-09-20T20:13:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:50 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T20:13:50.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:50.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: Saving service grafana spec with placement smithi146=a;count:1 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: Saving service grafana spec with placement smithi146=a;count:1 2024-09-20T20:13:50.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' 2024-09-20T20:13:50.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:50 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T20:13:51.606 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setuser ceph since I am not root 2024-09-20T20:13:51.607 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setgroup ceph since I am not root 2024-09-20T20:13:51.607 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:51.546+0000 7f5339781040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:13:51.607 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:51.604+0000 7f5339781040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:13:51.756 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: ignoring --setuser ceph since I am not root 2024-09-20T20:13:51.756 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: ignoring --setgroup ceph since I am not root 2024-09-20T20:13:51.756 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-mgr[27813]: -- 172.21.15.146:0/1613624863 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a20accd00 con 0x555a20aa1400 2024-09-20T20:13:51.756 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:51.545+0000 7efc5f140040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:13:51.756 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:51.602+0000 7efc5f140040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:13:51.756 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:51.753+0000 7efc5f140040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:13:51.979 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:51.755+0000 7f5339781040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:13:51.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:51.824+0000 7f5339781040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:13:51.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:51.977+0000 7f5339781040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:13:52.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:51.820+0000 7efc5f140040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:13:52.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:51.965+0000 7efc5f140040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:13:52.090 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-09-20T20:13:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCA1+1mv97xBBAAg4hvDUd92wvaC0Shfi/VKg== 2024-09-20T20:13:52.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:52.129+0000 7f5339781040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:13:52.407 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:52.106+0000 7efc5f140040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:13:52.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[30028]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T20:13:52.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[30028]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1486696611' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:52.453+0000 7f5339781040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:52.515+0000 7f5339781040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[23685]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[23685]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1486696611' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:52.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:52.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-mon[25603]: from='mgr.14150 172.21.15.78:0/920549535' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-mon[25603]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1486696611' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:52.411+0000 7efc5f140040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:52.472+0000 7efc5f140040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:13:52.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:52 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:52.652+0000 7efc5f140040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:13:52.815 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-20T20:13:52.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T20:13:52.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T20:13:52.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T20:13:53.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:52.705+0000 7f5339781040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:13:53.063 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T20:13:53.766 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:53 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:53.562+0000 7efc5f140040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:13:53.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:53 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:53.628+0000 7efc5f140040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:13:54.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:53.582+0000 7f5339781040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:13:54.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:53.646+0000 7f5339781040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:13:54.482 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.358+0000 7efc5f140040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:13:54.482 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.480+0000 7efc5f140040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:13:54.607 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.348+0000 7f5339781040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:13:54.607 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.463+0000 7f5339781040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:13:54.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.625+0000 7efc5f140040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:13:54.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.689+0000 7efc5f140040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:13:54.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.604+0000 7f5339781040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:13:54.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.665+0000 7f5339781040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:13:54.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.813+0000 7f5339781040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:13:54.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.881+0000 7f5339781040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:13:55.104 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-09-20T20:13:55.104 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCD1+1mvf7XBRAAMtinXHE+mBrNWEgBVB6LOQ== 2024-09-20T20:13:55.104 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.842+0000 7efc5f140040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:13:55.104 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.916+0000 7efc5f140040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:13:55.104 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:54.976+0000 7efc5f140040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:13:55.153 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:54.941+0000 7f5339781040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:13:55.153 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:55.096+0000 7f5339781040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:13:55.357 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-mon[25603]: from='client.? 172.21.15.146:0/3579852906' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:55.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:55.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T20:13:55.358 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:55.139+0000 7efc5f140040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:13:55.358 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:55.205+0000 7efc5f140040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:13:55.358 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:55.270+0000 7efc5f140040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:13:55.358 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:55.355+0000 7efc5f140040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:13:55.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-mon[30028]: from='client.? 172.21.15.146:0/3579852906' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-mon[23685]: from='client.? 172.21.15.146:0/3579852906' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:55.162+0000 7f5339781040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:13:55.428 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:55.227+0000 7f5339781040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:13:55.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:55.309+0000 7f5339781040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:13:55.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:55.367+0000 7f5339781040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:13:55.429 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:55 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:55.425+0000 7f5339781040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:13:55.757 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:55.416+0000 7efc5f140040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:13:55.757 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:55.474+0000 7efc5f140040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:13:55.778 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-20T20:13:55.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T20:13:55.778 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T20:13:55.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-20T20:13:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:56.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: Active manager daemon y restarted 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: Activating manager daemon y 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: mgrmap e17: y(active, starting, since 0.00580294s), standbys: x 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:13:56.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: Manager daemon y is now available 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: Standby manager daemon x restarted 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: Standby manager daemon x started 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:13:56.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]': finished 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: Active manager daemon y restarted 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: Activating manager daemon y 2024-09-20T20:13:56.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: mgrmap e17: y(active, starting, since 0.00580294s), standbys: x 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:56.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: Manager daemon y is now available 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:56.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]': finished 2024-09-20T20:13:56.418 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:56.230+0000 7f5339781040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:13:56.418 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:13:56] ENGINE Bus STARTING 2024-09-20T20:13:56.418 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: CherryPy Checker: 2024-09-20T20:13:56.418 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: The Application mounted at '' has an empty config. 2024-09-20T20:13:56.418 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: Active manager daemon y restarted 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: Activating manager daemon y 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: mgrmap e17: y(active, starting, since 0.00580294s), standbys: x 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:13:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: Manager daemon y is now available 2024-09-20T20:13:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/2142803664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]': finished 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:56.258+0000 7efc5f140040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:13:56.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: [20/Sep/2024:20:13:56] ENGINE Bus STARTING 2024-09-20T20:13:56.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: CherryPy Checker: 2024-09-20T20:13:56.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: The Application mounted at '' has an empty config. 2024-09-20T20:13:56.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: 2024-09-20T20:13:56.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: [20/Sep/2024:20:13:56] ENGINE Serving on http://:::9283 2024-09-20T20:13:56.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:13:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[27809]: [20/Sep/2024:20:13:56] ENGINE Bus STARTED 2024-09-20T20:13:56.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:13:56] ENGINE Serving on http://:::9283 2024-09-20T20:13:56.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:13:56] ENGINE Bus STARTED 2024-09-20T20:13:57.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]': finished 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[23685]: mgrmap e18: y(active, since 1.0113s), standbys: x 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]': finished 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:57.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:13:57.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-mon[30028]: mgrmap e18: y(active, since 1.0113s), standbys: x 2024-09-20T20:13:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]': finished 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:57 smithi146 ceph-mon[25603]: mgrmap e18: y(active, since 1.0113s), standbys: x 2024-09-20T20:13:57.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:57.511+0000 7f5291cb0640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:20:13:57] ENGINE Error in HTTPServer.serve 2024-09-20T20:13:57.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: Traceback (most recent call last): 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._connections.run(self.expiration_interval) 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._run(expiration_interval) 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: s = self.context.wrap_socket( 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: return self.sslsocket_class._create( 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self.do_handshake() 2024-09-20T20:13:57.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:13:57.788 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: self._sslobj.do_handshake() 2024-09-20T20:13:57.788 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:13:57.788 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:13:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:13:58.151 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:13:58.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:13:58.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:13:58.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:13:58.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:13:58.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: [20/Sep/2024:20:13:57] ENGINE Bus STARTING 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: [20/Sep/2024:20:13:57] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1272861866' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: [20/Sep/2024:20:13:57] ENGINE Bus STARTING 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: [20/Sep/2024:20:13:57] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:58 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1272861866' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T20:13:58.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:13:58.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:13:58.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:13:58.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:13:58.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:58.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: [20/Sep/2024:20:13:57] ENGINE Bus STARTING 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: [20/Sep/2024:20:13:57] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:13:58.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:58 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1272861866' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-20T20:13:58.789 INFO:teuthology.orchestra.run.smithi078.stdout:[{"version":26,"timestamp":"2024-09-20T20:13:56.283551+0000","name":"","changes":[{"name":"osd/host:smithi078/osd_memory_target","previous_value":"3942653747"}]}] 2024-09-20T20:13:58.790 INFO:tasks.ceph_manager:config epoch is 26 2024-09-20T20:13:58.790 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T20:13:58.790 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T20:13:58.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T20:13:58.984 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[30028]: [20/Sep/2024:20:13:57] ENGINE Serving on https://172.21.15.78:7151 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[30028]: [20/Sep/2024:20:13:57] ENGINE Bus STARTED 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[30028]: [20/Sep/2024:20:13:57] ENGINE Error in HTTPServer.serve 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:13:59.417 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.do_handshake() 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[30028]: Deploying daemon alertmanager.a on smithi078 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[30028]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:59.418 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[30028]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[23685]: [20/Sep/2024:20:13:57] ENGINE Serving on https://172.21.15.78:7151 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[23685]: [20/Sep/2024:20:13:57] ENGINE Bus STARTED 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[23685]: [20/Sep/2024:20:13:57] ENGINE Error in HTTPServer.serve 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:13:59.419 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[23685]: Deploying daemon alertmanager.a on smithi078 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[23685]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:59.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:13:59 smithi078 ceph-mon[23685]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:59 smithi146 ceph-mon[25603]: [20/Sep/2024:20:13:57] ENGINE Serving on https://172.21.15.78:7151 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:59 smithi146 ceph-mon[25603]: [20/Sep/2024:20:13:57] ENGINE Bus STARTED 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:59 smithi146 ceph-mon[25603]: [20/Sep/2024:20:13:57] ENGINE Error in HTTPServer.serve 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:13:59.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._run(expiration_interval) 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: s = self.context.wrap_socket( 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.sslsocket_class._create( 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.do_handshake() 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:13:59.607 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._sslobj.do_handshake() 2024-09-20T20:13:59.608 INFO:journalctl@ceph.mon.b.smithi146.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:13:59.608 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:59 smithi146 ceph-mon[25603]: Deploying daemon alertmanager.a on smithi078 2024-09-20T20:13:59.608 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:59 smithi146 ceph-mon[25603]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:13:59.608 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:13:59 smithi146 ceph-mon[25603]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T20:14:00.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:14:00] "GET /metrics HTTP/1.1" 200 23670 "" "Prometheus/2.43.0" 2024-09-20T20:14:01.387 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:14:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:01 smithi078 ceph-mon[23685]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:01 smithi078 ceph-mon[23685]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T20:14:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:01 smithi078 ceph-mon[30028]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:01 smithi078 ceph-mon[30028]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T20:14:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:01 smithi146 ceph-mon[25603]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:01 smithi146 ceph-mon[25603]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-20T20:14:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:02 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/796255947' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T20:14:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:02 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/796255947' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T20:14:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:02 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/796255947' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T20:14:02.926 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"flags":0,"active_gid":24335,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":1861129541},{"type":"v1","addr":"172.21.15.78:6801","nonce":1861129541}]},"active_addr":"172.21.15.78:6801/1861129541","active_change":"2024-09-20T20:13:56.233483+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24328,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/","prometheus":"http://172.21.15.78:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":60,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1786721494}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":626323715}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1237956431}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3750775165}]}]} 2024-09-20T20:14:02.929 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T20:14:02.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T20:14:02.929 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T20:14:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:03.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:03 smithi078 ceph-mon[30028]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:03.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:03 smithi078 ceph-mon[23685]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:03 smithi146 ceph-mon[25603]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:05.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:05 smithi078 ceph-mon[30028]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:14:05.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:05 smithi078 ceph-mon[23685]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:14:05.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:05 smithi146 ceph-mon[25603]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-20T20:14:07.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:07 smithi078 ceph-mon[23685]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T20:14:07.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:07 smithi078 ceph-mon[30028]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T20:14:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:07 smithi146 ceph-mon[25603]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-20T20:14:07.889 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:14:07.889 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":60,"fsid":"f2df6e16-778b-11ef-baf5-efdc52797490","created":"2024-09-20T20:08:20.356399+0000","modified":"2024-09-20T20:13:56.233368+0000","last_up_change":"2024-09-20T20:13:21.045468+0000","last_in_change":"2024-09-20T20:13:03.105710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T20:11:25.307285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9d0d6fc3-31fc-4889-b9e8-b088a65da5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6803","nonce":1463358775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6805","nonce":1463358775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6809","nonce":1463358775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6807","nonce":1463358775}]},"public_addr":"172.21.15.78:6803/1463358775","cluster_addr":"172.21.15.78:6805/1463358775","heartbeat_back_addr":"172.21.15.78:6809/1463358775","heartbeat_front_addr":"172.21.15.78:6807/1463358775","state":["exists","up"]},{"osd":1,"uuid":"24879284-6ef9-4428-a3f8-94aedad8a73d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6811","nonce":1479333671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6813","nonce":1479333671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6817","nonce":1479333671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6815","nonce":1479333671}]},"public_addr":"172.21.15.78:6811/1479333671","cluster_addr":"172.21.15.78:6813/1479333671","heartbeat_back_addr":"172.21.15.78:6817/1479333671","heartbeat_front_addr":"172.21.15.78:6815/1479333671","state":["exists","up"]},{"osd":2,"uuid":"7ca1720b-6579-46ec-9813-ad56a23aa8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6819","nonce":1690980420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6821","nonce":1690980420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6825","nonce":1690980420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6823","nonce":1690980420}]},"public_addr":"172.21.15.78:6819/1690980420","cluster_addr":"172.21.15.78:6821/1690980420","heartbeat_back_addr":"172.21.15.78:6825/1690980420","heartbeat_front_addr":"172.21.15.78:6823/1690980420","state":["exists","up"]},{"osd":3,"uuid":"a6c960e5-7b68-4454-b8a9-0534dd820ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6827","nonce":847381434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6829","nonce":847381434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6833","nonce":847381434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6831","nonce":847381434}]},"public_addr":"172.21.15.78:6827/847381434","cluster_addr":"172.21.15.78:6829/847381434","heartbeat_back_addr":"172.21.15.78:6833/847381434","heartbeat_front_addr":"172.21.15.78:6831/847381434","state":["exists","up"]},{"osd":4,"uuid":"57f45a84-12c0-41ff-8a83-53eb19266ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6801","nonce":1760773729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6803","nonce":1760773729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6807","nonce":1760773729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6805","nonce":1760773729}]},"public_addr":"172.21.15.146:6801/1760773729","cluster_addr":"172.21.15.146:6803/1760773729","heartbeat_back_addr":"172.21.15.146:6807/1760773729","heartbeat_front_addr":"172.21.15.146:6805/1760773729","state":["exists","up"]},{"osd":5,"uuid":"880306b7-38a4-4e4b-9019-cd6e2b24c069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6809","nonce":4165836261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6811","nonce":4165836261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6815","nonce":4165836261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6813","nonce":4165836261}]},"public_addr":"172.21.15.146:6809/4165836261","cluster_addr":"172.21.15.146:6811/4165836261","heartbeat_back_addr":"172.21.15.146:6815/4165836261","heartbeat_front_addr":"172.21.15.146:6813/4165836261","state":["exists","up"]},{"osd":6,"uuid":"9a5dd93d-ae84-41fd-836a-3c3dd96da843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6817","nonce":1470150825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6819","nonce":1470150825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6823","nonce":1470150825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6821","nonce":1470150825}]},"public_addr":"172.21.15.146:6817/1470150825","cluster_addr":"172.21.15.146:6819/1470150825","heartbeat_back_addr":"172.21.15.146:6823/1470150825","heartbeat_front_addr":"172.21.15.146:6821/1470150825","state":["exists","up"]},{"osd":7,"uuid":"0f13969f-4da8-4014-bad0-360aae8a5600","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6825","nonce":241106546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6827","nonce":241106546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6831","nonce":241106546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6829","nonce":241106546}]},"public_addr":"172.21.15.146:6825/241106546","cluster_addr":"172.21.15.146:6827/241106546","heartbeat_back_addr":"172.21.15.146:6831/241106546","heartbeat_front_addr":"172.21.15.146:6829/241106546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:10:35.573954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:10:57.762262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:11:21.846854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:11:45.190059+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:08.674507+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:32.376924+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:55.376155+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:13:19.360557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/294299824":"2024-09-21T20:13:56.233336+0000","172.21.15.78:6800/674566149":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6800/2188634654":"2024-09-21T20:08:50.399277+0000","172.21.15.78:6801/2188634654":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/3374519687":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/643935292":"2024-09-21T20:13:56.233336+0000","172.21.15.78:0/2835130118":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/2196865576":"2024-09-21T20:13:56.233336+0000","172.21.15.78:6801/3575808041":"2024-09-21T20:13:56.233336+0000","172.21.15.78:0/3702369749":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/2460914245":"2024-09-21T20:13:56.233336+0000","172.21.15.78:0/4123000022":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6801/674566149":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/2330055755":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/1372816629":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6800/3575808041":"2024-09-21T20:13:56.233336+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T20:14:08.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:08 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1609139385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:14:08.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:08 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1609139385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:14:08.514 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T20:14:08.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T20:14:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:08 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1609139385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:14:09.217 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:09.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:09 smithi078 ceph-mon[30028]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T20:14:09.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:09 smithi078 ceph-mon[23685]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T20:14:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:09 smithi146 ceph-mon[25603]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T20:14:10.683 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:14:10] "GET /metrics HTTP/1.1" 200 23670 "" "Prometheus/2.43.0" 2024-09-20T20:14:10.683 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:14:10] ENGINE Bus STOPPING 2024-09-20T20:14:11.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:14:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:14:11.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:14:11] ENGINE Bus STOPPED 2024-09-20T20:14:11.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:14:11] ENGINE Bus STARTING 2024-09-20T20:14:11.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:14:11] ENGINE Serving on http://:::9283 2024-09-20T20:14:11.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:14:11] ENGINE Bus STARTED 2024-09-20T20:14:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T20:14:11.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:12.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:12 smithi078 ceph-mon[23685]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T20:14:12.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:12 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T20:14:12.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:12 smithi078 ceph-mon[23685]: Deploying daemon grafana.a on smithi146 2024-09-20T20:14:12.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:12 smithi078 ceph-mon[30028]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T20:14:12.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:12 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T20:14:12.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:12 smithi078 ceph-mon[30028]: Deploying daemon grafana.a on smithi146 2024-09-20T20:14:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:12 smithi146 ceph-mon[25603]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T20:14:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:12 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T20:14:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:12 smithi146 ceph-mon[25603]: Deploying daemon grafana.a on smithi146 2024-09-20T20:14:13.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:13 smithi078 ceph-mon[30028]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:13.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:13 smithi078 ceph-mon[23685]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:13 smithi146 ceph-mon[25603]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:14.724 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:14:14.724 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":60,"fsid":"f2df6e16-778b-11ef-baf5-efdc52797490","created":"2024-09-20T20:08:20.356399+0000","modified":"2024-09-20T20:13:56.233368+0000","last_up_change":"2024-09-20T20:13:21.045468+0000","last_in_change":"2024-09-20T20:13:03.105710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T20:11:25.307285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9d0d6fc3-31fc-4889-b9e8-b088a65da5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6803","nonce":1463358775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6805","nonce":1463358775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6809","nonce":1463358775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1463358775},{"type":"v1","addr":"172.21.15.78:6807","nonce":1463358775}]},"public_addr":"172.21.15.78:6803/1463358775","cluster_addr":"172.21.15.78:6805/1463358775","heartbeat_back_addr":"172.21.15.78:6809/1463358775","heartbeat_front_addr":"172.21.15.78:6807/1463358775","state":["exists","up"]},{"osd":1,"uuid":"24879284-6ef9-4428-a3f8-94aedad8a73d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6811","nonce":1479333671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6813","nonce":1479333671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6817","nonce":1479333671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":1479333671},{"type":"v1","addr":"172.21.15.78:6815","nonce":1479333671}]},"public_addr":"172.21.15.78:6811/1479333671","cluster_addr":"172.21.15.78:6813/1479333671","heartbeat_back_addr":"172.21.15.78:6817/1479333671","heartbeat_front_addr":"172.21.15.78:6815/1479333671","state":["exists","up"]},{"osd":2,"uuid":"7ca1720b-6579-46ec-9813-ad56a23aa8e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6819","nonce":1690980420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6821","nonce":1690980420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6825","nonce":1690980420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":1690980420},{"type":"v1","addr":"172.21.15.78:6823","nonce":1690980420}]},"public_addr":"172.21.15.78:6819/1690980420","cluster_addr":"172.21.15.78:6821/1690980420","heartbeat_back_addr":"172.21.15.78:6825/1690980420","heartbeat_front_addr":"172.21.15.78:6823/1690980420","state":["exists","up"]},{"osd":3,"uuid":"a6c960e5-7b68-4454-b8a9-0534dd820ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6827","nonce":847381434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6829","nonce":847381434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6833","nonce":847381434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":847381434},{"type":"v1","addr":"172.21.15.78:6831","nonce":847381434}]},"public_addr":"172.21.15.78:6827/847381434","cluster_addr":"172.21.15.78:6829/847381434","heartbeat_back_addr":"172.21.15.78:6833/847381434","heartbeat_front_addr":"172.21.15.78:6831/847381434","state":["exists","up"]},{"osd":4,"uuid":"57f45a84-12c0-41ff-8a83-53eb19266ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6801","nonce":1760773729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6803","nonce":1760773729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6807","nonce":1760773729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1760773729},{"type":"v1","addr":"172.21.15.146:6805","nonce":1760773729}]},"public_addr":"172.21.15.146:6801/1760773729","cluster_addr":"172.21.15.146:6803/1760773729","heartbeat_back_addr":"172.21.15.146:6807/1760773729","heartbeat_front_addr":"172.21.15.146:6805/1760773729","state":["exists","up"]},{"osd":5,"uuid":"880306b7-38a4-4e4b-9019-cd6e2b24c069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6809","nonce":4165836261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6811","nonce":4165836261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6815","nonce":4165836261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":4165836261},{"type":"v1","addr":"172.21.15.146:6813","nonce":4165836261}]},"public_addr":"172.21.15.146:6809/4165836261","cluster_addr":"172.21.15.146:6811/4165836261","heartbeat_back_addr":"172.21.15.146:6815/4165836261","heartbeat_front_addr":"172.21.15.146:6813/4165836261","state":["exists","up"]},{"osd":6,"uuid":"9a5dd93d-ae84-41fd-836a-3c3dd96da843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6817","nonce":1470150825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6819","nonce":1470150825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6823","nonce":1470150825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1470150825},{"type":"v1","addr":"172.21.15.146:6821","nonce":1470150825}]},"public_addr":"172.21.15.146:6817/1470150825","cluster_addr":"172.21.15.146:6819/1470150825","heartbeat_back_addr":"172.21.15.146:6823/1470150825","heartbeat_front_addr":"172.21.15.146:6821/1470150825","state":["exists","up"]},{"osd":7,"uuid":"0f13969f-4da8-4014-bad0-360aae8a5600","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6825","nonce":241106546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6827","nonce":241106546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6831","nonce":241106546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":241106546},{"type":"v1","addr":"172.21.15.146:6829","nonce":241106546}]},"public_addr":"172.21.15.146:6825/241106546","cluster_addr":"172.21.15.146:6827/241106546","heartbeat_back_addr":"172.21.15.146:6831/241106546","heartbeat_front_addr":"172.21.15.146:6829/241106546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:10:35.573954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:10:57.762262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:11:21.846854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:11:45.190059+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:08.674507+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:32.376924+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:12:55.376155+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T20:13:19.360557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/294299824":"2024-09-21T20:13:56.233336+0000","172.21.15.78:6800/674566149":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6800/2188634654":"2024-09-21T20:08:50.399277+0000","172.21.15.78:6801/2188634654":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/3374519687":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/643935292":"2024-09-21T20:13:56.233336+0000","172.21.15.78:0/2835130118":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/2196865576":"2024-09-21T20:13:56.233336+0000","172.21.15.78:6801/3575808041":"2024-09-21T20:13:56.233336+0000","172.21.15.78:0/3702369749":"2024-09-21T20:08:50.399277+0000","172.21.15.78:0/2460914245":"2024-09-21T20:13:56.233336+0000","172.21.15.78:0/4123000022":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6801/674566149":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/2330055755":"2024-09-21T20:09:12.995362+0000","172.21.15.78:0/1372816629":"2024-09-21T20:09:12.995362+0000","172.21.15.78:6800/3575808041":"2024-09-21T20:13:56.233336+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T20:14:15.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T20:14:15.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T20:14:15.304 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T20:14:15.304 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T20:14:15.304 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T20:14:15.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T20:14:15.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T20:14:15.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T20:14:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:15 smithi078 ceph-mon[23685]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:15 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/925391657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:14:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:15 smithi078 ceph-mon[30028]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:15.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:15 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/925391657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:14:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:15 smithi146 ceph-mon[25603]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T20:14:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:15 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/925391657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T20:14:15.948 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:15.948 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:15.949 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:15.951 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:15.952 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:15.952 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:17 smithi078 ceph-mon[23685]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:17 smithi078 ceph-mon[30028]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:17 smithi146 ceph-mon[25603]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:19.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:19 smithi078 ceph-mon[23685]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:19.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:19 smithi078 ceph-mon[30028]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:19.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:19 smithi146 ceph-mon[25603]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:20.743 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:14:20] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-20T20:14:21.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:21 smithi078 ceph-mon[23685]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:21.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:21 smithi078 ceph-mon[30028]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:21 smithi146 ceph-mon[25603]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:23.017 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:22 smithi146 systemd[1]: Starting Ceph grafana.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:14:23.570 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:23 smithi078 ceph-mon[30028]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:23.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:23 smithi078 ceph-mon[23685]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:23.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:23 smithi146 ceph-mon[25603]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:23.879 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:23 smithi146 podman[46856]: 2024-09-20 20:14:23.689551661 +0000 UTC m=+0.015591734 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-20T20:14:23.879 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:23 smithi146 podman[46856]: 2024-09-20 20:14:23.788267418 +0000 UTC m=+0.114307473 container create f0f8f5c8c22f96fb6dbbeec8f0e66662ea4475e2fcd43961d03064313b06ed0b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a, name=ubi8, architecture=x86_64, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, release=239.1651231664, io.openshift.expose-services=, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-20T20:14:24.160 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:23 smithi146 podman[46856]: 2024-09-20 20:14:23.982118408 +0000 UTC m=+0.308158471 container init f0f8f5c8c22f96fb6dbbeec8f0e66662ea4475e2fcd43961d03064313b06ed0b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, version=8.5, release=239.1651231664, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-20T20:14:24.160 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:23 smithi146 podman[46856]: 2024-09-20 20:14:23.985464625 +0000 UTC m=+0.311504690 container start f0f8f5c8c22f96fb6dbbeec8f0e66662ea4475e2fcd43961d03064313b06ed0b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, release=239.1651231664, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, distribution-scope=public, io.openshift.expose-services=) 2024-09-20T20:14:24.160 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 bash[46856]: f0f8f5c8c22f96fb6dbbeec8f0e66662ea4475e2fcd43961d03064313b06ed0b 2024-09-20T20:14:24.160 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 systemd[1]: Started Ceph grafana.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:14:24.411 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158505882Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158784238Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158810547Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158822403Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158832324Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.15884284Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158852979Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158862893Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158872768Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158882762Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158897067Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-20T20:14:24.412 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158909103Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158918561Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=settings t=2024-09-20T20:14:24.158928678Z level=info msg="App mode production" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore t=2024-09-20T20:14:24.159002425Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore t=2024-09-20T20:14:24.15902905Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.16034188Z level=info msg="Starting DB migrations" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.160441085Z level=info msg="Executing migration" id="create migration_log table" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.161134688Z level=info msg="Executing migration" id="create user table" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.161668729Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.162190364Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.162716357Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.163318787Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.163724325Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-20T20:14:24.413 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.16447272Z level=info msg="Executing migration" id="create user table v2" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.164936518Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.165377711Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.165870923Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.166344113Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.166770098Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.167329074Z level=info msg="Executing migration" id="Update user table charset" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.167725427Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.168597495Z level=info msg="Executing migration" id="Add missing user data" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.169073926Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.169619362Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.170402498Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.17096339Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-20T20:14:24.414 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.1726275Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.173217861Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.173693555Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.174114761Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.174530974Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.174946974Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.175262818Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.17611657Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.176679124Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.177223738Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-20T20:14:24.416 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.178180338Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.17921848Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.180079376Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.180941226Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.181817447Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.182417904Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.18332817Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.184151418Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.18522146Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.185853526Z level=info msg="Executing migration" id="create star table" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.186770829Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.187663949Z level=info msg="Executing migration" id="create org table v1" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.188541878Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.189442845Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.190311621Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-20T20:14:24.417 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.191214546Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.192077845Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.194923008Z level=info msg="Executing migration" id="Update org table charset" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.197768885Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.200545681Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.203395771Z level=info msg="Executing migration" id="create dashboard table" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.206098634Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.20901459Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.211849445Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.21473436Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.217607242Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.220165394Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.22284299Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.2257765Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-20T20:14:24.418 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.228654967Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.23137245Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.234697028Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.23694235Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.239628749Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.242540926Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.243075298Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.245645233Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.248320025Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.251009814Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.251485302Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.254195131Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.254559107Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.256834216Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-20T20:14:24.419 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.257455615Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.25952802Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.260064899Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.260609303Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.263282139Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.263767718Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.265907593Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.268562007Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.269229643Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.270061541Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.271828576Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.27266083Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.273273835Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.274144717Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.274998436Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-20T20:14:24.420 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.275815815Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.276604881Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.2774472Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.278288944Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.279118674Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.279666885Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.280538604Z level=info msg="Executing migration" id="create data_source table" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.281400476Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.282268391Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.282816228Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.283702662Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.284549669Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.285723818Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-20T20:14:24.421 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.286154315Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.287352182Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.287871298Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.288717752Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.289296752Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.290173515Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.291013444Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.291929417Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.29255141Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.293342828Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.293674986Z level=info msg="Executing migration" id="Add uid column" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.294306087Z level=info msg="Executing migration" id="Update uid value" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.295159913Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-20T20:14:24.422 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.295619463Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.296519713Z level=info msg="Executing migration" id="create api_key table" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.297407918Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.299992074Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.300522772Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.303144738Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.305949165Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.308644568Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.311691752Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.314526034Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.31743652Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-20T20:14:24.423 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.32046814Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.323104432Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.32574904Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.328554957Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.331369135Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.33410968Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.336911583Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.339758741Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.343034749Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.345420874Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.348061729Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.350701456Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.353639662Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.356100578Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-20T20:14:24.424 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.358756399Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.361512108Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.364087762Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.366638175Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.369098832Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.371877424Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.374527415Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.377206162Z level=info msg="Executing migration" id="create quota table v1" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.379951584Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.382538274Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.385582185Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.387974698Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.390741263Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.391590663Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-20T20:14:24.425 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.392472437Z level=info msg="Executing migration" id="create session table" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.393369547Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.394233444Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.395068211Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.395992431Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.396425776Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.39670765Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.397035932Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.397361308Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.397692413Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.398157466Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.398470252Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.399147245Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.399490748Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-20T20:14:24.426 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.400114513Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.4007267Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.401067556Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.401517319Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.402027123Z level=info msg="Executing migration" id="create alert table v1" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.402813855Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.403669245Z level=info msg="Executing migration" id="add index alert state" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.404485298Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.404928545Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.405325728Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.405763483Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.406300743Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.407766049Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.408184028Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-20T20:14:24.427 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.408630434Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.408953426Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.409336762Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.409715272Z level=info msg="Executing migration" id="Add column is_default" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.410416849Z level=info msg="Executing migration" id="Add column frequency" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.411100739Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.411775379Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.41239797Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.412824028Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.413116157Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.413403173Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.413792401Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.414225203Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-20T20:14:24.428 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.414597951Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-20T20:14:24.663 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.414983797Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-20T20:14:24.663 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.415392843Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-20T20:14:24.663 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.415984754Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-20T20:14:24.663 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.416613374Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-20T20:14:24.663 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.416911601Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-20T20:14:24.663 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.417331771Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.417694898Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.41831693Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.418614148Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.419038198Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.419442911Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.419719652Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.420147016Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.420954882Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.421357138Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.421757703Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.422226381Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.423065426Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-20T20:14:24.664 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.425795512Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-20T20:14:24.665 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.426508304Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-20T20:14:24.665 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.429801067Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-20T20:14:24.665 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.433373666Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-20T20:14:24.665 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.436260197Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.43963229Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.440184607Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.443058717Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.445949913Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.448677482Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.449085118Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.451981712Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.455344325Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.457540324Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.458247445Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.460952218Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-20T20:14:24.666 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.463888461Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.466565809Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.469402775Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.472025162Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.474731575Z level=info msg="Executing migration" id="Move region to single row" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.477320168Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.479941074Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.482411444Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.484829279Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.487409894Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.49009942Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.492802032Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-20T20:14:24.667 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.495988901Z level=info msg="Executing migration" id="create test_data table" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.49640179Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.497949773Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.500532721Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.503101667Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.505716769Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.508325129Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.510817227Z level=info msg="Executing migration" id="create team table" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.513443715Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.514280513Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.515086693Z level=info msg="Executing migration" id="create team member table" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.515908809Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.516758386Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-20T20:14:24.668 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.517598313Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.518448876Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.519268315Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.520060271Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.520920034Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.521769347Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.522651235Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.523404036Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.523868417Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.524700396Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.525515966Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-20T20:14:24.669 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.526359085Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-20T20:14:24.670 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.526813649Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-20T20:14:24.670 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.527596878Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-20T20:14:24.670 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.528441869Z level=info msg="Executing migration" id="create tag table" 2024-09-20T20:14:24.670 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.529304362Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.530147418Z level=info msg="Executing migration" id="create login attempt table" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.530972832Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.531849025Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.532691393Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.534638968Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.535478741Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-20T20:14:24.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.536324872Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.537188039Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.53759275Z level=info msg="Executing migration" id="create user auth table" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.538444925Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.538918522Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.539313721Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.540163166Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.540991567Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.541809426Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.542690347Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.545468089Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.548238138Z level=info msg="Executing migration" id="create server_lock table" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.550885675Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-20T20:14:24.672 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.553736583Z level=info msg="Executing migration" id="create user auth token table" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.556468607Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.556948942Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.559807883Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.562387004Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.565039773Z level=info msg="Executing migration" id="create cache_data table" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.565503068Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.566370035Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.567262194Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.568094604Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.568408808Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.569235553Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-20T20:14:24.673 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.569728218Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.570197515Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.571070281Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.57136435Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.572192915Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.572612544Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.57309736Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.573996098Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.57479577Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.575748479Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.576630906Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.57747094Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-20T20:14:24.674 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.577908047Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.578778185Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.579620043Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.580514644Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.580975374Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.581838999Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.582750193Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.583620197Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.584084669Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.584976828Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.58859592Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.592053417Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-20T20:14:24.675 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.59250543Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.593334769Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.594171647Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.59628676Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.599152933Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.599755936Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.602623316Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.605658741Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.608535979Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.609379297Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.612341406Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.615329848Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-20T20:14:24.676 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.618034596Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.618872355Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.6217449Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.624389739Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.627196994Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.628040099Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.63106558Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.633708852Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.637085981Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.637545778Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.640303701Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.641243627Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.642998428Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.645741756Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-20T20:14:24.677 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.646605618Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.649204164Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.65183917Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.654576977Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.655456655Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.65816805Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.660867612Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.663445998Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-20T20:14:24.678 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.664388514Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.667047329Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.669776255Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.670704983Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.671154976Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.671995401Z level=info msg="Executing migration" id="create alert_image table" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.672869284Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.673747556Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.6740963Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.674950673Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-20T20:14:24.925 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.675789936Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.675885701Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.676630594Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.677019656Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.677856303Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.678010947Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.67815135Z level=info msg="alerts found to migrate" alerts=0 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.678722691Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.67957282Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.680075374Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.680939325Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.681746222Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-20T20:14:24.926 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.682599286Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.682947536Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.683991127Z level=info msg="Executing migration" id="create data_keys table" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.684652141Z level=info msg="Executing migration" id="create secrets table" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.685531125Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.690014745Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.69096085Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.691845301Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.696230252Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.700689731Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.701552077Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.70246114Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.703313372Z level=info msg="Executing migration" id="create permission table" 2024-09-20T20:14:24.927 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.703751815Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-20T20:14:24.928 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.704585214Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-20T20:14:24.928 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.705465921Z level=info msg="Executing migration" id="create role table" 2024-09-20T20:14:24.928 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.706317336Z level=info msg="Executing migration" id="add column display_name" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.707281636Z level=info msg="Executing migration" id="add column group_name" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.708246564Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.71109547Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.714126984Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.714840242Z level=info msg="Executing migration" id="create team role table" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.717664911Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.720459604Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.723163247Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.723806753Z level=info msg="Executing migration" id="create user role table" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.726743852Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.729624346Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.732124668Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.732593483Z level=info msg="Executing migration" id="create builtin role table" 2024-09-20T20:14:24.929 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.7354939Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.738424284Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.746107747Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.747291506Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.750153357Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.753167645Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.7557815Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.756267552Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.758931066Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.76162593Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.76431304Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.764754624Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.76749104Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.770098741Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.772699607Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-20T20:14:24.930 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.773192019Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.775943704Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.778570231Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.781357746Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.781818927Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.782281831Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.783372276Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.783703836Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.784165165Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.784638572Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.785121674Z level=info msg="Executing migration" id="add correlation config column" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.786167178Z level=info msg="Executing migration" id="create entity_events table" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.786664599Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.789431555Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.789567161Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T20:14:24.931 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.789905852Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.790010822Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.790318339Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.790732135Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.79120707Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.793406514Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.796318189Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.799017097Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.799505216Z level=info msg="Executing migration" id="Drop public config table" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.802167458Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.802641951Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.804895274Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.807574635Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.808179803Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.811297716Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.813813235Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-20T20:14:24.932 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.816525289Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.81692118Z level=info msg="Executing migration" id="add share column" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.819631379Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.822411847Z level=info msg="Executing migration" id="create file table" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.825068667Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.82557478Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.82811878Z level=info msg="Executing migration" id="create file_meta table" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.830424341Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.831464728Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.831781266Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.834494175Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.835303736Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-20T20:14:24.933 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.836153371Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.837287634Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.838112547Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.838941525Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.839766123Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.840138466Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.840936427Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.84201257Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.843085419Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.84357129Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.854019675Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.854834254Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-20T20:14:24.934 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.855608561Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.859322966Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.860178348Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.861023268Z level=info msg="Executing migration" id="create folder table" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.861864467Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.862361148Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.863205059Z level=info msg="Executing migration" id="Update folder title length" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.864045543Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=migrator t=2024-09-20T20:14:24.864925375Z level=info msg="migrations completed" performed=485 skipped=0 duration=704.511042ms 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore t=2024-09-20T20:14:24.865460035Z level=info msg="Created default organization" 2024-09-20T20:14:24.935 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=plugin.loader t=2024-09-20T20:14:24.922256993Z level=info msg="Plugin registered" pluginID=input 2024-09-20T20:14:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-mon[25603]: Deploying daemon node-exporter.a on smithi078 2024-09-20T20:14:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-mon[25603]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:25.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=plugin.loader t=2024-09-20T20:14:24.95355941Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-20T20:14:25.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=plugin.loader t=2024-09-20T20:14:24.953587134Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-20T20:14:25.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=secrets t=2024-09-20T20:14:24.953627648Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-20T20:14:25.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=query_data t=2024-09-20T20:14:24.955341546Z level=info msg="Query Service initialization" 2024-09-20T20:14:25.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=live.push_http t=2024-09-20T20:14:24.957659882Z level=info msg="Live Push Gateway initialization" 2024-09-20T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[30028]: Deploying daemon node-exporter.a on smithi078 2024-09-20T20:14:25.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[30028]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:25.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:25.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[23685]: Deploying daemon node-exporter.a on smithi078 2024-09-20T20:14:25.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:25 smithi078 ceph-mon[23685]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:25.954 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=infra.usagestats.collector t=2024-09-20T20:14:25.726729625Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-20T20:14:25.954 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=provisioning.datasources t=2024-09-20T20:14:25.781072427Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T20:14:25.954 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=provisioning.datasources t=2024-09-20T20:14:25.812697739Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-20T20:14:25.954 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=provisioning.alerting t=2024-09-20T20:14:25.834449196Z level=info msg="starting to provision alerting" 2024-09-20T20:14:25.954 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=provisioning.alerting t=2024-09-20T20:14:25.834479698Z level=info msg="finished to provision alerting" 2024-09-20T20:14:25.954 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=ngalert.state.manager t=2024-09-20T20:14:25.834766604Z level=info msg="Warming state cache for startup" 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=grafanaStorageLogger t=2024-09-20T20:14:25.834920495Z level=info msg="storage starting" 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=http.server t=2024-09-20T20:14:25.841866129Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=ngalert.state.manager t=2024-09-20T20:14:25.845106344Z level=info msg="State cache has been initialized" states=0 duration=10.339655ms 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=ngalert.multiorg.alertmanager t=2024-09-20T20:14:25.845143546Z level=info msg="starting MultiOrg Alertmanager" 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=ticker t=2024-09-20T20:14:25.845172824Z level=info msg=starting first_tick=2024-09-20T20:14:30Z 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore.transactions t=2024-09-20T20:14:25.847001048Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore.transactions t=2024-09-20T20:14:25.894017879Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T20:14:25.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore.transactions t=2024-09-20T20:14:25.90446193Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-20T20:14:26.650 INFO:teuthology.orchestra.run.smithi078.stdout:55834574892 2024-09-20T20:14:26.650 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T20:14:27.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:27 smithi078 ceph-mon[30028]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:27.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:27.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:27.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:27 smithi078 ceph-mon[23685]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:27.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:27.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:27.537 INFO:teuthology.orchestra.run.smithi078.stdout:133143986205 2024-09-20T20:14:27.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T20:14:27.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:27 smithi146 ceph-mon[25603]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:27.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:27.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:236223201296 2024-09-20T20:14:28.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T20:14:29.338 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:29 smithi146 ceph-mon[25603]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:29 smithi078 ceph-mon[23685]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:29 smithi078 ceph-mon[30028]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:14:30] "GET /metrics HTTP/1.1" 200 26198 "" "Prometheus/2.43.0" 2024-09-20T20:14:31.150 INFO:teuthology.orchestra.run.smithi078.stdout:77309411368 2024-09-20T20:14:31.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T20:14:31.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:31 smithi078 ceph-mon[23685]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:31.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:31 smithi078 ceph-mon[30028]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:31.717 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:31 smithi146 ceph-mon[25603]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:33.069 INFO:teuthology.orchestra.run.smithi078.stdout:111669149731 2024-09-20T20:14:33.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T20:14:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:33 smithi146 ceph-mon[25603]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:33 smithi078 ceph-mon[23685]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:33 smithi078 ceph-mon[30028]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:34.508 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:34.509 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:34.934 INFO:teuthology.orchestra.run.smithi078.stdout:201863462932 2024-09-20T20:14:34.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T20:14:35.493 INFO:teuthology.orchestra.run.smithi078.stdout:167503724570 2024-09-20T20:14:35.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T20:14:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:35 smithi146 ceph-mon[25603]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:35.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:35 smithi078 ceph-mon[30028]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:35.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:35 smithi078 ceph-mon[23685]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:36.897 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:37.233 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:37 smithi146 ceph-mon[25603]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:37 smithi078 ceph-mon[23685]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:37 smithi078 ceph-mon[30028]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:38.104 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:38.619 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:39 smithi078 ceph-mon[23685]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:39 smithi078 ceph-mon[30028]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:39.537 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:39 smithi078 podman[53126]: 2024-09-20 20:14:39.370973126 +0000 UTC m=+3.414305728 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T20:14:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:39 smithi146 ceph-mon[25603]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:39.976 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:39 smithi078 podman[53126]: 2024-09-20 20:14:39.695245711 +0000 UTC m=+3.738578307 container create e38df3435cd87d7c669af504bb440682401744f9ab035fd475b01f01a15e81b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T20:14:40.863 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:14:40] "GET /metrics HTTP/1.1" 200 26198 "" "Prometheus/2.43.0" 2024-09-20T20:14:40.864 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 podman[53126]: 2024-09-20 20:14:40.662375051 +0000 UTC m=+4.705707643 container init e38df3435cd87d7c669af504bb440682401744f9ab035fd475b01f01a15e81b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T20:14:40.864 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 podman[53126]: 2024-09-20 20:14:40.66539786 +0000 UTC m=+4.708730453 container start e38df3435cd87d7c669af504bb440682401744f9ab035fd475b01f01a15e81b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T20:14:40.864 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.669Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T20:14:40.864 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.669Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T20:14:40.864 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.669Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.669Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.669Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.669Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T20:14:40.865 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T20:14:40.866 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T20:14:40.866 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T20:14:40.866 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T20:14:40.866 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T20:14:40.867 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T20:14:40.868 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T20:14:40.869 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T20:14:40.870 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T20:14:40.870 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T20:14:40.870 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T20:14:40.870 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T20:14:40.870 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T20:14:40.870 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T20:14:40.871 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T20:14:40.872 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T20:14:40.872 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-a[53450]: ts=2024-09-20T20:14:40.670Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T20:14:40.872 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 bash[53126]: e38df3435cd87d7c669af504bb440682401744f9ab035fd475b01f01a15e81b9 2024-09-20T20:14:40.872 INFO:journalctl@ceph.node-exporter.a.smithi078.stdout:Sep 20 20:14:40 smithi078 systemd[1]: Started Ceph node-exporter.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:14:41.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:41 smithi146 ceph-mon[25603]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:41.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[23685]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[30028]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:41.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:41.739 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:41 smithi146 systemd[1]: Starting Ceph node-exporter.b for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:14:42.017 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:41 smithi146 bash[47786]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T20:14:42.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:42 smithi078 ceph-mon[30028]: Deploying daemon node-exporter.b on smithi146 2024-09-20T20:14:42.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:42 smithi078 ceph-mon[23685]: Deploying daemon node-exporter.b on smithi146 2024-09-20T20:14:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:42 smithi146 ceph-mon[25603]: Deploying daemon node-exporter.b on smithi146 2024-09-20T20:14:42.516 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:42 smithi146 bash[47786]: Getting image source signatures 2024-09-20T20:14:42.517 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:42 smithi146 bash[47786]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T20:14:42.517 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:42 smithi146 bash[47786]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T20:14:42.517 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:42 smithi146 bash[47786]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T20:14:43.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:43 smithi146 ceph-mon[25603]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:43 smithi078 ceph-mon[23685]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:43 smithi078 ceph-mon[30028]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:44.016 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:43 smithi146 bash[47786]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T20:14:44.016 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:43 smithi146 bash[47786]: Writing manifest to image destination 2024-09-20T20:14:44.423 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 podman[47786]: 2024-09-20 20:14:44.092688792 +0000 UTC m=+2.446046894 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T20:14:44.423 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 podman[47786]: 2024-09-20 20:14:44.18275886 +0000 UTC m=+2.536116949 container create 235fece1738e65f4a7853bba44d5a6187cc0ad8a5b655a344b5d4555d34e3cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T20:14:44.423 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 podman[47786]: 2024-09-20 20:14:44.418385923 +0000 UTC m=+2.771744022 container init 235fece1738e65f4a7853bba44d5a6187cc0ad8a5b655a344b5d4555d34e3cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T20:14:44.423 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 podman[47786]: 2024-09-20 20:14:44.421601092 +0000 UTC m=+2.774959182 container start 235fece1738e65f4a7853bba44d5a6187cc0ad8a5b655a344b5d4555d34e3cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T20:14:44.559 INFO:teuthology.orchestra.run.smithi078.stdout:133143986209 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.425Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.426Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.426Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.426Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.426Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.426Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T20:14:44.767 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T20:14:44.768 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T20:14:44.769 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T20:14:44.770 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T20:14:44.771 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T20:14:44.772 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T20:14:44.772 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T20:14:44.772 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T20:14:44.772 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-node-exporter-b[47860]: ts=2024-09-20T20:14:44.427Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T20:14:44.772 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 bash[47786]: 235fece1738e65f4a7853bba44d5a6187cc0ad8a5b655a344b5d4555d34e3cbf 2024-09-20T20:14:44.772 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Sep 20 20:14:44 smithi146 systemd[1]: Started Ceph node-exporter.b for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:14:45.198 INFO:teuthology.orchestra.run.smithi078.stdout:34359738419 2024-09-20T20:14:45.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T20:14:45.482 INFO:teuthology.orchestra.run.smithi078.stdout:77309411371 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2764783773' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:14:45.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2754834896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2764783773' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:14:45.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:45 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2754834896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T20:14:45.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:45.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2764783773' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T20:14:45.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:14:45.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:45 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2754834896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T20:14:46.057 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-09-20T20:14:46.057 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:46.430 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-09-20T20:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:46 smithi146 ceph-mon[25603]: Reconfiguring daemon agent.smithi146 on smithi146 2024-09-20T20:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:46 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:46 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:46 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:46 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:46 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[23685]: Reconfiguring daemon agent.smithi146 on smithi146 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[30028]: Reconfiguring daemon agent.smithi146 on smithi146 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:46 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:47.193 INFO:teuthology.orchestra.run.smithi078.stdout:201863462935 2024-09-20T20:14:47.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:47 smithi078 ceph-mon[23685]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:47.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:47 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/770158970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T20:14:47.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:47 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2140201753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T20:14:47.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:47 smithi078 ceph-mon[30028]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:47.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:47 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/770158970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T20:14:47.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:47 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2140201753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T20:14:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:47 smithi146 ceph-mon[25603]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:47 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/770158970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T20:14:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:47 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2140201753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T20:14:47.820 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462932 got 201863462935 for osd.6 2024-09-20T20:14:47.821 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:48.367 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:48.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-09-20T20:14:48.718 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:49.108 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986205 got 133143986209 for osd.4 2024-09-20T20:14:49.108 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:49 smithi146 ceph-mon[25603]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:49 smithi078 ceph-mon[30028]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:49 smithi078 ceph-mon[23685]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:50.915 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:14:50] "GET /metrics HTTP/1.1" 200 26204 "" "Prometheus/2.43.0" 2024-09-20T20:14:51.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:51 smithi078 ceph-mon[30028]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:51.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:51 smithi078 ceph-mon[23685]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:51 smithi146 ceph-mon[25603]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:52.141 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:14:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:53 smithi146 ceph-mon[25603]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:53.557 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:53 smithi078 ceph-mon[30028]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:53.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:53 smithi078 ceph-mon[23685]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:53.901 INFO:teuthology.orchestra.run.smithi078.stdout:236223201301 2024-09-20T20:14:54.471 INFO:teuthology.orchestra.run.smithi078.stdout:111669149736 2024-09-20T20:14:54.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:54 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3889001062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T20:14:54.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:54 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3889001062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T20:14:54.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:54 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3889001062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T20:14:55.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149731 got 111669149736 for osd.3 2024-09-20T20:14:55.151 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:55.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:55 smithi078 ceph-mon[23685]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:55.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:55 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/4093713361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T20:14:55.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:55 smithi078 ceph-mon[30028]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:55.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:55 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/4093713361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T20:14:55.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:55 smithi146 ceph-mon[25603]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:55.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:55 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/4093713361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T20:14:55.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201296 got 236223201301 for osd.7 2024-09-20T20:14:55.804 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:56.200 INFO:teuthology.orchestra.run.smithi078.stdout:167503724575 2024-09-20T20:14:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:56 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/67641699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T20:14:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:56 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/67641699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T20:14:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:56 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/67641699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T20:14:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:14:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:57 smithi146 ceph-mon[25603]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:57.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:57 smithi078 ceph-mon[30028]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:57.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:57.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:57 smithi078 ceph-mon[23685]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:57.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:14:57.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724570 got 167503724575 for osd.5 2024-09-20T20:14:57.943 DEBUG:teuthology.parallel:result is None 2024-09-20T20:14:59.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:14:59 smithi146 ceph-mon[25603]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:59.599 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:14:59 smithi078 ceph-mon[30028]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:14:59.599 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:14:59 smithi078 ceph-mon[23685]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:00.735 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:15:00] "GET /metrics HTTP/1.1" 200 26197 "" "Prometheus/2.43.0" 2024-09-20T20:15:01.156 INFO:teuthology.orchestra.run.smithi078.stdout:34359738423 2024-09-20T20:15:01.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:01 smithi146 ceph-mon[25603]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:01.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:01 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:01.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:01 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2603035900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T20:15:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:01 smithi078 ceph-mon[23685]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:01 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:01 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2603035900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T20:15:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:01 smithi078 ceph-mon[30028]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:01 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:01 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2603035900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T20:15:01.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738423 for osd.0 2024-09-20T20:15:01.942 DEBUG:teuthology.parallel:result is None 2024-09-20T20:15:01.942 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T20:15:01.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T20:15:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:03.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:03 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:03 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:03 smithi146 ceph-mon[25603]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:03.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:03 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:03 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[30028]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[23685]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:03 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:15:03.850 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-20T20:15:04.688 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-09-20T20:15:02.256879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":226624,"kb_used_data":6376,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749734592,"statfs":{"total":767960285184,"available":767728222208,"internally_reserved":0,"allocated":6529024,"data_stored":3701877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001851"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":256,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T20:13:58.312646+0000","last_change":"2024-09-20T20:13:25.076065+0000","last_active":"2024-09-20T20:13:58.312646+0000","last_peered":"2024-09-20T20:13:58.312646+0000","last_clean":"2024-09-20T20:13:58.312646+0000","last_became_active":"2024-09-20T20:13:25.075587+0000","last_became_peered":"2024-09-20T20:13:25.075587+0000","last_unstale":"2024-09-20T20:13:58.312646+0000","last_undegraded":"2024-09-20T20:13:58.312646+0000","last_fullsized":"2024-09-20T20:13:58.312646+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T20:11:25.836337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T20:11:25.836337+0000","last_clean_scrub_stamp":"2024-09-20T20:11:25.836337+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:25.735144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28180,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716972,"statfs":{"total":95995035648,"available":95966179328,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.44},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.374},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.41199999999999998}]},{"osd":4,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.316}]},{"osd":5,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36299999999999999}]},{"osd":6,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.35499999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Fri Sep 20 20:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Sep 20 20:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Sep 20 20:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Sep 20 20:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.375},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38900000000000001}]},{"osd":5,"last update":"Fri Sep 20 20:14:02 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.40200000000000002}]},{"osd":7,"last update":"Fri Sep 20 20:14:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.44},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.495}]}]},{"osd":5,"up_from":39,"seq":167503724577,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717556,"statfs":{"total":95995035648,"available":95966777344,"internally_reserved":0,"allocated":589824,"data_stored":241057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.435}]},{"osd":1,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.57299999999999995,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.0869999999999997,"5min":4.0869999999999997,"15min":4.0869999999999997},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72699999999999998,"5min":0.745,"15min":0.745},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.436,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69799999999999995,"5min":0.745,"15min":0.745},"last":0.442}]},{"osd":4,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.40899999999999997},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.30499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63700000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48199999999999998}]},{"osd":6,"last update":"Fri Sep 20 20:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Sep 20 20:14:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.374},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.505}]}]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53100000000000003,"15min":0.53700000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60899999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60199999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.622,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.497},{"interface":"front","average":{"1min":0.497,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.58299999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48799999999999999}]},{"osd":2,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.628,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.64500000000000002,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.59099999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.439}]},{"osd":5,"last update":"Fri Sep 20 20:14:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.35299999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"min":{"1min":0.32800000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.41499999999999998}]},{"osd":6,"last update":"Fri Sep 20 20:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.38900000000000001,"15min":0.38},"min":{"1min":0.28599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.438,"15min":0.42899999999999999},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45100000000000001}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40000000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36299999999999999,"15min":0.378},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53900000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.48599999999999999}]},{"osd":1,"last update":"Fri Sep 20 20:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.55500000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39200000000000002,"15min":0.40000000000000002},"min":{"1min":0.26400000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.56399999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Sep 20 20:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42199999999999999,"15min":0.436},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.58599999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41599999999999998,"15min":0.42199999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53800000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.45700000000000002}]},{"osd":4,"last update":"Fri Sep 20 20:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Sep 20 20:14:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69399999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.58599999999999997}]},{"osd":6,"last update":"Fri Sep 20 20:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65300000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.373,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.377,"15min":0.37},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56000000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri Sep 20 20:14:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.38200000000000001,"15min":0.38400000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.371},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.35099999999999998}]},{"osd":3,"last update":"Fri Sep 20 20:14:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44800000000000001,"15min":0.45400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.57599999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.433,"15min":0.441},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.32200000000000001}]},{"osd":4,"last update":"Fri Sep 20 20:14:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.433},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.49299999999999999}]},{"osd":5,"last update":"Fri Sep 20 20:14:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80100000000000005,"5min":1.03,"15min":1.03},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.871,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47899999999999998}]},{"osd":6,"last update":"Fri Sep 20 20:13:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31696,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713456,"statfs":{"total":95995035648,"available":95962578944,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.376,"15min":0.38300000000000001},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.499,"5min":1.998,"15min":1.998},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.39200000000000002},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Sep 20 20:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.379,"15min":0.373},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Sep 20 20:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.53800000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.30499999999999999}]},{"osd":4,"last update":"Fri Sep 20 20:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55000000000000004,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.67200000000000004,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55600000000000005,"15min":0.56299999999999994},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69499999999999995,"5min":0.747,"15min":0.747},"last":0.40899999999999997}]},{"osd":5,"last update":"Fri Sep 20 20:14:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36499999999999999}]},{"osd":6,"last update":"Fri Sep 20 20:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.45600000000000002}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 20:14:03 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45800000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.27200000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.438}]},{"osd":2,"last update":"Fri Sep 20 20:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38200000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38200000000000001,"15min":0.38500000000000001},"min":{"1min":0.248,"5min":0.245,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Sep 20 20:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.25700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.50800000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Sep 20 20:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74299999999999999,"5min":0.748,"15min":0.748},"last":0.58999999999999997}]},{"osd":5,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.625},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Sep 20 20:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.60199999999999998}]},{"osd":7,"last update":"Fri Sep 20 20:14:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T20:15:04.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T20:15:05.074 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:05.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:05 smithi078 ceph-mon[23685]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:15:05.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:05 smithi078 ceph-mon[23685]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:05.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:05 smithi078 ceph-mon[30028]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:15:05.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:05 smithi078 ceph-mon[30028]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:05.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:05 smithi146 ceph-mon[25603]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:15:05.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:05 smithi146 ceph-mon[25603]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi146 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:15:07.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: Reconfiguring daemon alertmanager.a on smithi078 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi146 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: Reconfiguring daemon alertmanager.a on smithi078 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:06 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi146 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: Reconfiguring daemon alertmanager.a on smithi078 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:06 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:07 smithi146 ceph-mon[25603]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:15:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:07 smithi078 ceph-mon[23685]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:15:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:07 smithi078 ceph-mon[30028]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:15:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:08 smithi078 ceph-mon[23685]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:08 smithi078 ceph-mon[30028]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:08 smithi146 ceph-mon[25603]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:10.234 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:15:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-20T20:15:10.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[50666]: ts=2024-09-20T20:15:10.324Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T20:15:10.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:10 smithi078 podman[55673]: 2024-09-20 20:15:10.49424817 +0000 UTC m=+0.283440184 container died ea69a9da14e339db72136010acf6e7ce3e1adb66152317611592c6b1f2f868cf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:15:10] "GET /metrics HTTP/1.1" 200 26197 "" "Prometheus/2.43.0" 2024-09-20T20:15:11.169 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:10 smithi078 podman[55673]: 2024-09-20 20:15:10.91663125 +0000 UTC m=+0.705823246 container cleanup ea69a9da14e339db72136010acf6e7ce3e1adb66152317611592c6b1f2f868cf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:11.170 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:10 smithi078 bash[55673]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T20:15:11.170 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:11 smithi078 podman[55712]: 2024-09-20 20:15:11.084346961 +0000 UTC m=+0.710679049 container remove ea69a9da14e339db72136010acf6e7ce3e1adb66152317611592c6b1f2f868cf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:11.170 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:11 smithi078 podman[55712]: 2024-09-20 20:15:11.168249553 +0000 UTC m=+0.794581633 volume remove 14d08b0558d3c4e4c123101b50e16ea057d598f8fac11e2a5b532efdcaf47ddb 2024-09-20T20:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[23685]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[23685]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[23685]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[30028]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[30028]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:15:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[30028]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.601 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-20T20:15:09.737138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":226624,"kb_used_data":6376,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749734592,"statfs":{"total":767960285184,"available":767728222208,"internally_reserved":0,"allocated":6529024,"data_stored":3701877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.480835"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":256,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T20:13:58.312646+0000","last_change":"2024-09-20T20:13:25.076065+0000","last_active":"2024-09-20T20:13:58.312646+0000","last_peered":"2024-09-20T20:13:58.312646+0000","last_clean":"2024-09-20T20:13:58.312646+0000","last_became_active":"2024-09-20T20:13:25.075587+0000","last_became_peered":"2024-09-20T20:13:25.075587+0000","last_unstale":"2024-09-20T20:13:58.312646+0000","last_undegraded":"2024-09-20T20:13:58.312646+0000","last_fullsized":"2024-09-20T20:13:58.312646+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T20:11:25.836337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T20:11:25.836337+0000","last_clean_scrub_stamp":"2024-09-20T20:11:25.836337+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T02:27:25.735144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201304,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28180,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716972,"statfs":{"total":95995035648,"available":95966179328,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.44400000000000001}]},{"osd":1,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.435},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.38700000000000001}]},{"osd":3,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.41399999999999998}]},{"osd":4,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.29699999999999999}]},{"osd":5,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.31},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.39900000000000002}]},{"osd":6,"last update":"Fri Sep 20 20:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.379}]}]},{"osd":6,"up_from":47,"seq":201863462940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.65900000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.434,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77400000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.437},{"interface":"front","average":{"1min":0.504,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68999999999999995,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72199999999999998,"5min":0.995,"15min":0.995},"last":0.376},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67600000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.38600000000000001}]},{"osd":3,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76500000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68700000000000006,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.35599999999999998}]},{"osd":4,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47799999999999998,"15min":0.48399999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68100000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.46899999999999997,"15min":0.47199999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65100000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.316}]},{"osd":5,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48299999999999998,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.60999999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.34100000000000003}]},{"osd":7,"last update":"Fri Sep 20 20:14:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.39600000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724578,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717556,"statfs":{"total":95995035648,"available":95966777344,"internally_reserved":0,"allocated":589824,"data_stored":241057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.55700000000000005}]},{"osd":1,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.57299999999999995,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.0869999999999997,"5min":4.0869999999999997,"15min":4.0869999999999997},"last":0.498}]},{"osd":2,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56799999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72699999999999998,"5min":0.745,"15min":0.745},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.436,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69799999999999995,"5min":0.745,"15min":0.745},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.41799999999999998,"15min":0.40899999999999997},"min":{"1min":0.29999999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.372},{"interface":"front","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39700000000000002},"min":{"1min":0.30499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63700000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41499999999999998}]},{"osd":6,"last update":"Fri Sep 20 20:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45800000000000002,"15min":0.45500000000000002},"min":{"1min":0.32500000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.76700000000000002,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.60099999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Sep 20 20:14:24 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.46600000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53100000000000003,"15min":0.53700000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60899999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.60199999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.622,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.58299999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.5},{"interface":"front","average":{"1min":0.5,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.628,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Sep 20 20:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.64500000000000002,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.59099999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51000000000000001}]},{"osd":5,"last update":"Fri Sep 20 20:14:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.35299999999999998},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"min":{"1min":0.32800000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.437}]},{"osd":6,"last update":"Fri Sep 20 20:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.38900000000000001,"15min":0.38},"min":{"1min":0.28599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.438,"15min":0.42899999999999999},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.40200000000000002}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.376},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.70099999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.36299999999999999,"15min":0.378},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53900000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35999999999999999,"15min":0.36799999999999999},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Fri Sep 20 20:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.55500000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39200000000000002,"15min":0.40000000000000002},"min":{"1min":0.26400000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.56399999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.27700000000000002}]},{"osd":2,"last update":"Fri Sep 20 20:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42199999999999999,"15min":0.436},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.58599999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41599999999999998,"15min":0.42199999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53800000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.311}]},{"osd":4,"last update":"Fri Sep 20 20:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53200000000000003,"15min":0.52900000000000003},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Sep 20 20:14:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69399999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57499999999999996}]},{"osd":6,"last update":"Fri Sep 20 20:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71699999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.504},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.58599999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717552,"statfs":{"total":95995035648,"available":95966773248,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:34 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.373,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.377,"15min":0.37},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56000000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Sep 20 20:14:34 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.38200000000000001,"15min":0.38400000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.57899999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41399999999999998}]},{"osd":3,"last update":"Fri Sep 20 20:14:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44800000000000001,"15min":0.45400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.57599999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.433,"15min":0.441},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.37}]},{"osd":4,"last update":"Fri Sep 20 20:14:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.501}]},{"osd":5,"last update":"Fri Sep 20 20:14:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80100000000000005,"5min":1.03,"15min":1.03},"last":0.624},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.871,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Sep 20 20:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.40000000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78900000000000003,"5min":0.871,"15min":0.871},"last":0.64300000000000002}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31696,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713456,"statfs":{"total":95995035648,"available":95962578944,"internally_reserved":0,"allocated":593920,"data_stored":241388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 20:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.376,"15min":0.38300000000000001},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.499,"5min":1.998,"15min":1.998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.39200000000000002},"min":{"1min":0.223,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Sep 20 20:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34799999999999998,"15min":0.34399999999999997},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.496,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.379,"15min":0.373},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.32200000000000001}]},{"osd":3,"last update":"Fri Sep 20 20:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42699999999999999,"15min":0.439},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40100000000000002,"15min":0.40899999999999997},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.53800000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.35299999999999998}]},{"osd":4,"last update":"Fri Sep 20 20:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.55000000000000004,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.67200000000000004,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55600000000000005,"15min":0.56299999999999994},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69499999999999995,"5min":0.747,"15min":0.747},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Sep 20 20:14:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.5},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.49099999999999999}]},{"osd":6,"last update":"Fri Sep 20 20:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.45800000000000002}]},{"osd":7,"last update":"Fri Sep 20 20:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.53000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1187840,"data_stored":831756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 20:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.34999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.32200000000000001}]},{"osd":2,"last update":"Fri Sep 20 20:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38200000000000001,"15min":0.38100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38200000000000001,"15min":0.38500000000000001},"min":{"1min":0.248,"5min":0.245,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Fri Sep 20 20:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.25700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.50800000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60599999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Sep 20 20:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.57899999999999996,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74299999999999999,"5min":0.748,"15min":0.748},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Sep 20 20:14:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.44},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Sep 20 20:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.46700000000000003}]},{"osd":7,"last update":"Fri Sep 20 20:14:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T20:15:11.603 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T20:15:11.603 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T20:15:11.603 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T20:15:11.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T20:15:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:11 smithi146 ceph-mon[25603]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:11 smithi146 ceph-mon[25603]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T20:15:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:11 smithi146 ceph-mon[25603]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:11.864 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:11 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T20:15:11.864 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:11 smithi078 systemd[1]: Stopped Ceph alertmanager.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:15:11.864 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:11 smithi078 systemd[1]: Starting Ceph alertmanager.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:15:12.462 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:12.519 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:12 smithi078 ceph-mon[30028]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-20T20:15:12.519 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:12 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:12.519 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:12 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:12.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:12 smithi078 ceph-mon[23685]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-20T20:15:12.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:12 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:12.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:12 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:12 smithi146 ceph-mon[25603]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-20T20:15:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:12 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:12 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:13 smithi078 ceph-mon[23685]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:13 smithi078 ceph-mon[30028]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:13.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:13 smithi146 ceph-mon[25603]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:14.694 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:14 smithi078 podman[55975]: 2024-09-20 20:15:14.304917683 +0000 UTC m=+0.016749648 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-20T20:15:14.694 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:14 smithi078 podman[55975]: 2024-09-20 20:15:14.6172316 +0000 UTC m=+0.329063557 volume create f0d693c3f77d0f5b51098a329b54f52c1ce1f43823c7c8bafb7a50cc356243b1 2024-09-20T20:15:14.695 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:14 smithi078 podman[55975]: 2024-09-20 20:15:14.692575894 +0000 UTC m=+0.404407853 container create df977df631df1bd0df6499f22025b40e53be6022169103aaed48aa8d8df24c23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:14.722 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:15:14.722 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false},"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-09-20T20:15:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:15 smithi146 ceph-mon[25603]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:15 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1715328683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-mon[23685]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1715328683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-mon[30028]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1715328683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 podman[55975]: 2024-09-20 20:15:15.662113501 +0000 UTC m=+1.373945462 container init df977df631df1bd0df6499f22025b40e53be6022169103aaed48aa8d8df24c23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 podman[55975]: 2024-09-20 20:15:15.665140293 +0000 UTC m=+1.376972260 container start df977df631df1bd0df6499f22025b40e53be6022169103aaed48aa8d8df24c23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.677Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.677Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.678Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.78 port=9094 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.679Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.701Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.701Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T20:15:15.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.705Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T20:15:15.975 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:15.705Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T20:15:15.975 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 bash[55975]: df977df631df1bd0df6499f22025b40e53be6022169103aaed48aa8d8df24c23 2024-09-20T20:15:15.975 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:15 smithi078 systemd[1]: Started Ceph alertmanager.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:15:16.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[30028]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[30028]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[30028]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[23685]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[23685]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:15:16.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:16 smithi078 ceph-mon[23685]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:16 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:16 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:16 smithi146 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:15:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:16 smithi146 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:15:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:16 smithi146 ceph-mon[25603]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:18.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:17 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:17.679Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000461312s 2024-09-20T20:15:18.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T20:15:19.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:18 smithi146 systemd[1]: Stopping Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:15:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:19.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:19 smithi078 ceph-mon[30028]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:19.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:19 smithi078 ceph-mon[23685]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:19.587 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-mon[25603]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.309Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T20:15:19.587 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T20:15:19.588 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T20:15:19.588 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T20:15:19.588 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T20:15:19.588 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T20:15:19.588 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[44706]: ts=2024-09-20T20:15:19.310Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T20:15:19.588 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 podman[52838]: 2024-09-20 20:15:19.320158297 +0000 UTC m=+0.298656818 container died f480cbdaafe9bbea9067ffe5d842bfbee64f7928777fd0d1e611becaa8f53b9d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:19.953 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 podman[52838]: 2024-09-20 20:15:19.692675926 +0000 UTC m=+0.671174437 container cleanup f480cbdaafe9bbea9067ffe5d842bfbee64f7928777fd0d1e611becaa8f53b9d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:19.953 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 bash[52838]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T20:15:20.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:19 smithi146 podman[52850]: 2024-09-20 20:15:19.952098943 +0000 UTC m=+0.629218443 container remove f480cbdaafe9bbea9067ffe5d842bfbee64f7928777fd0d1e611becaa8f53b9d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:20.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T20:15:20.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 systemd[1]: Stopped Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:15:20.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.196s CPU time. 2024-09-20T20:15:20.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 systemd[1]: Starting Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:15:20.611 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 podman[52943]: 2024-09-20 20:15:20.233841253 +0000 UTC m=+0.017024223 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T20:15:20.611 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 podman[52943]: 2024-09-20 20:15:20.35308496 +0000 UTC m=+0.136267911 container create f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:20.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE Bus STOPPING 2024-09-20T20:15:20.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:15:20.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE Bus STOPPED 2024-09-20T20:15:20.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE Bus STARTING 2024-09-20T20:15:20.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE Serving on http://:::9283 2024-09-20T20:15:20.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE Bus STARTED 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 podman[52943]: 2024-09-20 20:15:20.609855606 +0000 UTC m=+0.393038567 container init f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 podman[52943]: 2024-09-20 20:15:20.612583883 +0000 UTC m=+0.395766836 container start f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.635Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.635Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.635Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi146 (none))" 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.635Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T20:15:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.635Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.638Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.638Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.639Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.639Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.641Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.641Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.285µs 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.641Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.643Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.321µs wal_replay_duration=1.947432ms wbl_replay_duration=139ns total_replay_duration=2.065736ms 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.644Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.644Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T20:15:21.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.644Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T20:15:21.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.664Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.072695ms db_storage=725ns remote_storage=984ns web_handler=360ns query_engine=482ns scrape=612.41µs scrape_sd=93.14µs notify=14.461µs notify_sd=22.004µs rules=18.998881ms tracing=5.585µs 2024-09-20T20:15:21.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.664Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T20:15:21.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:15:20.664Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T20:15:21.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 bash[52943]: f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd 2024-09-20T20:15:21.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:15:20 smithi146 systemd[1]: Started Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:15:21.095 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:15:21.095 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false},"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-09-20T20:15:21.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:20] ENGINE Bus STOPPING 2024-09-20T20:15:21.707 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:15:21.707 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:21] ENGINE Bus STOPPED 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:21] ENGINE Bus STARTING 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:21] ENGINE Serving on http://:::9283 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:21] ENGINE Bus STARTED 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:21] ENGINE Bus STOPPING 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T20:15:21.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2676382411' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:15:21.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:21.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:21.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:21 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2676382411' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:15:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:21.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:21.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:21.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:21 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2676382411' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:22.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:15:22.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:22] ENGINE Bus STOPPED 2024-09-20T20:15:22.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:22] ENGINE Bus STARTING 2024-09-20T20:15:22.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:22] ENGINE Serving on http://:::9283 2024-09-20T20:15:22.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:15:22] ENGINE Bus STARTED 2024-09-20T20:15:22.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T20:15:22.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T20:15:22.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[30028]: Cluster is now healthy 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T20:15:22.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-20T20:15:22.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:22 smithi078 ceph-mon[23685]: Cluster is now healthy 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi078.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-20T20:15:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:22 smithi146 ceph-mon[25603]: Cluster is now healthy 2024-09-20T20:15:24.729 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T20:15:24.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[30028]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:24.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[23685]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:24.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:24 smithi146 ceph-mon[25603]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:24.949 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:26.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:15:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:15:25.681Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002409164s 2024-09-20T20:15:26.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:26 smithi078 ceph-mon[23685]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:26.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:26 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:26.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:26 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:26.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:26 smithi078 ceph-mon[30028]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:26.628 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:26 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:26.628 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:26 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:26 smithi146 ceph-mon[25603]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:26 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:26 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-20T20:15:26.817 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T20:15:27.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:27 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3406017024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:27.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:27 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3406017024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:27.536 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T20:15:27.536 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T20:15:27.536 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T20:15:27.545 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-20T20:15:27.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-20T20:15:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:27 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3406017024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T20:15:27.780 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:28.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:28 smithi078 ceph-mon[23685]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:28.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:28 smithi078 ceph-mon[30028]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:28 smithi146 ceph-mon[25603]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:30 smithi146 ceph-mon[25603]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:30 smithi078 ceph-mon[23685]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:15:30] "GET /metrics HTTP/1.1" 200 26267 "" "Prometheus/2.43.0" 2024-09-20T20:15:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:30 smithi078 ceph-mon[30028]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:31.426 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:15:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "718c15e6-0bbe-4f04-88de-bfe70d2a1770", 2024-09-20T20:15:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "r", 2024-09-20T20:15:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "current_period": "ea368f25-2729-45bd-8b8c-5f7799b8d08d", 2024-09-20T20:15:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1 2024-09-20T20:15:31.426 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:15:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:31 smithi146 ceph-mon[25603]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T20:15:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:31 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3292601099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T20:15:31.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:31 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T20:15:31.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:31 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T20:15:31.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:31 smithi146 ceph-mon[25603]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T20:15:31.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[30028]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T20:15:31.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3292601099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[30028]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[23685]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3292601099' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T20:15:31.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:31 smithi078 ceph-mon[23685]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T20:15:32.434 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-20T20:15:32.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:32 smithi146 ceph-mon[25603]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:32.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:32 smithi146 ceph-mon[25603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:15:32.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:32 smithi146 ceph-mon[25603]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T20:15:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:32 smithi078 ceph-mon[23685]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:32 smithi078 ceph-mon[23685]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:15:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:32 smithi078 ceph-mon[23685]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T20:15:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:32 smithi078 ceph-mon[30028]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:32 smithi078 ceph-mon[30028]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:15:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:32 smithi078 ceph-mon[30028]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T20:15:33.179 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:33.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:33 smithi078 ceph-mon[30028]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:15:33.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:33 smithi078 ceph-mon[30028]: Cluster is now healthy 2024-09-20T20:15:33.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:33 smithi078 ceph-mon[23685]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:15:33.662 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:33 smithi078 ceph-mon[23685]: Cluster is now healthy 2024-09-20T20:15:33.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:33 smithi146 ceph-mon[25603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:15:33.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:33 smithi146 ceph-mon[25603]: Cluster is now healthy 2024-09-20T20:15:34.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:34 smithi078 ceph-mon[30028]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:34.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:34 smithi078 ceph-mon[23685]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:34.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:34 smithi146 ceph-mon[25603]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:35.556 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:15:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "b93656fa-fc25-4cbf-ab2a-123fed4e24c4", 2024-09-20T20:15:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default", 2024-09-20T20:15:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "default", 2024-09-20T20:15:35.556 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": true, 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "", 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [], 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [] 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:15:35.557 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "718c15e6-0bbe-4f04-88de-bfe70d2a1770", 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled_features": [ 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: "resharding" 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-20T20:15:35.558 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:15:36.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-20T20:15:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:36 smithi146 ceph-mon[25603]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:36 smithi078 ceph-mon[23685]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:36 smithi078 ceph-mon[30028]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:36.884 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:38.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:15:38.406 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "19202a97-cc1e-44c9-af2b-02c84a14be85", 2024-09-20T20:15:38.406 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2024-09-20T20:15:38.406 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "z.rgw.meta:root", 2024-09-20T20:15:38.406 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "z.rgw.control", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "z.rgw.log", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "z.rgw.otp", 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2024-09-20T20:15:38.407 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "", 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "" 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:38.408 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0, 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: "inline_data": true 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "718c15e6-0bbe-4f04-88de-bfe70d2a1770", 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-20T20:15:38.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:15:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:38 smithi146 ceph-mon[25603]: pgmap v59: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 318 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-20T20:15:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:38 smithi078 ceph-mon[23685]: pgmap v59: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 318 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-20T20:15:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:38 smithi078 ceph-mon[30028]: pgmap v59: 33 pgs: 33 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 318 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-20T20:15:39.107 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-20T20:15:39.304 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:40.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:40 smithi078 ceph-mon[23685]: pgmap v60: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T20:15:40.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:40 smithi078 ceph-mon[30028]: pgmap v60: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T20:15:40.738 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:15:40] "GET /metrics HTTP/1.1" 200 26267 "" "Prometheus/2.43.0" 2024-09-20T20:15:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:40 smithi146 ceph-mon[25603]: pgmap v60: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T20:15:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:41 smithi146 ceph-mon[25603]: pgmap v61: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 543 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[23685]: pgmap v61: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 543 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[30028]: pgmap v61: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 543 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-20T20:15:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:42 smithi146 ceph-mon[25603]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T20:15:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1421655449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T20:15:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:42 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T20:15:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:42 smithi078 ceph-mon[23685]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T20:15:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1421655449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T20:15:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:42 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T20:15:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:42 smithi078 ceph-mon[30028]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T20:15:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1421655449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T20:15:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:42 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T20:15:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:43 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T20:15:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:43 smithi146 ceph-mon[25603]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T20:15:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:43 smithi146 ceph-mon[25603]: pgmap v64: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-09-20T20:15:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:43 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T20:15:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:43 smithi078 ceph-mon[30028]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T20:15:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:43 smithi078 ceph-mon[30028]: pgmap v64: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-09-20T20:15:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:43 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T20:15:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:43 smithi078 ceph-mon[23685]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T20:15:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:43 smithi078 ceph-mon[23685]: pgmap v64: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-09-20T20:15:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:44 smithi146 ceph-mon[25603]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T20:15:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:44 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1421655449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T20:15:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:44 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T20:15:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:44 smithi078 ceph-mon[30028]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T20:15:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:44 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1421655449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T20:15:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:44 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T20:15:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:44 smithi078 ceph-mon[23685]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T20:15:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:44 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1421655449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T20:15:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:44 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T20:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:45 smithi146 ceph-mon[25603]: pgmap v66: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-20T20:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:45 smithi146 ceph-mon[25603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:45 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T20:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:45 smithi146 ceph-mon[25603]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[30028]: pgmap v66: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[30028]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[30028]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[23685]: pgmap v66: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[23685]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T20:15:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:45 smithi078 ceph-mon[23685]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T20:15:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:46 smithi146 ceph-mon[25603]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T20:15:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:46 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/409789574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T20:15:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:46 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T20:15:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:46 smithi078 ceph-mon[23685]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T20:15:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:46 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/409789574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T20:15:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:46 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T20:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:46 smithi078 ceph-mon[30028]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T20:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:46 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/409789574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T20:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:46 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T20:15:47.910 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:15:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "fefa91a8-97ed-4c2a-8aa3-419f83516460", 2024-09-20T20:15:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-20T20:15:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "ea368f25-2729-45bd-8b8c-5f7799b8d08d", 2024-09-20T20:15:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [], 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "fefa91a8-97ed-4c2a-8aa3-419f83516460", 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "b93656fa-fc25-4cbf-ab2a-123fed4e24c4", 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default", 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "default", 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": true, 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "19202a97-cc1e-44c9-af2b-02c84a14be85", 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-20T20:15:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "19202a97-cc1e-44c9-af2b-02c84a14be85", 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": false, 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": false, 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": false, 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": true, 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "", 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "supported_features": [ 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "compress-encrypted", 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "resharding" 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-20T20:15:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "718c15e6-0bbe-4f04-88de-bfe70d2a1770", 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2024-09-20T20:15:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled_features": [ 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "resharding" 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "19202a97-cc1e-44c9-af2b-02c84a14be85", 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 3175090888 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "b93656fa-fc25-4cbf-ab2a-123fed4e24c4", 2024-09-20T20:15:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "19202a97-cc1e-44c9-af2b-02c84a14be85", 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2024-09-20T20:15:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "user_ratelimit": { 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_read_ops": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_write_ops": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_read_bytes": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_write_bytes": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_ratelimit": { 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_read_ops": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_write_ops": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_read_bytes": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_write_bytes": 0, 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false 2024-09-20T20:15:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "anonymous_ratelimit": { 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max_read_ops": 0, 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max_write_ops": 0, 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max_read_bytes": 0, 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max_write_bytes": 0, 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "718c15e6-0bbe-4f04-88de-bfe70d2a1770", 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "r", 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2024-09-20T20:15:47.917 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:15:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:47 smithi146 ceph-mon[25603]: pgmap v69: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:47 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T20:15:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:47 smithi146 ceph-mon[25603]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T20:15:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:47 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/409789574' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T20:15:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:47 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[23685]: pgmap v69: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[23685]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/409789574' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[30028]: pgmap v69: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[30028]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T20:15:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/409789574' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T20:15:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:47 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T20:15:48.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-20T20:15:48.980 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:48 smithi078 ceph-mon[23685]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T20:15:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:48 smithi078 ceph-mon[23685]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T20:15:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:48 smithi078 ceph-mon[30028]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T20:15:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:48 smithi078 ceph-mon[30028]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T20:15:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:48 smithi146 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T20:15:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:48 smithi146 ceph-mon[25603]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T20:15:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:49 smithi078 ceph-mon[23685]: pgmap v72: 129 pgs: 129 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T20:15:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:49 smithi078 ceph-mon[23685]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:15:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:49 smithi078 ceph-mon[23685]: Cluster is now healthy 2024-09-20T20:15:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:49 smithi078 ceph-mon[30028]: pgmap v72: 129 pgs: 129 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T20:15:50.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:49 smithi078 ceph-mon[30028]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:15:50.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:49 smithi078 ceph-mon[30028]: Cluster is now healthy 2024-09-20T20:15:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:49 smithi146 ceph-mon[25603]: pgmap v72: 129 pgs: 129 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-20T20:15:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:49 smithi146 ceph-mon[25603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:15:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:49 smithi146 ceph-mon[25603]: Cluster is now healthy 2024-09-20T20:15:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:15:50] "GET /metrics HTTP/1.1" 200 28597 "" "Prometheus/2.43.0" 2024-09-20T20:15:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled rgw.foo update... 2024-09-20T20:15:51.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch apply rgw smpl' 2024-09-20T20:15:51.896 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:52.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[30028]: pgmap v73: 129 pgs: 129 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 1 op/s 2024-09-20T20:15:52.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[30028]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[30028]: Saving service rgw.foo spec with placement count:2 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[23685]: pgmap v73: 129 pgs: 129 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 1 op/s 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[23685]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[23685]: Saving service rgw.foo spec with placement count:2 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:51 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:15:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:51 smithi146 ceph-mon[25603]: pgmap v73: 129 pgs: 129 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 1 op/s 2024-09-20T20:15:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:51 smithi146 ceph-mon[25603]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:51 smithi146 ceph-mon[25603]: Saving service rgw.foo spec with placement count:2 2024-09-20T20:15:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:51 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:51 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:15:54.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:53 smithi078 ceph-mon[23685]: pgmap v74: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-20T20:15:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:53 smithi078 ceph-mon[30028]: pgmap v74: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-20T20:15:54.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:53 smithi146 ceph-mon[25603]: pgmap v74: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-20T20:15:55.114 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled rgw.smpl update... 2024-09-20T20:15:55.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph osd pool create foo' 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[30028]: pgmap v75: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[30028]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[30028]: Saving service rgw.smpl spec with placement count:2 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[23685]: pgmap v75: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[23685]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[23685]: Saving service rgw.smpl spec with placement count:2 2024-09-20T20:15:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:56 smithi146 ceph-mon[25603]: pgmap v75: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-20T20:15:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:56 smithi146 ceph-mon[25603]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:15:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:56 smithi146 ceph-mon[25603]: Saving service rgw.smpl spec with placement count:2 2024-09-20T20:15:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:56.664 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:15:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T20:15:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T20:15:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T20:15:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T20:15:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:15:57.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:15:57.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:15:57.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:15:57.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:15:58.386 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: pgmap v76: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: Deploying daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-20T20:15:58.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:58 smithi146 ceph-mon[25603]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T20:15:58.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: pgmap v76: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: Deploying daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[30028]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: pgmap v76: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-20T20:15:58.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: Deploying daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:15:58.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:15:58.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T20:15:58.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T20:15:58.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T20:15:58.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-09-20T20:15:58.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:58 smithi078 ceph-mon[23685]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T20:15:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'foo' created 2024-09-20T20:15:59.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:59 smithi146 ceph-mon[25603]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T20:15:59.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:15:59 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3017942491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T20:15:59.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:59 smithi078 ceph-mon[23685]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T20:15:59.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:15:59 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3017942491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T20:15:59.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:59 smithi078 ceph-mon[30028]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T20:15:59.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:15:59 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3017942491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T20:15:59.890 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'rbd pool init foo' 2024-09-20T20:16:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: pgmap v80: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3017942491' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: pgmap v80: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:00.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3017942491' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: pgmap v80: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3017942491' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:00.915 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:16:00] "GET /metrics HTTP/1.1" 200 35492 "" "Prometheus/2.43.0" 2024-09-20T20:16:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[30028]: Deploying daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[30028]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[23685]: Deploying daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[23685]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:01 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.292 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:16:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:01 smithi146 ceph-mon[25603]: Deploying daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:16:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:01 smithi146 ceph-mon[25603]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T20:16:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:01 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:01 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:01 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:02.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:02 smithi078 ceph-mon[30028]: pgmap v83: 161 pgs: 32 unknown, 4 peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:02.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:02 smithi078 ceph-mon[30028]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T20:16:02.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:02 smithi078 ceph-mon[30028]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:16:02.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:02 smithi078 ceph-mon[23685]: pgmap v83: 161 pgs: 32 unknown, 4 peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:02.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:02 smithi078 ceph-mon[23685]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T20:16:02.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:02 smithi078 ceph-mon[23685]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:16:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:02 smithi146 ceph-mon[25603]: pgmap v83: 161 pgs: 32 unknown, 4 peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:02 smithi146 ceph-mon[25603]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T20:16:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:02 smithi146 ceph-mon[25603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T20:16:03.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:03 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2082618818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T20:16:03.366 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:03 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2082618818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T20:16:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:03 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2082618818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 6.8 KiB/s wr, 229 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2082618818' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 6.8 KiB/s wr, 229 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2082618818' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:04 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 6.8 KiB/s wr, 229 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2082618818' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:04.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:04 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:05 smithi146 ceph-mon[25603]: Saving service rgw.foo spec with placement count:2 2024-09-20T20:16:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:05 smithi146 ceph-mon[25603]: Deploying daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:16:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:05 smithi146 ceph-mon[25603]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T20:16:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:05 smithi146 ceph-mon[25603]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[30028]: Saving service rgw.foo spec with placement count:2 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[30028]: Deploying daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[30028]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[30028]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[23685]: Saving service rgw.foo spec with placement count:2 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[23685]: Deploying daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:16:05.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[23685]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T20:16:05.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:05 smithi078 ceph-mon[23685]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T20:16:06.261 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-20T20:16:06.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:06 smithi078 ceph-mon[23685]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 230 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:06.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:06 smithi078 ceph-mon[23685]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:16:06.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:06 smithi078 ceph-mon[30028]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 230 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:06.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:06 smithi078 ceph-mon[30028]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:16:06.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:06 smithi146 ceph-mon[25603]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 230 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:06.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:06 smithi146 ceph-mon[25603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T20:16:07.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:07 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:07 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:07 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:07.733 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: pgmap v90: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 5.6 KiB/s wr, 191 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:08 smithi146 ceph-mon[25603]: Deploying daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:16:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: pgmap v90: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 5.6 KiB/s wr, 191 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[23685]: Deploying daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: pgmap v90: 161 pgs: 161 active+clean; 584 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 5.6 KiB/s wr, 191 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:08.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:08.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:08 smithi078 ceph-mon[30028]: Deploying daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:16:10.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: pgmap v91: 161 pgs: 161 active+clean; 584 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 682 B/s wr, 169 op/s 2024-09-20T20:16:10.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T20:16:10.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: pgmap v91: 161 pgs: 161 active+clean; 584 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 682 B/s wr, 169 op/s 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:10.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.761 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:16:10] "GET /metrics HTTP/1.1" 200 35492 "" "Prometheus/2.43.0" 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: pgmap v91: 161 pgs: 161 active+clean; 584 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 682 B/s wr, 169 op/s 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:10 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[23685]: Saving service rgw.smpl spec with placement count:2 2024-09-20T20:16:11.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:11.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.587 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[30028]: Saving service rgw.smpl spec with placement count:2 2024-09-20T20:16:11.587 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.587 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:11.587 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:11 smithi146 ceph-mon[25603]: Saving service rgw.smpl spec with placement count:2 2024-09-20T20:16:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:11.881 INFO:teuthology.orchestra.run.smithi078.stdout:Scheduled iscsi.foo update... 2024-09-20T20:16:12.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:12 smithi146 ceph-mon[25603]: pgmap v93: 161 pgs: 161 active+clean; 584 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 598 B/s wr, 148 op/s 2024-09-20T20:16:12.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:12 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:12.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:12 smithi078 ceph-mon[30028]: pgmap v93: 161 pgs: 161 active+clean; 584 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 598 B/s wr, 148 op/s 2024-09-20T20:16:12.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:12 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:12.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:12 smithi078 ceph-mon[23685]: pgmap v93: 161 pgs: 161 active+clean; 584 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 598 B/s wr, 148 op/s 2024-09-20T20:16:12.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:12 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:12.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'sleep 120' 2024-09-20T20:16:13.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:13 smithi078 ceph-mon[30028]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:13.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:13 smithi078 ceph-mon[30028]: Saving service iscsi.foo spec with placement count:1 2024-09-20T20:16:13.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:13 smithi078 ceph-mon[23685]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:13.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:13 smithi078 ceph-mon[23685]: Saving service iscsi.foo spec with placement count:1 2024-09-20T20:16:13.765 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:16:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:13 smithi146 ceph-mon[25603]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:16:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:13 smithi146 ceph-mon[25603]: Saving service iscsi.foo spec with placement count:1 2024-09-20T20:16:14.570 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:14 smithi146 ceph-mon[25603]: pgmap v94: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 241 KiB/s rd, 639 B/s wr, 373 op/s 2024-09-20T20:16:14.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:14 smithi078 ceph-mon[23685]: pgmap v94: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 241 KiB/s rd, 639 B/s wr, 373 op/s 2024-09-20T20:16:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:14 smithi078 ceph-mon[30028]: pgmap v94: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 241 KiB/s rd, 639 B/s wr, 373 op/s 2024-09-20T20:16:15.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:15 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:15.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:15 smithi078 ceph-mon[23685]: pgmap v95: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 196 KiB/s rd, 519 B/s wr, 303 op/s 2024-09-20T20:16:15.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:15 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:15.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:15 smithi078 ceph-mon[30028]: pgmap v95: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 196 KiB/s rd, 519 B/s wr, 303 op/s 2024-09-20T20:16:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:15 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:15 smithi146 ceph-mon[25603]: pgmap v95: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 196 KiB/s rd, 519 B/s wr, 303 op/s 2024-09-20T20:16:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[23685]: pgmap v96: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 511 B/s wr, 298 op/s 2024-09-20T20:16:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[30028]: pgmap v96: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 511 B/s wr, 298 op/s 2024-09-20T20:16:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:18 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:18 smithi146 ceph-mon[25603]: pgmap v96: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 511 B/s wr, 298 op/s 2024-09-20T20:16:18.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:18 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:18 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:18 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:18.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:18 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:19 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:19 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:19 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:19 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:19 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:20 smithi146 ceph-mon[25603]: pgmap v97: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 102 B/s wr, 196 op/s 2024-09-20T20:16:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:20 smithi078 ceph-mon[23685]: pgmap v97: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 102 B/s wr, 196 op/s 2024-09-20T20:16:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:20 smithi078 ceph-mon[30028]: pgmap v97: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 102 B/s wr, 196 op/s 2024-09-20T20:16:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:16:20] "GET /metrics HTTP/1.1" 200 38321 "" "Prometheus/2.43.0" 2024-09-20T20:16:22.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:22 smithi146 ceph-mon[25603]: pgmap v98: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 90 B/s wr, 173 op/s 2024-09-20T20:16:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:22 smithi078 ceph-mon[23685]: pgmap v98: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 90 B/s wr, 173 op/s 2024-09-20T20:16:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:22 smithi078 ceph-mon[30028]: pgmap v98: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 90 B/s wr, 173 op/s 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE Bus STOPPING 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE Bus STOPPED 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE Bus STARTING 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE Serving on http://:::9283 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE Bus STARTED 2024-09-20T20:16:23.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:23] ENGINE Bus STOPPING 2024-09-20T20:16:24.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: pgmap v99: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 85 B/s wr, 164 op/s 2024-09-20T20:16:24.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: pgmap v100: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: pgmap v101: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: Checking dashboard <-> RGW credentials 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T20:16:24.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: pgmap v99: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 85 B/s wr, 164 op/s 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: pgmap v100: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: pgmap v101: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: Checking dashboard <-> RGW credentials 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:16:24.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T20:16:24.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: pgmap v99: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 85 B/s wr, 164 op/s 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: pgmap v100: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: pgmap v101: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: Checking dashboard <-> RGW credentials 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:16:24.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T20:16:24.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:24 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:24.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:16:24.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:24] ENGINE Bus STOPPED 2024-09-20T20:16:24.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:24] ENGINE Bus STARTING 2024-09-20T20:16:24.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:24] ENGINE Serving on http://:::9283 2024-09-20T20:16:24.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:24] ENGINE Bus STARTED 2024-09-20T20:16:25.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:25 smithi078 ceph-mon[30028]: Deploying daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:16:25.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:25 smithi078 ceph-mon[23685]: Deploying daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:16:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:25 smithi146 ceph-mon[25603]: Deploying daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:16:26.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:26 smithi078 ceph-mon[30028]: pgmap v102: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:26.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:26 smithi078 ceph-mon[23685]: pgmap v102: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:26 smithi146 ceph-mon[25603]: pgmap v102: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:27.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/282972941' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:27.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:27 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/282972941' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:27 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/282972941' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:16:28.487 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:28 smithi146 ceph-mon[25603]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T20:16:28.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:28 smithi146 ceph-mon[25603]: pgmap v103: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:28.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:28 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/979332302' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/294299824"}]: dispatch 2024-09-20T20:16:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:28 smithi078 ceph-mon[23685]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T20:16:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:28 smithi078 ceph-mon[23685]: pgmap v103: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:28 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/979332302' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/294299824"}]: dispatch 2024-09-20T20:16:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:28 smithi078 ceph-mon[30028]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T20:16:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:28 smithi078 ceph-mon[30028]: pgmap v103: 161 pgs: 161 active+clean; 584 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:28 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/979332302' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/294299824"}]: dispatch 2024-09-20T20:16:29.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/979332302' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/294299824"}]': finished 2024-09-20T20:16:29.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[23685]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T20:16:29.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2269759165' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]: dispatch 2024-09-20T20:16:29.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]: dispatch 2024-09-20T20:16:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:29 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/979332302' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/294299824"}]': finished 2024-09-20T20:16:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:29 smithi146 ceph-mon[25603]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T20:16:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:29 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2269759165' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]: dispatch 2024-09-20T20:16:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:29 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]: dispatch 2024-09-20T20:16:29.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/979332302' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/294299824"}]': finished 2024-09-20T20:16:29.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[30028]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T20:16:29.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2269759165' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]: dispatch 2024-09-20T20:16:29.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:29 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]: dispatch 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[30028]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 494 B/s wr, 6 op/s 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]': finished 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[30028]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[30028]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/438889281' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/2188634654"}]: dispatch 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[23685]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 494 B/s wr, 6 op/s 2024-09-20T20:16:30.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]': finished 2024-09-20T20:16:30.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[23685]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T20:16:30.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[23685]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-20T20:16:30.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/438889281' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/2188634654"}]: dispatch 2024-09-20T20:16:30.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:30 smithi146 ceph-mon[25603]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 494 B/s wr, 6 op/s 2024-09-20T20:16:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:30 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/674566149"}]': finished 2024-09-20T20:16:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:30 smithi146 ceph-mon[25603]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T20:16:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:30 smithi146 ceph-mon[25603]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-20T20:16:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:30 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/438889281' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/2188634654"}]: dispatch 2024-09-20T20:16:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:30 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:30.535 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:30] ENGINE Bus STOPPING 2024-09-20T20:16:30.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:16:30] "GET /metrics HTTP/1.1" 200 38389 "" "Prometheus/2.43.0" 2024-09-20T20:16:30.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:16:30.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:30] ENGINE Bus STOPPED 2024-09-20T20:16:30.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:30] ENGINE Bus STARTING 2024-09-20T20:16:30.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:30] ENGINE Serving on http://:::9283 2024-09-20T20:16:30.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:16:30] ENGINE Bus STARTED 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/438889281' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/2188634654"}]': finished 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 599 B/s wr, 7 op/s 2024-09-20T20:16:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 983 B/s wr, 12 op/s 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: Adding iSCSI gateway http://:@172.21.15.78:5000 to Dashboard 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi078"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi078"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:31.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:31 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2996112301' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/2188634654"}]: dispatch 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/438889281' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/2188634654"}]': finished 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 599 B/s wr, 7 op/s 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 983 B/s wr, 12 op/s 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T20:16:31.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: Adding iSCSI gateway http://:@172.21.15.78:5000 to Dashboard 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi078"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi078"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2996112301' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/2188634654"}]: dispatch 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/438889281' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/2188634654"}]': finished 2024-09-20T20:16:31.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 599 B/s wr, 7 op/s 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 983 B/s wr, 12 op/s 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: Adding iSCSI gateway http://:@172.21.15.78:5000 to Dashboard 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi078"}]: dispatch 2024-09-20T20:16:31.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi078"}]: dispatch 2024-09-20T20:16:31.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:31.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:31.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:31 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2996112301' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/2188634654"}]: dispatch 2024-09-20T20:16:32.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:32 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:32.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:32 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2996112301' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/2188634654"}]': finished 2024-09-20T20:16:32.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:32 smithi146 ceph-mon[25603]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T20:16:32.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:32 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/4143123150' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3374519687"}]: dispatch 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2996112301' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/2188634654"}]': finished 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[30028]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/4143123150' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3374519687"}]: dispatch 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2996112301' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/2188634654"}]': finished 2024-09-20T20:16:32.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[23685]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T20:16:32.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:32 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/4143123150' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3374519687"}]: dispatch 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/4143123150' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3374519687"}]': finished 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/587520174' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/643935292"}]: dispatch 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:33 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/4143123150' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3374519687"}]': finished 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/587520174' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/643935292"}]: dispatch 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/4143123150' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3374519687"}]': finished 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/587520174' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/643935292"}]: dispatch 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:33 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:34.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:34 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/587520174' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/643935292"}]': finished 2024-09-20T20:16:34.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:34 smithi146 ceph-mon[25603]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T20:16:34.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:34 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3626295384' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]: dispatch 2024-09-20T20:16:34.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:34 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]: dispatch 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/587520174' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/643935292"}]': finished 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[30028]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3626295384' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]: dispatch 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]: dispatch 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/587520174' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/643935292"}]': finished 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[23685]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T20:16:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3626295384' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]: dispatch 2024-09-20T20:16:34.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:34 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]: dispatch 2024-09-20T20:16:35.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:35 smithi146 ceph-mon[25603]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:35.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:35 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]': finished 2024-09-20T20:16:35.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:35 smithi146 ceph-mon[25603]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T20:16:35.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:35 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/317947766' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2196865576"}]: dispatch 2024-09-20T20:16:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[30028]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]': finished 2024-09-20T20:16:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[30028]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T20:16:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/317947766' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2196865576"}]: dispatch 2024-09-20T20:16:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[23685]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:35.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2835130118"}]': finished 2024-09-20T20:16:35.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[23685]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T20:16:35.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:35 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/317947766' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2196865576"}]: dispatch 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/317947766' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2196865576"}]': finished 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3068960042' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]: dispatch 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]: dispatch 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: Checking dashboard <-> RGW credentials 2024-09-20T20:16:36.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:36 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/317947766' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2196865576"}]': finished 2024-09-20T20:16:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T20:16:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3068960042' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]: dispatch 2024-09-20T20:16:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]: dispatch 2024-09-20T20:16:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: Checking dashboard <-> RGW credentials 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/317947766' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2196865576"}]': finished 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3068960042' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]: dispatch 2024-09-20T20:16:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]: dispatch 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: Checking dashboard <-> RGW credentials 2024-09-20T20:16:36.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:36 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:37.501 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:37 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]': finished 2024-09-20T20:16:37.501 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:37 smithi146 ceph-mon[25603]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T20:16:37.501 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:37 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:37.501 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:37 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:37.501 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:37 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3627892649' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3702369749"}]: dispatch 2024-09-20T20:16:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]': finished 2024-09-20T20:16:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[30028]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T20:16:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3627892649' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3702369749"}]: dispatch 2024-09-20T20:16:37.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/3575808041"}]': finished 2024-09-20T20:16:37.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[23685]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T20:16:37.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:16:37.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:37.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:37 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3627892649' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3702369749"}]: dispatch 2024-09-20T20:16:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:38 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3627892649' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3702369749"}]': finished 2024-09-20T20:16:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:38 smithi146 ceph-mon[25603]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T20:16:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:38 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3349685177' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]: dispatch 2024-09-20T20:16:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:38 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]: dispatch 2024-09-20T20:16:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:38 smithi146 ceph-mon[25603]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-20T20:16:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3627892649' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3702369749"}]': finished 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[23685]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3349685177' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]: dispatch 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]: dispatch 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[23685]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3627892649' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3702369749"}]': finished 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[30028]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3349685177' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]: dispatch 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]: dispatch 2024-09-20T20:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:38 smithi078 ceph-mon[30028]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]': finished 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[23685]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/4173779419' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]: dispatch 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]: dispatch 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]': finished 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[30028]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T20:16:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/4173779419' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]: dispatch 2024-09-20T20:16:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:39 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]: dispatch 2024-09-20T20:16:39.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:39 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2460914245"}]': finished 2024-09-20T20:16:39.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:39 smithi146 ceph-mon[25603]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T20:16:39.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:39 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/4173779419' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]: dispatch 2024-09-20T20:16:39.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:39 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]: dispatch 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:16:40] "GET /metrics HTTP/1.1" 200 38389 "" "Prometheus/2.43.0" 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]': finished 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1803174239' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]: dispatch 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]: dispatch 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]': finished 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1803174239' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]: dispatch 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]: dispatch 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:40.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:40.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:40.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-20T20:16:40.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-20T20:16:40.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:40 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/4123000022"}]': finished 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1803174239' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]: dispatch 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]: dispatch 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 5 op/s 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-20T20:16:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:40 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]': finished 2024-09-20T20:16:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[30028]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T20:16:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2844046001' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2330055755"}]: dispatch 2024-09-20T20:16:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]': finished 2024-09-20T20:16:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[23685]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T20:16:41.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2844046001' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2330055755"}]: dispatch 2024-09-20T20:16:41.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:41 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/674566149"}]': finished 2024-09-20T20:16:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:41 smithi146 ceph-mon[25603]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T20:16:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:41 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2844046001' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2330055755"}]: dispatch 2024-09-20T20:16:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:42.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2844046001' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2330055755"}]': finished 2024-09-20T20:16:42.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[30028]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T20:16:42.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2649872849' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1372816629"}]: dispatch 2024-09-20T20:16:42.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[30028]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:42.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2844046001' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2330055755"}]': finished 2024-09-20T20:16:42.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[23685]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T20:16:42.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2649872849' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1372816629"}]: dispatch 2024-09-20T20:16:42.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:42 smithi078 ceph-mon[23685]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2844046001' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2330055755"}]': finished 2024-09-20T20:16:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:42 smithi146 ceph-mon[25603]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T20:16:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2649872849' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1372816629"}]: dispatch 2024-09-20T20:16:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:42 smithi146 ceph-mon[25603]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:16:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:43 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2649872849' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1372816629"}]': finished 2024-09-20T20:16:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:43 smithi146 ceph-mon[25603]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T20:16:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:43 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3016241797' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/3575808041"}]: dispatch 2024-09-20T20:16:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:43 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2649872849' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1372816629"}]': finished 2024-09-20T20:16:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:43 smithi078 ceph-mon[23685]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T20:16:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:43 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3016241797' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/3575808041"}]: dispatch 2024-09-20T20:16:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:43 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2649872849' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1372816629"}]': finished 2024-09-20T20:16:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:43 smithi078 ceph-mon[30028]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T20:16:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:43 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3016241797' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/3575808041"}]: dispatch 2024-09-20T20:16:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:44 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3016241797' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/3575808041"}]': finished 2024-09-20T20:16:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:44 smithi146 ceph-mon[25603]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T20:16:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:44 smithi146 ceph-mon[25603]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:44 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3016241797' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/3575808041"}]': finished 2024-09-20T20:16:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:44 smithi078 ceph-mon[30028]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T20:16:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:44 smithi078 ceph-mon[30028]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:44 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3016241797' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/3575808041"}]': finished 2024-09-20T20:16:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:44 smithi078 ceph-mon[23685]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T20:16:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:44 smithi078 ceph-mon[23685]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:47 smithi078 ceph-mon[23685]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-20T20:16:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:47 smithi078 ceph-mon[30028]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-20T20:16:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:47 smithi146 ceph-mon[25603]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.496 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:48 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:48 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:48 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:48 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:48 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:16:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:49 smithi078 ceph-mon[23685]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:16:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:49 smithi078 ceph-mon[30028]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:16:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:49 smithi146 ceph-mon[25603]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T20:16:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:16:50] "GET /metrics HTTP/1.1" 200 38429 "" "Prometheus/2.43.0" 2024-09-20T20:16:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:51 smithi078 ceph-mon[23685]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:51 smithi078 ceph-mon[30028]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:51 smithi146 ceph-mon[25603]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:53 smithi146 ceph-mon[25603]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-09-20T20:16:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:53 smithi078 ceph-mon[23685]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-09-20T20:16:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:53 smithi078 ceph-mon[30028]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-09-20T20:16:55.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:55 smithi146 ceph-mon[25603]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-09-20T20:16:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:55 smithi078 ceph-mon[23685]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-09-20T20:16:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:55 smithi078 ceph-mon[30028]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-09-20T20:16:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:57 smithi146 ceph-mon[25603]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:57 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:57 smithi078 ceph-mon[23685]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:57 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:57 smithi078 ceph-mon[30028]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:16:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:57 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:16:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:58 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:58 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:58 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:16:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:16:59 smithi146 ceph-mon[25603]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:16:59 smithi078 ceph-mon[23685]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:16:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:16:59 smithi078 ceph-mon[30028]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:17:00] "GET /metrics HTTP/1.1" 200 38434 "" "Prometheus/2.43.0" 2024-09-20T20:17:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:01 smithi146 ceph-mon[25603]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:01 smithi078 ceph-mon[23685]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:01 smithi078 ceph-mon[30028]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:03 smithi146 ceph-mon[25603]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:03 smithi078 ceph-mon[23685]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:03 smithi078 ceph-mon[30028]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:05 smithi146 ceph-mon[25603]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:05 smithi078 ceph-mon[30028]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:05 smithi078 ceph-mon[23685]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:07 smithi078 ceph-mon[30028]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:07 smithi078 ceph-mon[23685]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:07 smithi146 ceph-mon[25603]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:08 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:08 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:08 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[23685]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[30028]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:09 smithi146 ceph-mon[25603]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:17:10] "GET /metrics HTTP/1.1" 200 38434 "" "Prometheus/2.43.0" 2024-09-20T20:17:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:11 smithi146 ceph-mon[25603]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:11 smithi078 ceph-mon[23685]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:11 smithi078 ceph-mon[30028]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:13.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:13 smithi146 ceph-mon[25603]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:13 smithi078 ceph-mon[23685]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:13 smithi078 ceph-mon[30028]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:15 smithi146 ceph-mon[25603]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:15 smithi078 ceph-mon[23685]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:15 smithi078 ceph-mon[30028]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:17 smithi146 ceph-mon[25603]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:17 smithi078 ceph-mon[23685]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:17 smithi078 ceph-mon[30028]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:18 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:18 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:18 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:19 smithi146 ceph-mon[25603]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:19 smithi078 ceph-mon[23685]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:19 smithi078 ceph-mon[30028]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:17:20] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.43.0" 2024-09-20T20:17:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:21 smithi146 ceph-mon[25603]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:21 smithi078 ceph-mon[23685]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:21 smithi078 ceph-mon[30028]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:23 smithi146 ceph-mon[25603]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:23 smithi078 ceph-mon[23685]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:23 smithi078 ceph-mon[30028]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:25 smithi146 ceph-mon[25603]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:25 smithi078 ceph-mon[23685]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:25 smithi078 ceph-mon[30028]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:26.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:26 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:26.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:26 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:26 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:27.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:27 smithi146 ceph-mon[25603]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:27.761 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:27 smithi078 ceph-mon[30028]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:27.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:27 smithi078 ceph-mon[23685]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:28.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:28 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:28.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:28 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:28 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:29 smithi146 ceph-mon[25603]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[23685]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[30028]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:17:30] "GET /metrics HTTP/1.1" 200 38430 "" "Prometheus/2.43.0" 2024-09-20T20:17:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:31 smithi146 ceph-mon[25603]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:31 smithi078 ceph-mon[23685]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:31 smithi078 ceph-mon[30028]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:33 smithi146 ceph-mon[25603]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:33 smithi078 ceph-mon[23685]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:33 smithi078 ceph-mon[30028]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:35.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:35 smithi146 ceph-mon[25603]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:35 smithi078 ceph-mon[23685]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:35 smithi078 ceph-mon[30028]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:37 smithi146 ceph-mon[25603]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:37 smithi078 ceph-mon[23685]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:37 smithi078 ceph-mon[30028]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:38 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:38 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:38 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:39 smithi146 ceph-mon[25603]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:39 smithi078 ceph-mon[23685]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:39 smithi078 ceph-mon[30028]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:40 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:17:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:40 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:17:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:40 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:17:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:17:40] "GET /metrics HTTP/1.1" 200 38430 "" "Prometheus/2.43.0" 2024-09-20T20:17:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:41 smithi146 ceph-mon[25603]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:41 smithi078 ceph-mon[23685]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:41 smithi078 ceph-mon[30028]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:43 smithi146 ceph-mon[25603]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:43 smithi078 ceph-mon[23685]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:43 smithi078 ceph-mon[30028]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:44 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:44 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:47 smithi146 ceph-mon[25603]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-09-20T20:17:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:47 smithi078 ceph-mon[30028]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-09-20T20:17:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:47 smithi078 ceph-mon[23685]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-09-20T20:17:48.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:48 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:48.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:48 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:48 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[23685]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[30028]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:49 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:49 smithi146 ceph-mon[25603]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:49 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:49 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:49 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:17:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:17:50] "GET /metrics HTTP/1.1" 200 38429 "" "Prometheus/2.43.0" 2024-09-20T20:17:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:51 smithi146 ceph-mon[25603]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:51 smithi078 ceph-mon[23685]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:51 smithi078 ceph-mon[30028]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:53.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:53 smithi146 ceph-mon[25603]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:53 smithi078 ceph-mon[30028]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:53 smithi078 ceph-mon[23685]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:55.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:55 smithi146 ceph-mon[25603]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:17:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:55 smithi078 ceph-mon[23685]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:17:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:55 smithi078 ceph-mon[30028]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:17:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:56.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:56.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:17:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:57 smithi146 ceph-mon[25603]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:57 smithi078 ceph-mon[30028]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:57 smithi078 ceph-mon[23685]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:17:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:58 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:58 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:58 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:17:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:17:59 smithi146 ceph-mon[25603]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:17:59 smithi078 ceph-mon[23685]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:17:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:17:59 smithi078 ceph-mon[30028]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:18:00] "GET /metrics HTTP/1.1" 200 38430 "" "Prometheus/2.43.0" 2024-09-20T20:18:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:01 smithi146 ceph-mon[25603]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:01 smithi078 ceph-mon[23685]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:01 smithi078 ceph-mon[30028]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:03 smithi146 ceph-mon[25603]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:03 smithi078 ceph-mon[23685]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:03 smithi078 ceph-mon[30028]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:05.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:05 smithi146 ceph-mon[25603]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:05 smithi078 ceph-mon[23685]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:05 smithi078 ceph-mon[30028]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:07.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:07 smithi078 ceph-mon[23685]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:07.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:07 smithi078 ceph-mon[30028]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:07 smithi146 ceph-mon[25603]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:08.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:08 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:08.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:08 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:08 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:09 smithi146 ceph-mon[25603]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[23685]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[30028]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:18:10] "GET /metrics HTTP/1.1" 200 38430 "" "Prometheus/2.43.0" 2024-09-20T20:18:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:11 smithi146 ceph-mon[25603]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:11 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:11 smithi078 ceph-mon[23685]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:11 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:11 smithi078 ceph-mon[30028]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:11 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:13 smithi146 ceph-mon[25603]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:13 smithi078 ceph-mon[23685]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:13 smithi078 ceph-mon[30028]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:15 smithi146 ceph-mon[25603]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:15 smithi078 ceph-mon[23685]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:15 smithi078 ceph-mon[30028]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:17.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:17 smithi078 ceph-mon[30028]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:17.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:17 smithi078 ceph-mon[23685]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:17 smithi146 ceph-mon[25603]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:18.054 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-20T20:18:18.261 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:18:18.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:18 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:18.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:18 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:18 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:19.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:19 smithi078 ceph-mon[30028]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:19.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:19 smithi078 ceph-mon[23685]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:19 smithi146 ceph-mon[25603]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:20.710 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:18:20] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.43.0" 2024-09-20T20:18:20.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-20T20:18:20.930 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:18:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:21 smithi146 ceph-mon[25603]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:21 smithi078 ceph-mon[30028]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:21 smithi078 ceph-mon[23685]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:23.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-20T20:18:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:18:23.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:23 smithi078 ceph-mon[30028]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:23.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:23 smithi078 ceph-mon[23685]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:23 smithi146 ceph-mon[25603]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:25 smithi146 ceph-mon[25603]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:25 smithi078 ceph-mon[23685]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:25 smithi078 ceph-mon[30028]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:26.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-20T20:18:26.331 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:18:26.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:26 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:26.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:26 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:26 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:27.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:27 smithi078 ceph-mon[23685]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:27.665 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:27 smithi078 ceph-mon[30028]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:27 smithi146 ceph-mon[25603]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:28.200 INFO:teuthology.orchestra.run.smithi078.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:18:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:18:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:18:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:18:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:18:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:18:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:28 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:28 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:28 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:28 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:28 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:28 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:28 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:18:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.649 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T20:18:29.658 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-20T20:18:29.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2df6e16-778b-11ef-baf5-efdc52797490 -e sha1=e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:30.549 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T20:18:30.871 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:18:30] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.43.0" 2024-09-20T20:18:31.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:31 smithi078 ceph-mon[23685]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:31.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:31 smithi078 ceph-mon[30028]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:31 smithi146 ceph-mon[25603]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:33.358 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:18:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:33 smithi146 ceph-mon[25603]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-09-20T20:18:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:33 smithi078 ceph-mon[30028]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-09-20T20:18:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:33 smithi078 ceph-mon[23685]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 516 B/s rd, 0 op/s 2024-09-20T20:18:33.962 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 9m - - 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 error 49s ago 8m - - 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3m) 5s ago 4m 25.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4m) 49s ago 4m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2m) 5s ago 2m 55.1M - 3.8 4404209dfb71 e2de8476e360 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:8443,8765 running (8m) 49s ago 8m 439M - 18.2.4-869-g0b2cc4d4 4404209dfb71 b46623d9a752 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (9m) 5s ago 9m 501M - 18.2.4-869-g0b2cc4d4 4404209dfb71 a768f0dec912 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (10m) 5s ago 10m 49.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (8m) 49s ago 8m 43.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (8m) 5s ago 8m 41.9M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3m) 5s ago 3m 14.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:18:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3m) 49s ago 3m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (8m) 5s ago 8m 50.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7m) 5s ago 7m 59.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7m) 5s ago 7m 56.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6m) 5s ago 6m 56.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6m) 49s ago 6m 62.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6m) 49s ago 6m 55.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5m) 49s ago 5m 47.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5m) 49s ago 5m 60.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3m) 49s ago 4m 42.9M - 2.43.0 a07b618ecd1d f312fa9916d6 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2m) 5s ago 2m 76.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2m) 49s ago 2m 75.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2m) 5s ago 2m 76.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:18:33.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2m) 49s ago 2m 78.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:18:34.320 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:18:34.321 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:18:34.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:18:34.321 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:18:34.321 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 17 2024-09-20T20:18:34.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:18:34.321 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7", 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "", 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e8baca1eac6c3c9c88bf2ea4493e448bd54936a7 image", 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:18:34.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:18:34.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:34 smithi078 ceph-mon[30028]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:34.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:34 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1773621990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:18:34.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:34 smithi078 ceph-mon[23685]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:34.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:34 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1773621990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:18:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:34 smithi146 ceph-mon[25603]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:34 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1773621990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:18:34.991 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:18:34.992 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:18:34.992 INFO:teuthology.orchestra.run.smithi078.stdout: Cephadm agent on host smithi146 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T20:18:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:35 smithi146 ceph-mon[25603]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:35 smithi146 ceph-mon[25603]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:35 smithi146 ceph-mon[25603]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:35 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3042582153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[23685]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[23685]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[23685]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3042582153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[30028]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[30028]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[30028]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:35 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3042582153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:18:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:36 smithi146 ceph-mon[25603]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:36 smithi078 ceph-mon[23685]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:36 smithi078 ceph-mon[30028]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:18:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:37 smithi146 ceph-mon[25603]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:37 smithi078 ceph-mon[23685]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:37 smithi078 ceph-mon[30028]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:18:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:38 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:38 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:38 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:39 smithi146 ceph-mon[25603]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:18:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:39 smithi078 ceph-mon[23685]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:18:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:39 smithi078 ceph-mon[30028]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:18:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:18:40] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.43.0" 2024-09-20T20:18:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:41 smithi146 ceph-mon[25603]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:41 smithi078 ceph-mon[30028]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:41 smithi078 ceph-mon[23685]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:43 smithi146 ceph-mon[25603]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:43 smithi078 ceph-mon[23685]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:43 smithi078 ceph-mon[30028]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:45 smithi146 ceph-mon[25603]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:45 smithi078 ceph-mon[30028]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:45 smithi078 ceph-mon[23685]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:47 smithi146 ceph-mon[25603]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:47 smithi078 ceph-mon[30028]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:47 smithi078 ceph-mon[23685]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:48 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:48 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:48 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:49 smithi146 ceph-mon[25603]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:49 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:49 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:49 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[30028]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[23685]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:49.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:49 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:18:50.750 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:18:50] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.43.0" 2024-09-20T20:18:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:51 smithi146 ceph-mon[25603]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:51 smithi078 ceph-mon[23685]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:51 smithi078 ceph-mon[30028]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:53 smithi146 ceph-mon[25603]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:53 smithi078 ceph-mon[30028]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:53 smithi078 ceph-mon[23685]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:55.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:55 smithi146 ceph-mon[25603]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:55 smithi078 ceph-mon[30028]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:55 smithi078 ceph-mon[23685]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:56.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:56 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:56 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:56 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:18:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:57 smithi146 ceph-mon[25603]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:57 smithi078 ceph-mon[30028]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:57 smithi078 ceph-mon[23685]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:18:58.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:58 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:58 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:58 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:18:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:18:59 smithi146 ceph-mon[25603]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:18:59 smithi078 ceph-mon[23685]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:18:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:18:59 smithi078 ceph-mon[30028]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:00.977 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:19:00] "GET /metrics HTTP/1.1" 200 38435 "" "Prometheus/2.43.0" 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (unknown) 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[23685]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi146 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:01.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (unknown) 2024-09-20T20:19:01.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T20:19:01.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:01.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:01.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:19:01.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:01.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:00 smithi078 ceph-mon[30028]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi146 2024-09-20T20:19:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: Upgrade: Target is version 19.3.0-5127-ge8baca1e (unknown) 2024-09-20T20:19:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d'] 2024-09-20T20:19:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:19:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:00 smithi146 ceph-mon[25603]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d on smithi146 2024-09-20T20:19:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:03 smithi146 ceph-mon[25603]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:03 smithi078 ceph-mon[23685]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:03 smithi078 ceph-mon[30028]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:05.285 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:19:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:05 smithi078 ceph-mon[23685]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:05 smithi078 ceph-mon[30028]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:05 smithi146 ceph-mon[25603]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 9m - - 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 error 81s ago 9m - - 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3m) 16s ago 4m 25.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4m) 81s ago 4m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2m) 16s ago 2m 55.1M - 3.8 4404209dfb71 e2de8476e360 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:8443,8765 running (8m) 81s ago 8m 439M - 18.2.4-869-g0b2cc4d4 4404209dfb71 b46623d9a752 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (10m) 16s ago 10m 502M - 18.2.4-869-g0b2cc4d4 4404209dfb71 a768f0dec912 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (10m) 16s ago 10m 50.6M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:19:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (9m) 81s ago 9m 43.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (9m) 16s ago 9m 41.7M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4m) 16s ago 4m 13.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4m) 81s ago 4m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (8m) 16s ago 8m 51.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (8m) 16s ago 8m 59.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7m) 16s ago 7m 56.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7m) 16s ago 7m 56.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7m) 81s ago 7m 62.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6m) 81s ago 6m 55.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6m) 81s ago 6m 47.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5m) 81s ago 5m 60.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3m) 81s ago 5m 42.9M - 2.43.0 a07b618ecd1d f312fa9916d6 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3m) 16s ago 3m 76.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:19:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3m) 81s ago 3m 75.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:19:05.816 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2m) 16s ago 2m 76.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:19:05.816 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2m) 81s ago 2m 78.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:06.126 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:19:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:19:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:19:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 17 2024-09-20T20:19:06.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:19:06.127 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:19:06.397 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:19:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:19:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:19:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:19:06.397 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:19:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "0/23 daemons upgraded", 2024-09-20T20:19:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d image on host smithi146", 2024-09-20T20:19:06.398 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:19:06.398 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:19:06.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:06 smithi078 ceph-mon[30028]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:06.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:06 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1428123861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:19:06.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:06 smithi078 ceph-mon[23685]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:06.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:06 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1428123861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:19:06.719 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:19:06.719 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:19:06.719 INFO:teuthology.orchestra.run.smithi078.stdout: Cephadm agent on host smithi146 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T20:19:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:06 smithi146 ceph-mon[25603]: from='client.24986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:06 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1428123861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:19:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[23685]: from='client.24992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[23685]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[23685]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[23685]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/836961287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:19:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[30028]: from='client.24992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[30028]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[30028]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:07.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[30028]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:07 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/836961287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:19:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:07 smithi146 ceph-mon[25603]: from='client.24992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:07 smithi146 ceph-mon[25603]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:07 smithi146 ceph-mon[25603]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:07 smithi146 ceph-mon[25603]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:07 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/836961287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:19:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:08 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:08 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:08 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:09.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[23685]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:09.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[30028]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:09.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:09 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:09 smithi146 ceph-mon[25603]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:09 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:19:10] "GET /metrics HTTP/1.1" 200 38435 "" "Prometheus/2.43.0" 2024-09-20T20:19:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:11 smithi078 ceph-mon[23685]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:11 smithi078 ceph-mon[30028]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:11 smithi146 ceph-mon[25603]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:12 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:12 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:12 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:13 smithi146 ceph-mon[25603]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:13 smithi078 ceph-mon[23685]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:13 smithi078 ceph-mon[30028]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:15 smithi146 ceph-mon[25603]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:15 smithi078 ceph-mon[23685]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:15 smithi078 ceph-mon[30028]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:17.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:17 smithi146 ceph-mon[25603]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:17 smithi078 ceph-mon[30028]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:17 smithi078 ceph-mon[23685]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:18.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:18 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:18 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:18 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:19 smithi146 ceph-mon[25603]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:19 smithi078 ceph-mon[23685]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:19 smithi078 ceph-mon[30028]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:19:20] "GET /metrics HTTP/1.1" 200 38433 "" "Prometheus/2.43.0" 2024-09-20T20:19:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:21 smithi146 ceph-mon[25603]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:21 smithi078 ceph-mon[23685]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:21 smithi078 ceph-mon[30028]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:23 smithi146 ceph-mon[25603]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:23 smithi078 ceph-mon[23685]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:23 smithi078 ceph-mon[30028]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:25 smithi146 ceph-mon[25603]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:25 smithi078 ceph-mon[23685]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:25 smithi078 ceph-mon[30028]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:27 smithi146 ceph-mon[25603]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:27 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:27 smithi078 ceph-mon[23685]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:27 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:27 smithi078 ceph-mon[30028]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:27 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:28.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:28 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:28.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:28 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:28 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:29 smithi146 ceph-mon[25603]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:29 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[23685]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[30028]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:29 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:30.906 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:19:30] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.43.0" 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.x 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: Deploying daemon mgr.x on smithi146 2024-09-20T20:19:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[30028]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: Upgrade: Updating mgr.x 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: Deploying daemon mgr.x on smithi146 2024-09-20T20:19:31.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:31 smithi078 ceph-mon[23685]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.x 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: Deploying daemon mgr.x on smithi146 2024-09-20T20:19:31.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:31 smithi146 ceph-mon[25603]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:31.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:31 smithi146 systemd[1]: Stopping Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:19:32.157 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:31 smithi146 podman[66175]: 2024-09-20 20:19:31.891949548 +0000 UTC m=+0.125730885 container died b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T20:19:32.408 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 podman[66175]: 2024-09-20 20:19:32.229961325 +0000 UTC m=+0.463742664 container cleanup b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:19:32.408 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 bash[66175]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x 2024-09-20T20:19:32.673 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 podman[66205]: 2024-09-20 20:19:32.406415948 +0000 UTC m=+0.506736063 container remove b46623d9a752107926a94a19e19e10394c797eadee59602e7fac7d0ef55e809b (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T20:19:32.673 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:19:32.673 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T20:19:32.673 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 systemd[1]: Stopped Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:19:32.673 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Consumed 11.173s CPU time. 2024-09-20T20:19:33.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:32 smithi146 systemd[1]: Starting Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:19:33.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:33 smithi146 ceph-mon[25603]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:33.387 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 podman[66332]: 2024-09-20 20:19:32.977269897 +0000 UTC m=+0.016349843 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:19:33.387 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 podman[66332]: 2024-09-20 20:19:33.283105587 +0000 UTC m=+0.322185519 container create 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2) 2024-09-20T20:19:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:33 smithi078 ceph-mon[23685]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:33 smithi078 ceph-mon[30028]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:33.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:19:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:19:33.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:19:33.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 podman[66332]: 2024-09-20 20:19:33.515691469 +0000 UTC m=+0.554771411 container init 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T20:19:33.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 podman[66332]: 2024-09-20 20:19:33.518726796 +0000 UTC m=+0.557806729 container start 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:19:33.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 ceph-mgr[66351]: -- 172.21.15.146:0/3806643510 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d25eb0d0e0 con 0x55d25eaeb000 2024-09-20T20:19:33.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 bash[66332]: 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a 2024-09-20T20:19:33.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:33 smithi146 systemd[1]: Started Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:19:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:34 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:34 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:34 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:34 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:34 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:34 smithi146 ceph-mon[25603]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[30028]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:34 smithi078 ceph-mon[23685]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:37.018 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:19:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:37 smithi078 ceph-mon[30028]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:37 smithi078 ceph-mon[23685]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:37.539 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:37 smithi146 ceph-mon[25603]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 10m - - 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 error 113s ago 10m - - 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4m) 8s ago 5m 24.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5m) 113s ago 5m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3m) 8s ago 3m 56.8M - 3.8 4404209dfb71 e2de8476e360 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:8443,9283,8765 starting - - - - 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (11m) 8s ago 11m 503M - 18.2.4-869-g0b2cc4d4 4404209dfb71 a768f0dec912 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (11m) 8s ago 11m 51.1M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (9m) 113s ago 9m 43.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:19:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (9m) 8s ago 9m 42.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4m) 8s ago 4m 14.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4m) 113s ago 4m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (9m) 8s ago 9m 51.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (8m) 8s ago 8m 60.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8m) 8s ago 8m 56.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7m) 8s ago 7m 57.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7m) 113s ago 7m 62.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7m) 113s ago 7m 55.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6m) 113s ago 6m 47.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6m) 113s ago 6m 60.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4m) 113s ago 5m 42.9M - 2.43.0 a07b618ecd1d f312fa9916d6 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3m) 8s ago 3m 77.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3m) 113s ago 3m 75.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:19:37.553 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3m) 8s ago 3m 76.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:19:37.554 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3m) 113s ago 3m 78.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:19:37.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:19:37.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:19:37.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:19:37.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 17 2024-09-20T20:19:37.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:19:37.867 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:19:38.133 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "", 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:19:38.134 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:19:38.451 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:19:38.451 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:19:38.451 INFO:teuthology.orchestra.run.smithi078.stdout: Cephadm agent on host smithi146 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T20:19:38.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:38.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[30028]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:38.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[30028]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:38.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1390921773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:19:38.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:38.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[23685]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:38.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[23685]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:38.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:38 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1390921773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:19:38.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:38.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-mon[25603]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:38.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-mon[25603]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:38.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1390921773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:19:38.866 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:38.681+0000 7f06340a1040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:19:38.866 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:38.681+0000 7f06340a1040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ^ 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: SyntaxError: EOL while scanning string literal 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:38.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:38.682+0000 7f06340a1040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:19:38.868 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:38.682+0000 7f06340a1040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:19:38.868 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:38.863+0000 7f06340a1040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:19:39.133 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:38] ENGINE Bus STARTING 2024-09-20T20:19:39.134 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: CherryPy Checker: 2024-09-20T20:19:39.134 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: The Application mounted at '' has an empty config. 2024-09-20T20:19:39.134 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:39.134 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:38] ENGINE Serving on http://:::9283 2024-09-20T20:19:39.134 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:38] ENGINE Bus STARTED 2024-09-20T20:19:39.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:39 smithi146 ceph-mon[25603]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:39.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:39 smithi146 ceph-mon[25603]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:39.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:39 smithi146 ceph-mon[25603]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:39.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:39 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2298825277' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:19:39.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:39 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:19:39.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:39 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[23685]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[23685]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[23685]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2298825277' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[23685]: Standby manager daemon x restarted 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[23685]: Standby manager daemon x started 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[30028]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[30028]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:19:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[30028]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2298825277' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:19:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:19:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:39 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:19:40.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:40 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:19:40.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:40 smithi146 ceph-mon[25603]: mgrmap e22: y(active, since 5m), standbys: x 2024-09-20T20:19:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:40 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:19:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:40 smithi078 ceph-mon[23685]: mgrmap e22: y(active, since 5m), standbys: x 2024-09-20T20:19:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:40 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:19:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:40 smithi078 ceph-mon[30028]: mgrmap e22: y(active, since 5m), standbys: x 2024-09-20T20:19:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:19:40] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.43.0" 2024-09-20T20:19:41.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:41 smithi146 ceph-mon[25603]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:41.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:41 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[23685]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[30028]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:41 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:42 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:42 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:42 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:43 smithi146 ceph-mon[25603]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:43 smithi078 ceph-mon[23685]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:43 smithi078 ceph-mon[30028]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:43.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:19:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:19:43.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:19:43.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:19:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:19:43.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:19:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:45 smithi146 ceph-mon[25603]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:45 smithi078 ceph-mon[30028]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:45 smithi078 ceph-mon[23685]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:47 smithi078 ceph-mon[23685]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:47 smithi078 ceph-mon[30028]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:47 smithi146 ceph-mon[25603]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:19:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:48 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:48 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:48 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:49.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:49 smithi078 ceph-mon[30028]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:49.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:49 smithi078 ceph-mon[23685]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:49.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:49 smithi146 ceph-mon[25603]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:19:49.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:49] ENGINE Bus STOPPING 2024-09-20T20:19:50.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:49.874+0000 7f5334721640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-20T20:19:50.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setuser ceph since I am not root 2024-09-20T20:19:50.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ignoring --setgroup ceph since I am not root 2024-09-20T20:19:50.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.019+0000 7f3995fcd040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:19:50.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.078+0000 7f3995fcd040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:19:50.266 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:19:50.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:49] ENGINE Bus STOPPED 2024-09-20T20:19:50.442 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.227+0000 7f3995fcd040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.293+0000 7f3995fcd040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:19:50.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: Activating manager daemon x 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: mgrmap e23: x(active, starting, since 0.00897437s) 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:19:50.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[30028]: Manager daemon x is now available 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: Activating manager daemon x 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: mgrmap e23: x(active, starting, since 0.00897437s) 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:19:50.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:19:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-mon[23685]: Manager daemon x is now available 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: Activating manager daemon x 2024-09-20T20:19:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24335 172.21.15.78:0/471735703' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: mgrmap e23: x(active, starting, since 0.00897437s) 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:19:50.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:19:50.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:19:50.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:19:50.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-mon[25603]: Manager daemon x is now available 2024-09-20T20:19:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.439+0000 7f3995fcd040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:19:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.581+0000 7f3995fcd040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:19:51.003 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:50] ENGINE Bus STARTING 2024-09-20T20:19:51.004 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:51] ENGINE Serving on http://:::9283 2024-09-20T20:19:51.101 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.927+0000 7f3995fcd040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:19:51.101 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:50.990+0000 7f3995fcd040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:19:51.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:19:51] ENGINE Bus STARTED 2024-09-20T20:19:51.460 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:51.177+0000 7f3995fcd040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: Checking for cert/key for grafana.a 2024-09-20T20:19:51.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: Migrating grafana.a cert to cert store 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: Migrating grafana.a key to cert store 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:19:51.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: mgrmap e24: x(active, since 1.01328s) 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-mon[25603]: [20/Sep/2024:20:19:51] ENGINE Bus STARTING 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:51.569+0000 7f0598578640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:20:19:51] ENGINE Error in HTTPServer.serve 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: self._connections.run(self.expiration_interval) 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:19:51.835 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: self._run(expiration_interval) 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: s = self.context.wrap_socket( 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.sslsocket_class._create( 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: self.do_handshake() 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: self._sslobj.do_handshake() 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:19:51.836 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:19:51 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: Checking for cert/key for grafana.a 2024-09-20T20:19:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: Migrating grafana.a cert to cert store 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: Migrating grafana.a key to cert store 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:19:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: mgrmap e24: x(active, since 1.01328s) 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[23685]: [20/Sep/2024:20:19:51] ENGINE Bus STARTING 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T20:19:52.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: Checking for cert/key for grafana.a 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: Migrating grafana.a cert to cert store 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: Migrating grafana.a key to cert store 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:19:52.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:19:52.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T20:19:52.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T20:19:52.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: mgrmap e24: x(active, since 1.01328s) 2024-09-20T20:19:52.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:19:52.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:52.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:51 smithi078 ceph-mon[30028]: [20/Sep/2024:20:19:51] ENGINE Bus STARTING 2024-09-20T20:19:52.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:52.079+0000 7f3995fcd040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:19:52.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:52.148+0000 7f3995fcd040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: Reconfiguring daemon agent.smithi146 on smithi146 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: [20/Sep/2024:20:19:51] ENGINE Serving on https://172.21.15.146:7150 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: [20/Sep/2024:20:19:51] ENGINE Error in HTTPServer.serve 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._run(expiration_interval) 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:19:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: s = self.context.wrap_socket( 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.sslsocket_class._create( 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.do_handshake() 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._sslobj.do_handshake() 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: [20/Sep/2024:20:19:51] ENGINE Serving on http://172.21.15.146:8765 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: [20/Sep/2024:20:19:51] ENGINE Bus STARTED 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: Reconfiguring daemon agent.smithi078 on smithi078 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: Deploying cephadm binary to smithi146 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: Health check failed: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) (MGR_MODULE_DEPENDENCY) 2024-09-20T20:19:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:19:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:52 smithi146 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:52.872+0000 7f3995fcd040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:52.988+0000 7f3995fcd040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: Reconfiguring daemon agent.smithi146 on smithi146 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: [20/Sep/2024:20:19:51] ENGINE Serving on https://172.21.15.146:7150 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: [20/Sep/2024:20:19:51] ENGINE Error in HTTPServer.serve 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:19:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:19:53.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: [20/Sep/2024:20:19:51] ENGINE Serving on http://172.21.15.146:8765 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: [20/Sep/2024:20:19:51] ENGINE Bus STARTED 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: Reconfiguring daemon agent.smithi078 on smithi078 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: Deploying cephadm binary to smithi146 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: Health check failed: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) (MGR_MODULE_DEPENDENCY) 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[23685]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: Reconfiguring daemon agent.smithi146 on smithi146 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: [20/Sep/2024:20:19:51] ENGINE Serving on https://172.21.15.146:7150 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: [20/Sep/2024:20:19:51] ENGINE Error in HTTPServer.serve 2024-09-20T20:19:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:19:53.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.do_handshake() 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: [20/Sep/2024:20:19:51] ENGINE Serving on http://172.21.15.146:8765 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: [20/Sep/2024:20:19:51] ENGINE Bus STARTED 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: Reconfiguring daemon agent.smithi078 on smithi078 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: Deploying cephadm binary to smithi146 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: Health check failed: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) (MGR_MODULE_DEPENDENCY) 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:19:53.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:52 smithi078 ceph-mon[30028]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:53.410 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.131+0000 7f3995fcd040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:19:53.411 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.193+0000 7f3995fcd040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:19:53.411 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.340+0000 7f3995fcd040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:19:53.411 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.407+0000 7f3995fcd040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:19:53.701 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.467+0000 7f3995fcd040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:19:53.702 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.630+0000 7f3995fcd040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:19:53.702 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.698+0000 7f3995fcd040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:19:53.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:19:53.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:19:53.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:19:53.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:19:53.962 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.772+0000 7f3995fcd040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:19:53.962 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.866+0000 7f3995fcd040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:19:53.962 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:53.931+0000 7f3995fcd040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: Deploying cephadm binary to smithi078 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: mgrmap e25: x(active, since 3s) 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: Deploying cephadm binary to smithi078 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: mgrmap e25: x(active, since 3s) 2024-09-20T20:19:53.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:53.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:53.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:53.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:19:53.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: Deploying cephadm binary to smithi078 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: mgrmap e25: x(active, since 3s) 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:19:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:19:54.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:54.000+0000 7f3995fcd040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:19:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:54 smithi146 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:54 smithi146 ceph-mon[25603]: Standby manager daemon y started 2024-09-20T20:19:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:54 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T20:19:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:54 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:19:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:54 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T20:19:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:54 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:19:55.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[30028]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:55.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[30028]: Standby manager daemon y started 2024-09-20T20:19:55.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T20:19:55.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:19:55.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[23685]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[23685]: Standby manager daemon y started 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-mon[23685]: from='mgr.? 172.21.15.78:0/2675707604' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:19:55.118 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:54.854+0000 7f3995fcd040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:19:55.119 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:19:54] ENGINE Bus STARTING 2024-09-20T20:19:55.119 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: CherryPy Checker: 2024-09-20T20:19:55.119 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: The Application mounted at '' has an empty config. 2024-09-20T20:19:55.119 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: 2024-09-20T20:19:55.119 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:19:54] ENGINE Serving on http://:::9283 2024-09-20T20:19:55.119 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: [20/Sep/2024:20:19:54] ENGINE Bus STARTED 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-20T20:19:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:19:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: mgrmap e26: x(active, since 5s), standbys: y 2024-09-20T20:19:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:19:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:19:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:57 smithi146 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:57 smithi078 ceph-mon[23685]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:57 smithi078 ceph-mon[30028]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:19:58.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:58 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:58.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:58 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:58 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:19:59.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:19:59 smithi078 ceph-mon[30028]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:19:59.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:19:59 smithi078 ceph-mon[23685]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:19:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:19:59 smithi146 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:20:00.498 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:00 smithi146 ceph-mon[25603]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.498 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:00 smithi146 ceph-mon[25603]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:00 smithi146 ceph-mon[25603]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-mon[23685]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-mon[23685]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-mon[23685]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-mon[30028]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-mon[30028]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-mon[30028]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:20:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T20:20:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:01 smithi146 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:20:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:01 smithi078 ceph-mon[23685]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:20:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:01 smithi078 ceph-mon[30028]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:20:03.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:03 smithi078 ceph-mon[30028]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:20:03.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:03 smithi078 ceph-mon[23685]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:20:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:03 smithi146 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:20:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:03.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:03.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:05 smithi146 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:20:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:05 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:05 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:05 smithi078 ceph-mon[23685]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:20:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:05 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:05 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:05 smithi078 ceph-mon[30028]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:20:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:05 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:05 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:06 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:06 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:07 smithi146 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:20:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:07 smithi078 ceph-mon[23685]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:20:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:07 smithi078 ceph-mon[30028]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:20:08.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:08 smithi146 ceph-mon[25603]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:08.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:08 smithi078 ceph-mon[30028]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:08.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:08 smithi078 ceph-mon[23685]: from='client.14907 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:08.763 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:20:09.301 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:20:09.301 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 starting - - - - 2024-09-20T20:20:09.301 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 error 4s ago 10m - - 2024-09-20T20:20:09.301 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4m) 19s ago 5m 24.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5m) 4s ago 5m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3m) 19s ago 3m 56.9M - 3.8 4404209dfb71 e2de8476e360 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:8443,9283,8765 running (36s) 4s ago 9m 480M - 19.3.0-5127-ge8baca1e 82f5f45939a4 7d2b4071021a 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (11m) 19s ago 11m 505M - 18.2.4-869-g0b2cc4d4 4404209dfb71 a768f0dec912 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (11m) 19s ago 11m 53.1M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (10m) 4s ago 10m 48.7M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 19s ago 10m 44.0M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5m) 19s ago 5m 13.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5m) 4s ago 5m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (9m) 19s ago 9m 51.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (9m) 19s ago 9m 60.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8m) 19s ago 8m 56.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:20:09.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (8m) 19s ago 8m 57.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (8m) 4s ago 8m 64.4M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7m) 4s ago 7m 57.2M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7m) 4s ago 7m 48.4M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6m) 4s ago 6m 61.6M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4m) 4s ago 6m 50.5M - 2.43.0 a07b618ecd1d f312fa9916d6 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4m) 19s ago 4m 77.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4m) 4s ago 4m 77.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4m) 19s ago 4m 77.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:20:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3m) 4s ago 3m 80.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:20:09.610 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:20:09.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:20:09.610 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:20:09.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:09.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:20:09.610 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 16, 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:20:09.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:20:09.612 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: Adjusting osd_memory_target on smithi146 to 2736M 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: Adjusting osd_memory_target on smithi146 to 2736M 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:20:09.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:20:09.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: Adjusting osd_memory_target on smithi146 to 2736M 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:20:09.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:20:09.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "", 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:20:09.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:20:10.212 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 1 Cephadm Agent(s) are not reporting. Hosts may be offline; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:10.212 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_AGENT_DOWN: 1 Cephadm Agent(s) are not reporting. Hosts may be offline 2024-09-20T20:20:10.212 INFO:teuthology.orchestra.run.smithi078.stdout: Cephadm agent on host smithi146 has not reported in 60.0 seconds. Agent is assumed down and host may be offline. 2024-09-20T20:20:10.212 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:10.212 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:10.745 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:20:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T20:20:10.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2482314571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:20:10.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2562570634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2482314571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:20:10.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:10 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2562570634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2482314571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-09-20T20:20:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:10 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2562570634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: Deploying daemon agent.smithi078 on smithi078 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: Deploying daemon agent.smithi078 on smithi078 2024-09-20T20:20:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: Deploying daemon agent.smithi078 on smithi078 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:11.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:11.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:12.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:12 smithi078 ceph-mon[23685]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:12.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:12 smithi078 ceph-mon[23685]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:12.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:12 smithi078 ceph-mon[23685]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:12.634 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:12 smithi078 ceph-mon[30028]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:12.634 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:12 smithi078 ceph-mon[30028]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:12.635 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:12 smithi078 ceph-mon[30028]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:12 smithi146 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:12 smithi146 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:12 smithi146 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:13.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:13.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:14 smithi078 ceph-mon[23685]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:14 smithi078 ceph-mon[30028]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:14 smithi146 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:16 smithi146 ceph-mon[25603]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:16 smithi078 ceph-mon[23685]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:16 smithi078 ceph-mon[30028]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 917 B/s rd, 0 op/s 2024-09-20T20:20:17.893 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:20:17.689+0000 7f05c35ce640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Creating ceph-iscsi config... 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return _run_code(code, main_globals, None, 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: exec(code, run_globals) 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:17.894 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:20:17.895 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.wait_async( 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return future.result(timeout) 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.__get_result() 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: raise self._exception 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: out, err, code = await self._run_cephadm( 2024-09-20T20:20:17.896 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: raise OrchestratorError( 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Creating ceph-iscsi config... 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return _run_code(code, main_globals, None, 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: exec(code, run_globals) 2024-09-20T20:20:17.897 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:17.898 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:18.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:18 smithi146 ceph-mon[25603]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:18.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:20:18.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:20:18.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[23685]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:20:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[30028]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:19.635 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:19.635 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:19.635 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:19.635 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:19.635 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:19.636 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:19.637 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.wait_async( 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: return future.result(timeout) 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.__get_result() 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise self._exception 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:19.638 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise OrchestratorError( 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:19.639 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: Deploying daemon agent.smithi146 on smithi146 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 1 op/s 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.640 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:19 smithi146 ceph-mon[25603]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi078.bwbwwj on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.wait_async( 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:19.788 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.a.smithi078.stdout: return future.result(timeout) 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.789 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.790 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.wait_async( 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: return future.result(timeout) 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.__get_result() 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise self._exception 2024-09-20T20:20:19.791 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:19.792 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.793 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:19.794 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: Deploying daemon agent.smithi146 on smithi146 2024-09-20T20:20:19.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 1 op/s 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.__get_result() 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise self._exception 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:19.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:19.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: Deploying daemon agent.smithi146 on smithi146 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 1 op/s 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[30028]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi078.bwbwwj on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:19 smithi078 ceph-mon[23685]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi078.bwbwwj on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:20.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:20:20.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:20:20.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-mon[25603]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 1 op/s 2024-09-20T20:20:20.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-mon[25603]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[23914]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:20:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T20:20:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[23685]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:20:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[23685]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:20:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[23685]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 1 op/s 2024-09-20T20:20:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[23685]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[30028]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:20:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[30028]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:20:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[30028]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 1 op/s 2024-09-20T20:20:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:20 smithi078 ceph-mon[30028]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:20.647Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.78:8765: connect: connection refused" 2024-09-20T20:20:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:20.648Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.78:8765: connect: connection refused" 2024-09-20T20:20:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:20.651Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.78:8765: connect: connection refused" 2024-09-20T20:20:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:20.651Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.78:8765: connect: connection refused" 2024-09-20T20:20:21.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:20.651Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.78:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.78:8765: connect: connection refused" 2024-09-20T20:20:21.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:21.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:21.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:21 smithi146 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:21 smithi078 ceph-mon[30028]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:21 smithi078 ceph-mon[23685]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:22 smithi146 ceph-mon[25603]: Detected new or changed devices on smithi146 2024-09-20T20:20:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:22 smithi078 ceph-mon[23685]: Detected new or changed devices on smithi146 2024-09-20T20:20:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:22 smithi078 ceph-mon[30028]: Detected new or changed devices on smithi146 2024-09-20T20:20:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:23.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:23.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:24.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:23 smithi146 ceph-mon[25603]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:23 smithi078 ceph-mon[23685]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:23 smithi078 ceph-mon[30028]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-09-20T20:20:25.695 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 systemd[1]: Stopping Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:20:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-mon[25603]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.795Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T20:20:26.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T20:20:26.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[52981]: ts=2024-09-20T20:20:25.796Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T20:20:26.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:25 smithi146 podman[76129]: 2024-09-20 20:20:25.806263422 +0000 UTC m=+0.131434228 container died f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:20:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:25 smithi078 ceph-mon[23685]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:25 smithi078 ceph-mon[30028]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:26.425 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:26 smithi146 podman[76129]: 2024-09-20 20:20:26.164097963 +0000 UTC m=+0.489268769 container cleanup f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:20:26.425 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:26 smithi146 bash[76129]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T20:20:26.766 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:26 smithi146 podman[76155]: 2024-09-20 20:20:26.423760422 +0000 UTC m=+0.617906574 container remove f312fa9916d6233ae699d23d7dd36326ad284b0831db7ec6b961d74ae77c6fbd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:20:27.391 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:27 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T20:20:27.391 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:27 smithi146 systemd[1]: Stopped Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:20:27.391 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:27 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 2.258s CPU time. 2024-09-20T20:20:27.391 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:27 smithi146 systemd[1]: Starting Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:20:28.260 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 podman[76410]: 2024-09-20 20:20:27.914717038 +0000 UTC m=+0.021501295 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T20:20:28.260 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 podman[76410]: 2024-09-20 20:20:28.039617026 +0000 UTC m=+0.146401269 container create bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:20:28.260 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 podman[76410]: 2024-09-20 20:20:28.258953759 +0000 UTC m=+0.365737995 container init bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:20:28.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:28 smithi078 ceph-mon[23685]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:28.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:28 smithi078 ceph-mon[30028]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-mon[25603]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 podman[76410]: 2024-09-20 20:20:28.262155019 +0000 UTC m=+0.368939261 container start bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.285Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.285Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.286Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi146 (none))" 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.286Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.286Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T20:20:28.517 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.287Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.288Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.289Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.289Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.291Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.291Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.523µs 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.291Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.292Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.300Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.301Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-20T20:20:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.301Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.267µs wal_replay_duration=9.727415ms wbl_replay_duration=131ns total_replay_duration=9.76474ms 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.302Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.302Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.302Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.323Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.699636ms db_storage=798ns remote_storage=1.264µs web_handler=388ns query_engine=507ns scrape=384.808µs scrape_sd=105.731µs notify=14.139µs notify_sd=16.424µs rules=19.840439ms tracing=5.974µs 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.323Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:20:28.323Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 bash[76410]: bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 2024-09-20T20:20:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:20:28 smithi146 systemd[1]: Started Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:29 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1776448550' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:20:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1776448550' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:20:29.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T20:20:29.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:29.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:29.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:29.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:29.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:29 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1776448550' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: Health check update: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi078.bwbwwj on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1395752026' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1786721494"}]: dispatch 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:20:30.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: Health check update: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi078.bwbwwj on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1395752026' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1786721494"}]: dispatch 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:20:30.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:30 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: Health check update: 2 failed cephadm daemon(s) ['daemon iscsi.foo.smithi078.bwbwwj on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1395752026' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1786721494"}]: dispatch 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:20:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:30 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.y 2024-09-20T20:20:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[30028]: Deploying daemon mgr.y on smithi078 2024-09-20T20:20:31.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1395752026' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1786721494"}]': finished 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[30028]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2221578098' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]: dispatch 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]: dispatch 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[23685]: Upgrade: Updating mgr.y 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[23685]: Deploying daemon mgr.y on smithi078 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1395752026' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1786721494"}]': finished 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[23685]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2221578098' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]: dispatch 2024-09-20T20:20:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:31 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]: dispatch 2024-09-20T20:20:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:31 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.y 2024-09-20T20:20:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:31 smithi146 ceph-mon[25603]: Deploying daemon mgr.y on smithi078 2024-09-20T20:20:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:31 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1395752026' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1786721494"}]': finished 2024-09-20T20:20:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:31 smithi146 ceph-mon[25603]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T20:20:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:31 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2221578098' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]: dispatch 2024-09-20T20:20:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:31 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]: dispatch 2024-09-20T20:20:31.978 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:31 smithi078 systemd[1]: Stopping Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[30028]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]': finished 2024-09-20T20:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[30028]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T20:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2563814977' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3750775165"}]: dispatch 2024-09-20T20:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[23685]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]': finished 2024-09-20T20:20:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[23685]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T20:20:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:32 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2563814977' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3750775165"}]: dispatch 2024-09-20T20:20:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:32 smithi146 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:32 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/2526592432"}]': finished 2024-09-20T20:20:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:32 smithi146 ceph-mon[25603]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T20:20:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:32 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2563814977' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3750775165"}]: dispatch 2024-09-20T20:20:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[23685]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-20T20:20:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2563814977' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3750775165"}]': finished 2024-09-20T20:20:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[23685]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T20:20:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/468021568' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1237956431"}]: dispatch 2024-09-20T20:20:33.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:33 smithi078 podman[71129]: 2024-09-20 20:20:33.728619698 +0000 UTC m=+0.479075682 container died a768f0dec912601b0a4753818b3f433637c3f8ab030dc98fd8cbc9daad32da42 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-20T20:20:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[30028]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-20T20:20:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2563814977' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3750775165"}]': finished 2024-09-20T20:20:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[30028]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T20:20:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/468021568' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1237956431"}]: dispatch 2024-09-20T20:20:33.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:33.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:33.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:33.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:33 smithi146 ceph-mon[25603]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-20T20:20:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:33 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2563814977' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/3750775165"}]': finished 2024-09-20T20:20:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:33 smithi146 ceph-mon[25603]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T20:20:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:33 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/468021568' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1237956431"}]: dispatch 2024-09-20T20:20:34.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 podman[71129]: 2024-09-20 20:20:34.172544288 +0000 UTC m=+0.923000250 container cleanup a768f0dec912601b0a4753818b3f433637c3f8ab030dc98fd8cbc9daad32da42 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T20:20:34.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 bash[71129]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y 2024-09-20T20:20:34.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/468021568' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1237956431"}]': finished 2024-09-20T20:20:34.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[23685]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T20:20:34.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/998860638' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]: dispatch 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]: dispatch 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 podman[71200]: 2024-09-20 20:20:34.474279293 +0000 UTC m=+0.744579973 container remove a768f0dec912601b0a4753818b3f433637c3f8ab030dc98fd8cbc9daad32da42 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 systemd[1]: Stopped Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Consumed 58.127s CPU time. 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/468021568' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1237956431"}]': finished 2024-09-20T20:20:34.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[30028]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T20:20:34.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/998860638' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]: dispatch 2024-09-20T20:20:34.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:34 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]: dispatch 2024-09-20T20:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:34 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/468021568' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/1237956431"}]': finished 2024-09-20T20:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:34 smithi146 ceph-mon[25603]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T20:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:34 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/998860638' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]: dispatch 2024-09-20T20:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:34 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]: dispatch 2024-09-20T20:20:35.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:34 smithi078 systemd[1]: Starting Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:20:35.371 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 podman[71463]: 2024-09-20 20:20:35.001458284 +0000 UTC m=+0.020287857 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:20:35.372 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 podman[71463]: 2024-09-20 20:20:35.152350463 +0000 UTC m=+0.171180027 container create 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T20:20:35.637 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 podman[71463]: 2024-09-20 20:20:35.370188332 +0000 UTC m=+0.389017896 container init 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T20:20:35.637 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 podman[71463]: 2024-09-20 20:20:35.373163342 +0000 UTC m=+0.391992911 container start 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mgr[71517]: -- 172.21.15.78:0/1076723618 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c862dc10e0 con 0x55c862d9f400 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 bash[71463]: 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:35 smithi078 systemd[1]: Started Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]': finished 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1101504494' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/1861129541"}]: dispatch 2024-09-20T20:20:35.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]': finished 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1101504494' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/1861129541"}]: dispatch 2024-09-20T20:20:35.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:35.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:35.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:35.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:35.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:35.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:35 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6800/1861129541"}]': finished 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1101504494' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/1861129541"}]: dispatch 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:35 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:36.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:20:36] "GET /metrics HTTP/1.1" 200 38656 "" "Prometheus/2.43.0" 2024-09-20T20:20:36.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1101504494' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/1861129541"}]': finished 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:36.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1418930833' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/626323715"}]: dispatch 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1101504494' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/1861129541"}]': finished 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:36.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:36.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:36.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:36.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1418930833' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/626323715"}]: dispatch 2024-09-20T20:20:36.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:36 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1101504494' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:6801/1861129541"}]': finished 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1418930833' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/626323715"}]: dispatch 2024-09-20T20:20:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:36 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_AGENT_DOWN (was: 1 Cephadm Agent(s) are not reporting. Hosts may be offline) 2024-09-20T20:20:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:37 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1418930833' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/626323715"}]': finished 2024-09-20T20:20:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:37 smithi146 ceph-mon[25603]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T20:20:38.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:37 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1418930833' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/626323715"}]': finished 2024-09-20T20:20:38.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:37 smithi078 ceph-mon[30028]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T20:20:38.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:37 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1418930833' entity='client.iscsi.foo.smithi078.bwbwwj' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.78:0/626323715"}]': finished 2024-09-20T20:20:38.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:37 smithi078 ceph-mon[23685]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T20:20:38.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:38 smithi078 ceph-mon[30028]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:20:38.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:38 smithi078 ceph-mon[23685]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:20:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:38 smithi146 ceph-mon[25603]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T20:20:40.510 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='client.25133 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:40 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='client.25133 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:40.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:40.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='client.25133 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:40.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:40.510+0000 7f7f945ec040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:40.510+0000 7f7f945ec040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:20:40.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: ^ 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: SyntaxError: EOL while scanning string literal 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:40.511+0000 7f7f945ec040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:40.511+0000 7f7f945ec040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:20:40.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:40.698+0000 7f7f945ec040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: [20/Sep/2024:20:20:40] ENGINE Bus STARTING 2024-09-20T20:20:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: CherryPy Checker: 2024-09-20T20:20:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: The Application mounted at '' has an empty config. 2024-09-20T20:20:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: 2024-09-20T20:20:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: [20/Sep/2024:20:20:40] ENGINE Serving on http://:::9283 2024-09-20T20:20:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[71501]: [20/Sep/2024:20:20:40] ENGINE Bus STARTED 2024-09-20T20:20:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:20:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 11m - - 2024-09-20T20:20:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 11m - - 2024-09-20T20:20:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5m) 1s ago 6m 24.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:20:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6m) 12s ago 6m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:20:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (12s) 1s ago 4m 80.2M - 3.8 4404209dfb71 69df476c5223 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:8443,9283,8765 running (68s) 12s ago 10m 488M - 19.3.0-5127-ge8baca1e 82f5f45939a4 7d2b4071021a 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 unknown 1s ago 12m 436M - 18.2.4-869-g0b2cc4d4 4404209dfb71 a768f0dec912 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (12m) 1s ago 12m 57.5M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (10m) 12s ago 10m 48.7M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 1s ago 10m 49.5M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6m) 1s ago 6m 13.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5m) 12s ago 5m 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (10m) 1s ago 10m 52.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (9m) 1s ago 9m 60.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (9m) 1s ago 9m 57.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (9m) 1s ago 9m 58.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (8m) 12s ago 8m 64.5M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (8m) 12s ago 8m 57.3M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:20:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7m) 12s ago 7m 48.5M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:20:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7m) 12s ago 7m 61.7M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:20:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 unknown 12s ago 6m 50.5M - a07b618ecd1d f312fa9916d6 2024-09-20T20:20:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4m) 1s ago 4m 77.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:20:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4m) 12s ago 4m 77.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:20:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4m) 1s ago 4m 77.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:20:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4m) 12s ago 4m 80.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:20:41.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:20:41.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:20:41.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:20:41.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:41.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:20:41.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:20:41.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:20:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[30028]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[30028]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[30028]: Health check update: 2 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[30028]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[30028]: Standby manager daemon y restarted 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[30028]: Standby manager daemon y started 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[23685]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[23685]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[23685]: Health check update: 2 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[23685]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[23685]: Standby manager daemon y restarted 2024-09-20T20:20:41.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:41 smithi078 ceph-mon[23685]: Standby manager daemon y started 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:20:41.679 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:20:41.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:41 smithi146 ceph-mon[25603]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:41.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:41 smithi146 ceph-mon[25603]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:41.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:41 smithi146 ceph-mon[25603]: Health check update: 2 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:41.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:41 smithi146 ceph-mon[25603]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:41.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:41 smithi146 ceph-mon[25603]: Standby manager daemon y restarted 2024-09-20T20:20:41.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:41 smithi146 ceph-mon[25603]: Standby manager daemon y started 2024-09-20T20:20:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN 2 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state']; Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state', 'daemon prometheus.a on smithi146 is in unknown state'] 2024-09-20T20:20:42.000 INFO:teuthology.orchestra.run.smithi078.stdout: daemon mgr.y on smithi078 is in unknown state 2024-09-20T20:20:42.000 INFO:teuthology.orchestra.run.smithi078.stdout: daemon prometheus.a on smithi146 is in unknown state 2024-09-20T20:20:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:42.000 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:20:42.369 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:20:42.271+0000 7f05c35ce640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Creating ceph-iscsi config... 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return _run_code(code, main_globals, None, 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: exec(code, run_globals) 2024-09-20T20:20:42.370 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:42.371 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.wait_async( 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return future.result(timeout) 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return self.__get_result() 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: raise self._exception 2024-09-20T20:20:42.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: out, err, code = await self._run_cephadm( 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: raise OrchestratorError( 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Creating ceph-iscsi config... 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: Traceback (most recent call last): 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: return _run_code(code, main_globals, None, 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:42.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: exec(code, run_globals) 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:42.374 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:42.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:42.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:42.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:42.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: mgrmap e27: x(active, since 51s), standbys: y 2024-09-20T20:20:42.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3176492116' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2518974353' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: mgrmap e27: x(active, since 51s), standbys: y 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3176492116' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2518974353' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:20:42.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:42 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: mgrmap e27: x(active, since 51s), standbys: y 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3176492116' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2518974353' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:20:42.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:42 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:43.683 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:43.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:43.684 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:43.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[30028]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[30028]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[30028]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:43.684 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:43.685 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.wait_async( 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: return future.result(timeout) 2024-09-20T20:20:43.686 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.__get_result() 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise self._exception 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.687 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:43.688 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[23685]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[23685]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[23685]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:43.689 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:43.690 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.wait_async( 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: return future.result(timeout) 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.__get_result() 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise self._exception 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:20:43.691 INFO:journalctl@ceph.mon.a.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:43.692 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:43.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:43 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:43 smithi146 ceph-mon[25603]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:43 smithi146 ceph-mon[25603]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:43 smithi146 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.wait_async( 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: return future.result(timeout) 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.__get_result() 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise self._exception 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:20:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise OrchestratorError( 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:20:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:43 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:43 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:20:44.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:44 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:44.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:44 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:44.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:44 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:20:44.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:44 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:44.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:20:44.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:44 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:45.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:45 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.y 2024-09-20T20:20:45.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:45 smithi146 ceph-mon[25603]: Deploying daemon mgr.y on smithi078 2024-09-20T20:20:45.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:45 smithi146 ceph-mon[25603]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 1 op/s 2024-09-20T20:20:45.567 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:45 smithi078 systemd[1]: Stopping Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:20:45.568 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:45 smithi078 ceph-mon[23685]: Upgrade: Updating mgr.y 2024-09-20T20:20:45.568 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:45 smithi078 ceph-mon[23685]: Deploying daemon mgr.y on smithi078 2024-09-20T20:20:45.568 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:45 smithi078 ceph-mon[23685]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 1 op/s 2024-09-20T20:20:45.568 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:45 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.y 2024-09-20T20:20:45.568 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:45 smithi078 ceph-mon[30028]: Deploying daemon mgr.y on smithi078 2024-09-20T20:20:45.568 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:45 smithi078 ceph-mon[30028]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 1 op/s 2024-09-20T20:20:45.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:45 smithi078 podman[72556]: 2024-09-20 20:20:45.566041634 +0000 UTC m=+0.129268564 container died 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:20:46.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:45 smithi078 podman[72556]: 2024-09-20 20:20:45.968305266 +0000 UTC m=+0.531532194 container cleanup 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-20T20:20:46.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:45 smithi078 bash[72556]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y 2024-09-20T20:20:46.266 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:46 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:20:46] "GET /metrics HTTP/1.1" 200 38656 "" "Prometheus/2.43.0" 2024-09-20T20:20:46.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 podman[72568]: 2024-09-20 20:20:46.202810364 +0000 UTC m=+0.633182931 container remove 67fceb9616217756a228906ca31e78cccfc7f262feefc7e257f713de49dc4e2e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T20:20:46.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:20:46.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-20T20:20:46.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 systemd[1]: Stopped Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:20:46.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Consumed 6.096s CPU time. 2024-09-20T20:20:46.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 systemd[1]: Starting Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:20:46.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 podman[72695]: 2024-09-20 20:20:46.652978444 +0000 UTC m=+0.015681329 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:20:46.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 podman[72695]: 2024-09-20 20:20:46.762623192 +0000 UTC m=+0.125326062 container create 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-20T20:20:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:46 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:46 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:46 smithi146 ceph-mon[25603]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 1 op/s 2024-09-20T20:20:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:46 smithi146 ceph-mon[25603]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 1 op/s 2024-09-20T20:20:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:46 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[30028]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 1 op/s 2024-09-20T20:20:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[30028]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 1 op/s 2024-09-20T20:20:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[23685]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 1 op/s 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[23685]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 1 op/s 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:46 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 podman[72695]: 2024-09-20 20:20:46.981887519 +0000 UTC m=+0.344590389 container init 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T20:20:47.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:46 smithi078 podman[72695]: 2024-09-20 20:20:46.984728443 +0000 UTC m=+0.347431314 container start 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T20:20:47.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mgr[72714]: -- 172.21.15.78:0/3696131235 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a53ef370e0 con 0x55a53ef15400 2024-09-20T20:20:47.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:47 smithi078 bash[72695]: 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd 2024-09-20T20:20:47.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:47 smithi078 systemd[1]: Started Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: Health check update: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-09-20T20:20:48.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[30028]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: Health check update: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:48.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:48.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:47 smithi078 ceph-mon[23685]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: Health check update: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:20:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:47 smithi146 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:20:49.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:48 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state']) 2024-09-20T20:20:49.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:48 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state']) 2024-09-20T20:20:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:48 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state']) 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:49 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-09-20T20:20:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:49 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1803549458' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1803549458' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:50 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1803549458' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:20:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:50 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.043+0000 7f0fa63dd040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.043+0000 7f0fa63dd040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:20:52.234 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: ^ 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: SyntaxError: EOL while scanning string literal 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.043+0000 7f0fa63dd040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.043+0000 7f0fa63dd040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.231+0000 7f0fa63dd040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:52.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:51 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:52.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:51 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:52.500 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: [20/Sep/2024:20:20:52] ENGINE Bus STARTING 2024-09-20T20:20:52.500 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: CherryPy Checker: 2024-09-20T20:20:52.500 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: The Application mounted at '' has an empty config. 2024-09-20T20:20:52.500 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: 2024-09-20T20:20:52.500 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: [20/Sep/2024:20:20:52] ENGINE Serving on http://:::9283 2024-09-20T20:20:52.500 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[72710]: [20/Sep/2024:20:20:52] ENGINE Bus STARTED 2024-09-20T20:20:53.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Standby manager daemon y restarted 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Standby manager daemon y started 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Standby manager daemon y restarted 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Standby manager daemon y started 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:52 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Standby manager daemon y restarted 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Standby manager daemon y started 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:53.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:52 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:53.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:20:53.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:20:53.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:53.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:53.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: mgrmap e28: x(active, since 63s), standbys: y 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:53 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: mgrmap e28: x(active, since 63s), standbys: y 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: mgrmap e28: x(active, since 63s), standbys: y 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:54.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:53 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:54 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:55.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:55.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:54 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 111 B/s rd, 0 op/s 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 111 B/s rd, 0 op/s 2024-09-20T20:20:56.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:55 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:20:56] "GET /metrics HTTP/1.1" 200 38656 "" "Prometheus/2.43.0" 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 111 B/s rd, 0 op/s 2024-09-20T20:20:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:56.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:56.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:56.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:55 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:56 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:57.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:57.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:57.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:57.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:56 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:57 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:58.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:58.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:58.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:57 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:58 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:20:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:20:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:20:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:58 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:20:59 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:00.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:00.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:00.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:20:59 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:01.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[23685]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:01.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:21:01.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:00 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:21:01.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:21:01.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:00 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T20:21:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:01 smithi146 ceph-mon[25603]: Health check failed: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:21:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:01 smithi078 ceph-mon[30028]: Health check failed: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:21:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:01 smithi078 ceph-mon[23685]: Health check failed: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.y 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: Deploying daemon mgr.y on smithi078 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[30028]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-09-20T20:21:03.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: Upgrade: Updating mgr.y 2024-09-20T20:21:03.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:03.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:03.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:03.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:03.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: Deploying daemon mgr.y on smithi078 2024-09-20T20:21:03.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-mon[23685]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.y 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: Deploying daemon mgr.y on smithi078 2024-09-20T20:21:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:03 smithi146 ceph-mon[25603]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-09-20T20:21:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:03.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:03.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:05 smithi146 ceph-mon[25603]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-09-20T20:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:05 smithi078 ceph-mon[23685]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-09-20T20:21:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:05 smithi078 ceph-mon[30028]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-09-20T20:21:06.266 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:21:06] "GET /metrics HTTP/1.1" 200 38654 "" "Prometheus/2.43.0" 2024-09-20T20:21:06.437 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:06 smithi078 systemd[1]: Stopping Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:07.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:06 smithi078 podman[76371]: 2024-09-20 20:21:06.745185424 +0000 UTC m=+0.153983148 container died 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-20T20:21:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:06 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:06 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:07.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 podman[76371]: 2024-09-20 20:21:07.298002303 +0000 UTC m=+0.706800027 container cleanup 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T20:21:07.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 bash[76371]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y 2024-09-20T20:21:07.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:07 smithi078 ceph-mon[30028]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state']) 2024-09-20T20:21:07.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:07 smithi078 ceph-mon[23685]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state']) 2024-09-20T20:21:07.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 podman[76399]: 2024-09-20 20:21:07.616558262 +0000 UTC m=+0.870962555 container remove 8adfa37d296bc08d31625f2131506a5c92f5d8deb97dd55016e4d51e02348cfd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T20:21:07.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:21:07.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-20T20:21:07.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 systemd[1]: Stopped Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:21:07.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.y.service: Consumed 6.103s CPU time. 2024-09-20T20:21:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:07 smithi146 ceph-mon[25603]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon mgr.y on smithi078 is in unknown state']) 2024-09-20T20:21:08.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:07 smithi078 systemd[1]: Starting Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:08.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 podman[76530]: 2024-09-20 20:21:08.063364501 +0000 UTC m=+0.016159641 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:21:08.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 podman[76530]: 2024-09-20 20:21:08.176085881 +0000 UTC m=+0.128881050 container create 38852dc0bdd7aed8e1e99725dac8b5c43af73fef4b2fbc05cb579d974702dab4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T20:21:08.693 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 podman[76530]: 2024-09-20 20:21:08.386556131 +0000 UTC m=+0.339351258 container init 38852dc0bdd7aed8e1e99725dac8b5c43af73fef4b2fbc05cb579d974702dab4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-20T20:21:08.693 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 podman[76530]: 2024-09-20 20:21:08.389467721 +0000 UTC m=+0.342262856 container start 38852dc0bdd7aed8e1e99725dac8b5c43af73fef4b2fbc05cb579d974702dab4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0) 2024-09-20T20:21:08.694 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 ceph-mgr[76550]: -- 172.21.15.78:0/655009446 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d78f5290e0 con 0x55d78f507400 2024-09-20T20:21:08.694 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 bash[76530]: 38852dc0bdd7aed8e1e99725dac8b5c43af73fef4b2fbc05cb579d974702dab4 2024-09-20T20:21:08.694 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:08 smithi078 systemd[1]: Started Ceph mgr.y for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:21:09.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-09-20T20:21:09.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:09.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:09 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-09-20T20:21:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:09 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:11.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.378 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.913 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:12.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:12.994 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 11m - - 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 11m - - 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5m) 6s ago 7m 27.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6m) 13s ago 6m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (23s) 6s ago 4m 54.2M - 3.8 4404209dfb71 ce37493ebb45 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:8443,9283,8765 running (100s) 13s ago 11m 499M - 19.3.0-5127-ge8baca1e 82f5f45939a4 7d2b4071021a 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 starting - - - - 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (12m) 6s ago 12m 55.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (11m) 13s ago 11m 46.0M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (11m) 6s ago 11m 45.5M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6m) 6s ago 6m 13.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6m) 13s ago 6m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (10m) 6s ago 10m 52.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:21:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (10m) 6s ago 10m 61.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (9m) 6s ago 9m 57.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (9m) 6s ago 9m 58.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (9m) 13s ago 9m 65.0M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (8m) 13s ago 8m 57.8M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (8m) 13s ago 8m 49.2M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7m) 13s ago 7m 62.1M 2736M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (45s) 13s ago 7m 37.7M - 2.43.0 a07b618ecd1d bc69428bcd11 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5m) 6s ago 5m 78.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5m) 13s ago 5m 78.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5m) 6s ago 5m 78.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:21:12.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5m) 13s ago 5m 81.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:21:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.336 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:21:13.336 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:21:13.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:21:13.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:21:13.337 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:21:13.615 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:13.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:13.615 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:13.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1773245691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1773245691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:13.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:13.642 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "", 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:21:13.643 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:13.722+0000 7f2f267db040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:13.722+0000 7f2f267db040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:21:13.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:21:13.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:21:13.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ^ 2024-09-20T20:21:13.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: SyntaxError: EOL while scanning string literal 2024-09-20T20:21:13.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:13.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:13.723+0000 7f2f267db040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:21:13.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:13.723+0000 7f2f267db040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:21:13.979 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:13.979 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:13.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1773245691' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:13 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:14.203 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:13.924+0000 7f2f267db040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:14.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:13] ENGINE Bus STARTING 2024-09-20T20:21:14.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: CherryPy Checker: 2024-09-20T20:21:14.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: The Application mounted at '' has an empty config. 2024-09-20T20:21:14.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:14.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:14] ENGINE Serving on http://:::9283 2024-09-20T20:21:14.204 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:14] ENGINE Bus STARTED 2024-09-20T20:21:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Standby manager daemon y restarted 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Standby manager daemon y started 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2982580578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Standby manager daemon y restarted 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Standby manager daemon y started 2024-09-20T20:21:15.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Standby manager daemon y restarted 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Standby manager daemon y started 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2982580578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:21:15.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2982580578' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: mgrmap e29: x(active, since 85s), standbys: y 2024-09-20T20:21:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: mgrmap e29: x(active, since 85s), standbys: y 2024-09-20T20:21:15.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:15.810 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: mgrmap e29: x(active, since 85s), standbys: y 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:21:16] "GET /metrics HTTP/1.1" 200 38654 "" "Prometheus/2.43.0" 2024-09-20T20:21:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:17.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:18.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:18.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:19 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:19.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:19 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:20 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:20.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:20.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:20 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:21.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:21.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:21.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:21.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:21 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:21 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:22 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:22.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:22 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:23.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:23.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:23.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:23.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:23.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:23.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:23.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:23.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:23 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:23 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:24.451 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:24] ENGINE Bus STOPPING 2024-09-20T20:21:24.516 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:24.210+0000 7f062f032640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-20T20:21:24.517 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ignoring --setuser ceph since I am not root 2024-09-20T20:21:24.517 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ignoring --setgroup ceph since I am not root 2024-09-20T20:21:24.517 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mgr[66351]: -- 172.21.15.146:0/3990866830 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565526a650e0 con 0x565526a42800 2024-09-20T20:21:24.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:21:24.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:24] ENGINE Bus STOPPED 2024-09-20T20:21:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: Activating manager daemon y 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.24838 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: mgrmap e30: y(active, starting, since 0.00582112s) 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:21:24.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[30028]: Manager daemon y is now available 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: Activating manager daemon y 2024-09-20T20:21:24.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.24838 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: mgrmap e30: y(active, starting, since 0.00582112s) 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:21:24.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:21:24.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:21:24.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:21:24.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:24 smithi078 ceph-mon[23685]: Manager daemon y is now available 2024-09-20T20:21:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 172.21.15.146:0/942873609' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: Activating manager daemon y 2024-09-20T20:21:25.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.24838 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: mgrmap e30: y(active, starting, since 0.00582112s) 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:21:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:21:25.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:21:25.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:21:25.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:21:25.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:24 smithi146 ceph-mon[25603]: Manager daemon y is now available 2024-09-20T20:21:25.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:25] ENGINE Bus STARTING 2024-09-20T20:21:25.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:25] ENGINE Serving on http://:::9283 2024-09-20T20:21:25.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:21:25] ENGINE Bus STARTED 2024-09-20T20:21:25.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:25.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: mgrmap e31: y(active, since 1.01737s) 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]': finished 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: mgrmap e31: y(active, since 1.01737s) 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:21:25.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:25.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:21:25.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:21:25.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]': finished 2024-09-20T20:21:25.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:25.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:21:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: mgrmap e31: y(active, since 1.01737s) 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]': finished 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:25 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:26.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:25.961+0000 7f2e88cae640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:20:21:25] ENGINE Error in HTTPServer.serve 2024-09-20T20:21:26.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:26.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self._connections.run(self.expiration_interval) 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self._run(expiration_interval) 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: s = self.context.wrap_socket( 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.sslsocket_class._create( 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self.do_handshake() 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:21:26.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self._sslobj.do_handshake() 2024-09-20T20:21:26.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:21:26.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:25 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:27.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: [20/Sep/2024:20:21:25] ENGINE Bus STARTING 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: [20/Sep/2024:20:21:25] ENGINE Serving on https://172.21.15.78:7150 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: [20/Sep/2024:20:21:25] ENGINE Error in HTTPServer.serve 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:21:27.401 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.do_handshake() 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:21:27.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: [20/Sep/2024:20:21:26] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: [20/Sep/2024:20:21:26] ENGINE Bus STARTED 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: [20/Sep/2024:20:21:25] ENGINE Bus STARTING 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: [20/Sep/2024:20:21:25] ENGINE Serving on https://172.21.15.78:7150 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: [20/Sep/2024:20:21:25] ENGINE Error in HTTPServer.serve 2024-09-20T20:21:27.404 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:21:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: [20/Sep/2024:20:21:26] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: [20/Sep/2024:20:21:26] ENGINE Bus STARTED 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: mgrmap e32: y(active, since 2s) 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: mgrmap e32: y(active, since 2s) 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:27 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:21:27.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:21:27.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: [20/Sep/2024:20:21:25] ENGINE Bus STARTING 2024-09-20T20:21:27.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:21:27.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: [20/Sep/2024:20:21:25] ENGINE Serving on https://172.21.15.78:7150 2024-09-20T20:21:27.604 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: [20/Sep/2024:20:21:25] ENGINE Error in HTTPServer.serve 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._run(expiration_interval) 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: s = self.context.wrap_socket( 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.sslsocket_class._create( 2024-09-20T20:21:27.605 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.do_handshake() 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._sslobj.do_handshake() 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: [20/Sep/2024:20:21:26] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: [20/Sep/2024:20:21:26] ENGINE Bus STARTED 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.606 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: mgrmap e32: y(active, since 2s) 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:27.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:27 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:28.306Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:28.306Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:28.309Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:28.309Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:28.518 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:28.310Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:28.519 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:28.310Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:28.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:28 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[23685]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[23685]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[23685]: mgrmap e33: y(active, since 4s) 2024-09-20T20:21:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[30028]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[30028]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[30028]: mgrmap e33: y(active, since 4s) 2024-09-20T20:21:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:29 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.563 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:29.361Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.146:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.146:8765: connect: connection refused" 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-mon[25603]: mgrmap e33: y(active, since 4s) 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.275+0000 7fee91ae6040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.275+0000 7fee91ae6040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:21:29.564 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ^ 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: SyntaxError: EOL while scanning string literal 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.276+0000 7fee91ae6040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:21:29.565 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.276+0000 7fee91ae6040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.456+0000 7fee91ae6040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:21:29] ENGINE Bus STARTING 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: CherryPy Checker: 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: The Application mounted at '' has an empty config. 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:21:29] ENGINE Serving on http://:::9283 2024-09-20T20:21:29.566 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:29 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: [20/Sep/2024:20:21:29] ENGINE Bus STARTED 2024-09-20T20:21:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:30 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:30 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:21:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:30 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:30 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:30 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:21:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:30.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[23685]: Standby manager daemon x started 2024-09-20T20:21:30.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:30.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:30 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[30028]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[30028]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[30028]: mgrmap e34: y(active, since 6s), standbys: x 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[23685]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[23685]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[23685]: mgrmap e34: y(active, since 6s), standbys: x 2024-09-20T20:21:31.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:31 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:21:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:31 smithi146 ceph-mon[25603]: from='client.25250 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:21:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:31 smithi146 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:31 smithi146 ceph-mon[25603]: mgrmap e34: y(active, since 6s), standbys: x 2024-09-20T20:21:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:31 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:21:32.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[23685]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:21:32.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:32 smithi078 ceph-mon[30028]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:21:32.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:32 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:32 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:32 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:32.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:32 smithi146 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:21:33.721 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:33.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:33.721 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:33.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:35.183 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:34.850+0000 7f2eb5d08640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:35.183 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Creating ceph-iscsi config... 2024-09-20T20:21:35.183 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:35.183 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:35.183 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return _run_code(code, main_globals, None, 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: exec(code, run_globals) 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:35.184 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.wait_async( 2024-09-20T20:21:35.185 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return future.result(timeout) 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.__get_result() 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: raise self._exception 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: out, err, code = await self._run_cephadm( 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: raise OrchestratorError( 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:35.186 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Creating ceph-iscsi config... 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return _run_code(code, main_globals, None, 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:35.187 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: exec(code, run_globals) 2024-09-20T20:21:35.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:35.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:35.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:35.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:35.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:35.188 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:35.189 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:35.189 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:35.189 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:35.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:35 smithi078 ceph-mon[23685]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:21:35.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:35 smithi078 ceph-mon[30028]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:21:35.544 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:35 smithi146 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:21:36.259 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[66347]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:21:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:36 smithi146 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.wait_async( 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: return future.result(timeout) 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.__get_result() 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise self._exception 2024-09-20T20:21:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise OrchestratorError( 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:36 smithi146 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:21:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:36 smithi146 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:21:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:36 smithi078 ceph-mon[23685]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:36.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.wait_async( 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: return future.result(timeout) 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:36 smithi078 ceph-mon[30028]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:36.539 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:36.540 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.wait_async( 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: return future.result(timeout) 2024-09-20T20:21:36.541 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.__get_result() 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise self._exception 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:21:36.542 INFO:journalctl@ceph.mon.a.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:36.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:36 smithi078 ceph-mon[23685]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:36 smithi078 ceph-mon[23685]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:21:36.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.__get_result() 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise self._exception 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:36.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:36.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:36.547 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:36.547 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:36.547 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:36.547 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:36.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:36 smithi078 ceph-mon[30028]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T20:21:36.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:36 smithi078 ceph-mon[30028]: Reconfiguring daemon prometheus.a on smithi146 2024-09-20T20:21:37.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:37 smithi146 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:21:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:37 smithi078 ceph-mon[23685]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:21:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:37 smithi078 ceph-mon[30028]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:21:38.472 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 systemd[1]: Stopping Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T20:21:38.723 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.470Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T20:21:38.724 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.471Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T20:21:38.724 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.471Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T20:21:38.724 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.471Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T20:21:38.724 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[76441]: ts=2024-09-20T20:21:38.471Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T20:21:38.724 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 podman[83767]: 2024-09-20 20:21:38.480691036 +0000 UTC m=+0.110198366 container died bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:21:39.005 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 podman[83767]: 2024-09-20 20:21:38.811502734 +0000 UTC m=+0.441010075 container cleanup bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:21:39.006 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:38 smithi146 bash[83767]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T20:21:39.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 podman[83779]: 2024-09-20 20:21:39.004217316 +0000 UTC m=+0.525041428 container remove bc69428bcd113afcfdd1533fbbcd32a5875f8534767ce9b8cfcd6b83883c6182 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:21:39.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T20:21:39.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 systemd[1]: Stopped Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:21:39.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.152s CPU time. 2024-09-20T20:21:39.267 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 systemd[1]: Starting Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:39 smithi078 ceph-mon[30028]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:39 smithi078 ceph-mon[23685]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:39.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:39.585 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 podman[83870]: 2024-09-20 20:21:39.354854501 +0000 UTC m=+0.017652129 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T20:21:39.585 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 podman[83870]: 2024-09-20 20:21:39.480425089 +0000 UTC m=+0.143222744 container create 843675fd120ffc310cb27a8616ba5591ad229ad0c582cf5185cf6db947ff363d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:21:40.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 podman[83870]: 2024-09-20 20:21:39.674050877 +0000 UTC m=+0.336848499 container init 843675fd120ffc310cb27a8616ba5591ad229ad0c582cf5185cf6db947ff363d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:21:40.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 podman[83870]: 2024-09-20 20:21:39.6769086 +0000 UTC m=+0.339706210 container start 843675fd120ffc310cb27a8616ba5591ad229ad0c582cf5185cf6db947ff363d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T20:21:40.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.700Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T20:21:40.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.700Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T20:21:40.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.700Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi146 (none))" 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.700Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.700Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.701Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.702Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.703Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.703Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.706Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.706Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.28µs 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.706Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.707Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.716Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.717Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-20T20:21:40.018 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.717Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.717Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.495µs wal_replay_duration=11.548019ms wbl_replay_duration=118ns total_replay_duration=11.58833ms 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.719Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.719Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.719Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 bash[83870]: 843675fd120ffc310cb27a8616ba5591ad229ad0c582cf5185cf6db947ff363d 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 systemd[1]: Started Ceph prometheus.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.740Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.031154ms db_storage=784ns remote_storage=1.631µs web_handler=327ns query_engine=641ns scrape=352.423µs scrape_sd=110.472µs notify=14.456µs notify_sd=48.565µs rules=20.167742ms tracing=7.228µs 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.740Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T20:21:40.019 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 20:21:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T20:21:39.740Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T20:21:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:40 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:40 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:40 smithi146 ceph-mon[25603]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T20:21:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:40 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:40 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:40 smithi146 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[23685]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[23685]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[30028]: Failed to get Dashboard config for Prometheus: dashboard get-prometheus-api-host failed: command not known retval: -22 2024-09-20T20:21:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:40 smithi078 ceph-mon[30028]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:41 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.x 2024-09-20T20:21:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:41 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:41 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:41 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:41 smithi146 ceph-mon[25603]: Deploying daemon mgr.x on smithi146 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[23685]: Upgrade: Updating mgr.x 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[23685]: Deploying daemon mgr.x on smithi146 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.x 2024-09-20T20:21:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:41 smithi078 ceph-mon[30028]: Deploying daemon mgr.x on smithi146 2024-09-20T20:21:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:42 smithi078 ceph-mon[23685]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:42 smithi078 ceph-mon[30028]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:43.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:42 smithi146 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:21:43.142 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:42 smithi146 systemd[1]: Stopping Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:43.434 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 podman[84147]: 2024-09-20 20:21:43.139467795 +0000 UTC m=+0.153886048 container died 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0) 2024-09-20T20:21:43.434 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 podman[84147]: 2024-09-20 20:21:43.432854384 +0000 UTC m=+0.447272629 container cleanup 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T20:21:43.766 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 bash[84147]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x 2024-09-20T20:21:43.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 podman[84159]: 2024-09-20 20:21:43.625842553 +0000 UTC m=+0.483961937 container remove 7d2b4071021a98e4947cb176b2587b56235aa8b3c2de92b12365f8f47a76c62a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T20:21:43.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:21:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:43.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:43.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:44.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T20:21:44.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 systemd[1]: Stopped Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:21:44.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:43 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Consumed 1min 1.084s CPU time. 2024-09-20T20:21:44.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 systemd[1]: Starting Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:44.279 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:21:44.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 podman[84286]: 2024-09-20 20:21:44.176131834 +0000 UTC m=+0.016281119 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:21:44.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 podman[84286]: 2024-09-20 20:21:44.311061426 +0000 UTC m=+0.151210738 container create c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T20:21:44.532 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 podman[84286]: 2024-09-20 20:21:44.529848304 +0000 UTC m=+0.369997591 container init c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T20:21:44.808 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 12m - - 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 12m - - 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6m) 14s ago 7m 27.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7m) 13s ago 7m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (55s) 14s ago 5m 54.5M - 3.8 4404209dfb71 ce37493ebb45 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 starting - - - - 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (36s) 14s ago 13m 483M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (13m) 14s ago 13m 63.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (11m) 13s ago 11m 50.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (11m) 14s ago 11m 52.1M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7m) 14s ago 7m 14.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7m) 13s ago 7m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (11m) 14s ago 11m 53.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:21:44.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (10m) 14s ago 10m 61.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (10m) 14s ago 10m 57.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10m) 14s ago 10m 58.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (9m) 13s ago 9m 65.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (9m) 13s ago 9m 58.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (8m) 13s ago 8m 49.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (8m) 13s ago 8m 62.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 starting - - - - 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5m) 14s ago 5m 78.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5m) 13s ago 5m 78.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5m) 14s ago 5m 78.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:21:44.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5m) 13s ago 5m 81.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:21:45.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 podman[84286]: 2024-09-20 20:21:44.533158139 +0000 UTC m=+0.373307417 container start c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T20:21:45.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 ceph-mgr[84305]: -- 172.21.15.146:0/2247075396 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3ebd550e0 con 0x55c3ebd33400 2024-09-20T20:21:45.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 bash[84286]: c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca 2024-09-20T20:21:45.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:44 smithi146 systemd[1]: Started Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:21:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:21:45.153 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "", 2024-09-20T20:21:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T20:21:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:21:45.437 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/2893623179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:45.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:45.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:45 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/2893623179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:21:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:21:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:21:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:45 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2893623179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:21:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[23685]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[23685]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[23685]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[23685]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1727087017' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[30028]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[30028]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[30028]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[30028]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:46 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1727087017' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:21:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:46 smithi146 ceph-mon[25603]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:46 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:46 smithi146 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:46 smithi146 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:46 smithi146 ceph-mon[25603]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:46 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1727087017' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:21:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:47 smithi078 ceph-mon[23685]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:47 smithi078 ceph-mon[23685]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:21:47.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:47 smithi078 ceph-mon[30028]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:47.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:47 smithi078 ceph-mon[30028]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:21:47.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:47 smithi146 ceph-mon[25603]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:21:47.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:47 smithi146 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:21:49.147+0000 7f2eb5d08640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Creating ceph-iscsi config... 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:49.284 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return _run_code(code, main_globals, None, 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: exec(code, run_globals) 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:49.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.wait_async( 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:49.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return future.result(timeout) 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.__get_result() 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: raise self._exception 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: out, err, code = await self._run_cephadm( 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: raise OrchestratorError( 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:49.287 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Creating ceph-iscsi config... 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return _run_code(code, main_globals, None, 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: exec(code, run_globals) 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:49.288 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:49.289 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:49.289 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:49.289 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:49 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:49.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi078.bwbwwj", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.473+0000 7f0954eb1040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:21:49.658 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.473+0000 7f0954eb1040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: ^ 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: SyntaxError: EOL while scanning string literal 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.474+0000 7f0954eb1040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:21:49.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.474+0000 7f0954eb1040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:21:49.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.655+0000 7f0954eb1040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:49.910 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: [20/Sep/2024:20:21:49] ENGINE Bus STARTING 2024-09-20T20:21:49.910 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: CherryPy Checker: 2024-09-20T20:21:49.910 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: The Application mounted at '' has an empty config. 2024-09-20T20:21:49.911 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: 2024-09-20T20:21:49.911 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: [20/Sep/2024:20:21:49] ENGINE Serving on http://:::9283 2024-09-20T20:21:49.911 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:49 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[84301]: [20/Sep/2024:20:21:49] ENGINE Bus STARTED 2024-09-20T20:21:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.wait_async( 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: return future.result(timeout) 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.__get_result() 2024-09-20T20:21:50.539 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise self._exception 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:50.540 INFO:journalctl@ceph.mon.a.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:50.541 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.542 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:50.543 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.wait_async( 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: return future.result(timeout) 2024-09-20T20:21:50.544 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.__get_result() 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise self._exception 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: raise OrchestratorError( 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:50.545 INFO:journalctl@ceph.mon.c.smithi078.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: exec(code, run_globals) 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:50.546 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: Standby manager daemon x restarted 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: Standby manager daemon x started 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:50.548 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:50.612 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:50.613 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in _check_daemons 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.wait_async( 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T20:21:50.614 INFO:journalctl@ceph.mon.b.smithi146.stdout: return future.result(timeout) 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.__get_result() 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise self._exception 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1404, in _create_daemon 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: out, err, code = await self._run_cephadm( 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1747, in _run_cephadm 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise OrchestratorError( 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi-foo-smithi078-bwbwwj 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.615 INFO:journalctl@ceph.mon.b.smithi146.stdout: /usr/bin/podman: stderr Error: no such container ceph-f2df6e16-778b-11ef-baf5-efdc52797490-iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: Reconfig daemon iscsi.foo.smithi078.bwbwwj ... 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/iscsi-gateway.cfg 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/iscsi.foo.smithi078.bwbwwj/tcmu-runner-entrypoint.sh 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-09-20T20:21:50.616 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5628, in 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 5616, in main 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/tmp/tmp7xifg09j.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj: Job for ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service failed because the control process exited with error code. 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" and "journalctl -xeu ceph-f2df6e16-778b-11ef-baf5-efdc52797490@iscsi.foo.smithi078.bwbwwj.service" for details. 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi078.bwbwwj (dependencies changed)... 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi078.bwbwwj on smithi078 2024-09-20T20:21:50.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:21:50.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:21:50.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:50 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:50.926 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:21:50] "GET /metrics HTTP/1.1" 200 38658 "" "Prometheus/2.43.0" 2024-09-20T20:21:51.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:51 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:51.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:51 smithi146 ceph-mon[25603]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:51.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:51 smithi146 ceph-mon[25603]: mgrmap e35: y(active, since 26s), standbys: x 2024-09-20T20:21:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:51 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:51 smithi078 ceph-mon[23685]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:51 smithi078 ceph-mon[23685]: mgrmap e35: y(active, since 26s), standbys: x 2024-09-20T20:21:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:51 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:21:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:51 smithi078 ceph-mon[30028]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:21:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:51 smithi078 ceph-mon[30028]: mgrmap e35: y(active, since 26s), standbys: x 2024-09-20T20:21:52.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:52.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:52.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:52.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[30028]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T20:21:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:52.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:52.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:52.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:52 smithi078 ceph-mon[23685]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T20:21:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:52 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:52 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:52 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:52 smithi146 ceph-mon[25603]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T20:21:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:53.517Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:21:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:21:53.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:21:54.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:54.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:54.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[30028]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T20:21:54.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:54.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1792951267' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:21:54.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:54.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:54.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[23685]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T20:21:54.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:54.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:54 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1792951267' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:21:54.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:54 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:54.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:54 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:54.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:54 smithi146 ceph-mon[25603]: Failed to get Dashboard config for iSCSI: dashboard iscsi-gateway-list failed: command not known retval: -22 2024-09-20T20:21:54.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:54 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:21:54.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:54 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1792951267' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T20:21:55.284 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:55 smithi146 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T20:21:55.284 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:55 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:55 smithi078 ceph-mon[23685]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T20:21:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:55 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:55 smithi078 ceph-mon[30028]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T20:21:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:55 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:21:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:57 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:21:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:21:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:21:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:57 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:21:58.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:58 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.x 2024-09-20T20:21:58.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:58 smithi146 ceph-mon[25603]: Deploying daemon mgr.x on smithi146 2024-09-20T20:21:58.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:58 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:58 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:58 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[23685]: Upgrade: Updating mgr.x 2024-09-20T20:21:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[23685]: Deploying daemon mgr.x on smithi146 2024-09-20T20:21:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.x 2024-09-20T20:21:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[30028]: Deploying daemon mgr.x on smithi146 2024-09-20T20:21:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:58.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:58 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:58 smithi146 systemd[1]: Stopping Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:21:59.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:59 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:59 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:59 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:21:59 smithi146 ceph-mon[25603]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:21:59.406 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 podman[85586]: 2024-09-20 20:21:59.141595033 +0000 UTC m=+0.233848776 container died c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[23685]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:21:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:21:59 smithi078 ceph-mon[30028]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:21:59.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 podman[85586]: 2024-09-20 20:21:59.485685003 +0000 UTC m=+0.577938745 container cleanup c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T20:21:59.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 bash[85586]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x 2024-09-20T20:22:00.023 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 podman[85599]: 2024-09-20 20:21:59.770082483 +0000 UTC m=+0.627122374 container remove c8ebd78969a51558e4bbe2474ca996292cf2c48cb15b0bd0f66d2bface1e1bca (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-20T20:22:00.023 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:22:00.023 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T20:22:00.023 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 systemd[1]: Stopped Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:00.023 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:21:59 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Consumed 5.967s CPU time. 2024-09-20T20:22:00.335 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 systemd[1]: Starting Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:00.594 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 podman[85730]: 2024-09-20 20:22:00.236672264 +0000 UTC m=+0.016358392 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:22:00.594 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 podman[85730]: 2024-09-20 20:22:00.354607829 +0000 UTC m=+0.134293950 container create 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:22:00.594 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 podman[85730]: 2024-09-20 20:22:00.565310796 +0000 UTC m=+0.344996916 container init 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-20T20:22:00.594 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 podman[85730]: 2024-09-20 20:22:00.568206493 +0000 UTC m=+0.347892612 container start 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-20T20:22:01.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 ceph-mgr[85749]: -- 172.21.15.146:0/762475979 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f83d7110e0 con 0x55f83d6ef000 2024-09-20T20:22:01.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 bash[85730]: 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 2024-09-20T20:22:01.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:00 smithi146 systemd[1]: Started Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:22:00] "GET /metrics HTTP/1.1" 200 38655 "" "Prometheus/2.43.0" 2024-09-20T20:22:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:22:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:01 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-20T20:22:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:01.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:01 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.466 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:02 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:02 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:03 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:03.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:03.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:03.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:03.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:03.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:04 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:04.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:04 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T20:22:05.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:05.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:05.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:05 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:05.974 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:05.819+0000 7f5aadd48040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:05.819+0000 7f5aadd48040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: ^ 2024-09-20T20:22:05.975 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: SyntaxError: EOL while scanning string literal 2024-09-20T20:22:05.976 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:05.976 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:05.820+0000 7f5aadd48040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:22:05.976 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:05 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:05.820+0000 7f5aadd48040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:06.012+0000 7f5aadd48040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: [20/Sep/2024:20:22:06] ENGINE Bus STARTING 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: CherryPy Checker: 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: The Application mounted at '' has an empty config. 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: [20/Sep/2024:20:22:06] ENGINE Serving on http://:::9283 2024-09-20T20:22:06.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: [20/Sep/2024:20:22:06] ENGINE Bus STARTED 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.620 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:06 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Standby manager daemon x restarted 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: Standby manager daemon x started 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:06.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:06.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:06 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: mgrmap e36: y(active, since 42s), standbys: x 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:07 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: mgrmap e36: y(active, since 42s), standbys: x 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: mgrmap e36: y(active, since 42s), standbys: x 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:07 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:08 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:08.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:08.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:08.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:08.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:08 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:09 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:09.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:09 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:10 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:22:10] "GET /metrics HTTP/1.1" 200 38657 "" "Prometheus/2.43.0" 2024-09-20T20:22:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:11 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:11.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:11.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:11 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.316 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.317 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:12 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:12 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:13 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:13.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:13.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:14.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:14.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.525 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.526 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.527 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.528 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.529 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.530 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.530 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.530 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.530 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.530 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.530 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:14 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:14.738 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:14.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:14 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.360 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.361 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.361 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.361 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.361 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.361 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:15 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:15.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:15 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.073 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:16.396 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:22:16.401 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:22:16.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:16 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T20:22:16.635 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:22:16.635 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 12m - - 2024-09-20T20:22:16.635 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 12m - - 2024-09-20T20:22:16.635 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7m) 0s ago 8m 27.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:22:16.635 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7m) 0s ago 7m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (24s) 0s ago 5m 54.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (16s) 0s ago 12m 221M - 19.3.0-5127-ge8baca1e 82f5f45939a4 9fa5df75e3da 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (68s) 0s ago 13m 499M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (13m) 0s ago 13m 66.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (12m) 0s ago 12m 52.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (12m) 0s ago 12m 54.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7m) 0s ago 7m 14.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7m) 0s ago 7m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (11m) 0s ago 11m 53.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (11m) 0s ago 11m 61.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (10m) 0s ago 10m 58.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10m) 0s ago 10m 59.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (10m) 0s ago 10m 65.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (9m) 0s ago 9m 58.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (9m) 0s ago 9m 49.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:22:16.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (9m) 0s ago 9m 62.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:22:16.637 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (37s) 0s ago 8m 35.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:22:16.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6m) 0s ago 6m 79.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:22:16.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6m) 0s ago 6m 78.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:22:16.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6m) 0s ago 6m 79.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:22:16.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6m) 0s ago 6m 81.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:22:16.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T20:22:16.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T20:22:16.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:22:16.742 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:16 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:22:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:22:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T20:22:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:22:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:22:16.953 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:22:17.229 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:22:17.230 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:22:17.547 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:17.548 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Upgrade: Setting container_image for all mgr 2024-09-20T20:22:17.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: Upgrade: It appears safe to stop mon.a 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1972683316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:17.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Upgrade: Setting container_image for all mgr 2024-09-20T20:22:17.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: Upgrade: It appears safe to stop mon.a 2024-09-20T20:22:17.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:17.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:17.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:17 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/1972683316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Upgrade: Setting container_image for all mgr 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: Upgrade: It appears safe to stop mon.a 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:17 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1972683316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:18.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3848371818' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:22:18.388 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:18 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='client.? 172.21.15.78:0/3848371818' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:22:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[23685]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3848371818' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:22:18.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:18 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-mon[30028]: Upgrade: Updating mon.a 2024-09-20T20:22:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-mon[30028]: Deploying daemon mon.a on smithi078 2024-09-20T20:22:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-mon[30028]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-mon[23685]: Upgrade: Updating mon.a 2024-09-20T20:22:19.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-mon[23685]: Deploying daemon mon.a on smithi078 2024-09-20T20:22:19.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-mon[23685]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:19.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 systemd[1]: Stopping Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:19 smithi146 ceph-mon[25603]: Upgrade: Updating mon.a 2024-09-20T20:22:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:19 smithi146 ceph-mon[25603]: Deploying daemon mon.a on smithi078 2024-09-20T20:22:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:19 smithi146 ceph-mon[25603]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:20.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:19] ENGINE Bus STOPPING 2024-09-20T20:22:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[23681]: 2024-09-20T20:22:19.614+0000 7f388579c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:22:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[23681]: 2024-09-20T20:22:19.614+0000 7f388579c640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T20:22:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:19 smithi078 podman[87769]: 2024-09-20 20:22:19.828947776 +0000 UTC m=+0.354153670 container died e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, RELEASE=reef-0b2cc4d, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T20:22:20.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 podman[87769]: 2024-09-20 20:22:20.206361989 +0000 UTC m=+0.731567876 container cleanup e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T20:22:20.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 bash[87769]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a 2024-09-20T20:22:20.476 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:22:20.476 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:20] ENGINE Bus STOPPED 2024-09-20T20:22:20.477 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:20] ENGINE Bus STARTING 2024-09-20T20:22:20.477 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:20] ENGINE Serving on http://:::9283 2024-09-20T20:22:20.477 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:20] ENGINE Bus STARTED 2024-09-20T20:22:20.744 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:22:20] "GET /metrics HTTP/1.1" 200 38657 "" "Prometheus/2.43.0" 2024-09-20T20:22:20.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 podman[87782]: 2024-09-20 20:22:20.474501231 +0000 UTC m=+0.639783844 container remove e445af1fa8971f4ef6205bffaf6259c132dad8f64005c7957144efdb92d364b2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, io.buildah.version=1.37.2, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:22:20.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T20:22:20.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 systemd[1]: Stopped Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:20.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service: Consumed 18.205s CPU time. 2024-09-20T20:22:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:20 smithi078 systemd[1]: Starting Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:21.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 podman[87924]: 2024-09-20 20:22:20.953722849 +0000 UTC m=+0.015612165 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:22:21.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 podman[87924]: 2024-09-20 20:22:21.072600168 +0000 UTC m=+0.134489483 container create a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:22:21.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 podman[87924]: 2024-09-20 20:22:21.287124315 +0000 UTC m=+0.349013636 container init a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-20T20:22:21.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 podman[87924]: 2024-09-20 20:22:21.290673164 +0000 UTC m=+0.352562479 container start a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:22:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:22:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T20:22:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: pidfile_write: ignore empty --pid-file 2024-09-20T20:22:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: load: jerasure load: lrc 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Git sha 0 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: DB SUMMARY 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: DB Session ID: 5XYKOO792GTB9G99RBUY 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: MANIFEST file: MANIFEST-000015 size: 2489 Bytes 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000044.sst 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000042.log size: 2277003 ; 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:22:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.env: 0x55c845eab120 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.info_log: 0x55c848587700 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.statistics: (nil) 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.use_fsync: 0 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:22:21.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.db_log_dir: 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.wal_dir: 2024-09-20T20:22:21.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:22:21.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:22:21.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:22:21.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:22:21.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.write_buffer_manager: 0x55c84858a1e0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.unordered_write: 0 2024-09-20T20:22:21.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.row_cache: None 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.wal_filter: None 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.wal_compression: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:22:21.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:22:21.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_open_files: -1 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Compression algorithms supported: 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kZSTD supported: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:22:21.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.merge_operator: 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_filter: None 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:22:21.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c848587320) 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x55c84857b610 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:22:21.796 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-09-20T20:22:21.797 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression: NoCompression 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:22:21.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.num_levels: 7 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:22:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:22:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:22:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:22:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:22:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.ttl: 2592000 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:22:21.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 46, last_sequence is 21235, log_number is 42,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 42 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 42 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d6e19c1c-7286-43f7-95ba-2c0353c411d2 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863741317205, "job": 1, "event": "recovery_started", "wal_files": [42]} 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #42 mode 2 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863741325107, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 47, "file_size": 2127824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 21236, "largest_seqno": 24116, "table_properties": {"data_size": 2118227, "index_size": 4328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4229, "raw_key_size": 41270, "raw_average_key_size": 24, "raw_value_size": 2091058, "raw_average_value_size": 1238, "num_data_blocks": 176, "num_entries": 1688, "num_filter_entries": 1688, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726863741, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d6e19c1c-7286-43f7-95ba-2c0353c411d2", "db_session_id": "5XYKOO792GTB9G99RBUY", "orig_file_number": 47, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863741325228, "job": 1, "event": "recovery_finished"} 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/version_set.cc:5047] Creating manifest 49 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000042.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8485aae00 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: DB pointer 0x55c8486ac000 2024-09-20T20:22:21.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:22:21.806 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 1/0 2.03 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: L6 1/0 9.54 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 11.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 530.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:22:21.807 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 165.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 165.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x55c84857b610#2 capacity: 512.00 MB usage: 9.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???) e3 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).mds e1 new map 2024-09-20T20:22:21.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).mds e1 print_map 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 ceph-mon[87942]: mon.a@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 bash[87924]: a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 2024-09-20T20:22:21.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:21 smithi078 systemd[1]: Started Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: ignoring --setuser ceph since I am not root 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: ignoring --setgroup ceph since I am not root 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mgr[85749]: -- 172.21.15.146:0/3733487721 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5555a353b0e0 con 0x5555a3519000 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: mon.a calling monitor election 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: monmap epoch 3 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: min_mon_release 18 (reef) 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: election_strategy: 1 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: fsmap 2024-09-20T20:22:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T20:22:23.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: mgrmap e36: y(active, since 58s), standbys: x 2024-09-20T20:22:23.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: from='mgr.25262 ' entity='' 2024-09-20T20:22:23.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:22 smithi146 ceph-mon[25603]: mgrmap e37: y(active, since 58s), standbys: x 2024-09-20T20:22:23.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ignoring --setuser ceph since I am not root 2024-09-20T20:22:23.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ignoring --setgroup ceph since I am not root 2024-09-20T20:22:23.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mgr[76550]: -- 172.21.15.78:0/2446310455 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558c572db0e0 con 0x558c572b8800 2024-09-20T20:22:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:22:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: mon.a calling monitor election 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: monmap epoch 3 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: min_mon_release 18 (reef) 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: election_strategy: 1 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:22:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: fsmap 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: mgrmap e36: y(active, since 58s), standbys: x 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: from='mgr.25262 ' entity='' 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[87942]: mgrmap e37: y(active, since 58s), standbys: x 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:22:23.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: from='mgr.25262 172.21.15.78:0/1471396352' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: mon.a calling monitor election 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: monmap epoch 3 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: min_mon_release 18 (reef) 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: election_strategy: 1 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:22:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: fsmap 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: mgrmap e36: y(active, since 58s), standbys: x 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: from='mgr.25262 ' entity='' 2024-09-20T20:22:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:22 smithi078 ceph-mon[30028]: mgrmap e37: y(active, since 58s), standbys: x 2024-09-20T20:22:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:23.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:23.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: Active manager daemon y restarted 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: Activating manager daemon y 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: mgrmap e38: y(active, starting, since 0.00693265s), standbys: x 2024-09-20T20:22:27.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:22:27.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.557+0000 7fb73b436040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.557+0000 7fb73b436040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:22:27.843 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: ^ 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: SyntaxError: EOL while scanning string literal 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.558+0000 7fb73b436040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.558+0000 7fb73b436040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.732+0000 7fb73b436040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: [20/Sep/2024:20:22:27] ENGINE Bus STARTING 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: CherryPy Checker: 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: The Application mounted at '' has an empty config. 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: [20/Sep/2024:20:22:27] ENGINE Serving on http://:::9283 2024-09-20T20:22:27.844 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:27 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[85745]: [20/Sep/2024:20:22:27] ENGINE Bus STARTED 2024-09-20T20:22:28.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:27.586+0000 7fd02bff3040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:22:28.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:27.586+0000 7fd02bff3040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:22:28.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:22:28.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:22:28.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ^ 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: SyntaxError: EOL while scanning string literal 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:27.587+0000 7fd02bff3040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:27.587+0000 7fd02bff3040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:22:28.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:27.764+0000 7fd02bff3040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: Standby manager daemon x restarted 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: Standby manager daemon x started 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: Active manager daemon y restarted 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: Activating manager daemon y 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: mgrmap e38: y(active, starting, since 0.00693265s), standbys: x 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:22:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:22:28.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: Active manager daemon y restarted 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: Activating manager daemon y 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: mgrmap e38: y(active, starting, since 0.00693265s), standbys: x 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T20:22:28.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:27 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: Manager daemon y is now available 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:22:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"} : dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"} : dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:22:29.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:28 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:28.797+0000 7fcfc9d8b640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:28] ENGINE Bus STARTING 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: CherryPy Checker: 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: The Application mounted at '' has an empty config. 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:28] ENGINE Serving on http://:::9283 2024-09-20T20:22:29.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:22:28] ENGINE Bus STARTED 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: Manager daemon y is now available 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"} : dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"} : dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:22:29.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: Manager daemon y is now available 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"} : dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"} : dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:29.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:29.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T20:22:29.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-20T20:22:29.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T20:22:29.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:28 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-20T20:22:29.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:29.435+0000 7fcf8a1fe640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:20:22:29] ENGINE Error in HTTPServer.serve 2024-09-20T20:22:29.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: Traceback (most recent call last): 2024-09-20T20:22:29.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:22:29.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self._connections.run(self.expiration_interval) 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self._run(expiration_interval) 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: s = self.context.wrap_socket( 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self.sslsocket_class._create( 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self.do_handshake() 2024-09-20T20:22:29.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:22:29.788 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self._sslobj.do_handshake() 2024-09-20T20:22:29.788 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:22:29.788 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:22:29.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:22:29.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: mgrmap e39: y(active, since 1.03068s), standbys: x 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:29.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:29 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: mgrmap e39: y(active, since 1.03068s), standbys: x 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi146:/etc/ceph/ceph.conf 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi146:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: Updating smithi078:/var/lib/ceph/f2df6e16-778b-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: mgrmap e39: y(active, since 1.03068s), standbys: x 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:30.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:30.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:29 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:30.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:22:30.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:22:30.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: [20/Sep/2024:20:22:29] ENGINE Bus STARTING 2024-09-20T20:22:30.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: [20/Sep/2024:20:22:29] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:22:30.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: [20/Sep/2024:20:22:29] ENGINE Serving on https://172.21.15.78:7151 2024-09-20T20:22:30.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: [20/Sep/2024:20:22:29] ENGINE Bus STARTED 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: [20/Sep/2024:20:22:29] ENGINE Error in HTTPServer.serve 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._run(expiration_interval) 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: s = self.context.wrap_socket( 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:22:30.944 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.sslsocket_class._create( 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.do_handshake() 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout: self._sslobj.do_handshake() 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:22:30.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:30 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:22:30] "GET /metrics HTTP/1.1" 200 35855 "" "Prometheus/2.43.0" 2024-09-20T20:22:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:22:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:22:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: [20/Sep/2024:20:22:29] ENGINE Bus STARTING 2024-09-20T20:22:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: [20/Sep/2024:20:22:29] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: [20/Sep/2024:20:22:29] ENGINE Serving on https://172.21.15.78:7151 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: [20/Sep/2024:20:22:29] ENGINE Bus STARTED 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: [20/Sep/2024:20:22:29] ENGINE Error in HTTPServer.serve 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:22:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:22:31.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: [20/Sep/2024:20:22:29] ENGINE Bus STARTING 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: [20/Sep/2024:20:22:29] ENGINE Serving on http://172.21.15.78:8765 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: [20/Sep/2024:20:22:29] ENGINE Serving on https://172.21.15.78:7151 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: [20/Sep/2024:20:22:29] ENGINE Bus STARTED 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: [20/Sep/2024:20:22:29] ENGINE Error in HTTPServer.serve 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: Traceback (most recent call last): 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T20:22:31.039 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._run(expiration_interval) 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: s = self.context.wrap_socket( 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: return self.sslsocket_class._create( 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: self.do_handshake() 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T20:22:31.040 INFO:journalctl@ceph.mon.c.smithi078.stdout: self._sslobj.do_handshake() 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: mgrmap e40: y(active, since 2s), standbys: x 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:22:31.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:30 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:31.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:31 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.x 2024-09-20T20:22:31.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:31 smithi146 ceph-mon[25603]: Deploying daemon mgr.x on smithi146 2024-09-20T20:22:31.871 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:31 smithi146 systemd[1]: Stopping Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:31 smithi078 ceph-mon[87942]: Upgrade: Updating mgr.x 2024-09-20T20:22:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:31 smithi078 ceph-mon[87942]: Deploying daemon mgr.x on smithi146 2024-09-20T20:22:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:31 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.x 2024-09-20T20:22:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:31 smithi078 ceph-mon[30028]: Deploying daemon mgr.x on smithi146 2024-09-20T20:22:32.426 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 podman[90906]: 2024-09-20 20:22:32.155238727 +0000 UTC m=+0.142698854 container died 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:22:32.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 podman[90906]: 2024-09-20 20:22:32.515472108 +0000 UTC m=+0.502932234 container cleanup 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:22:32.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 bash[90906]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:32 smithi146 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:32 smithi146 ceph-mon[25603]: mgrmap e41: y(active, since 4s), standbys: x 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 podman[90920]: 2024-09-20 20:22:32.774866494 +0000 UTC m=+0.619157201 container remove 9fa5df75e3da0b0d1a7641ba115d144a1f178b98ec4b3949285f36258ac35ca4 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 systemd[1]: Stopped Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:33.071 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:32 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Consumed 11.254s CPU time. 2024-09-20T20:22:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:32 smithi078 ceph-mon[87942]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:32 smithi078 ceph-mon[87942]: mgrmap e41: y(active, since 4s), standbys: x 2024-09-20T20:22:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:32 smithi078 ceph-mon[30028]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:32 smithi078 ceph-mon[30028]: mgrmap e41: y(active, since 4s), standbys: x 2024-09-20T20:22:33.386 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 systemd[1]: Starting Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:33.688 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 podman[91150]: 2024-09-20 20:22:33.288096835 +0000 UTC m=+0.017267144 image pull 4404209dfb7164a89dfef6a4afd640826057f9881e9f406755db840649fd95fd quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce 2024-09-20T20:22:33.688 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 podman[91150]: 2024-09-20 20:22:33.392546466 +0000 UTC m=+0.121716812 container create b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:22:33.688 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 podman[91150]: 2024-09-20 20:22:33.587002677 +0000 UTC m=+0.316172993 container init b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T20:22:33.688 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 podman[91150]: 2024-09-20 20:22:33.590179579 +0000 UTC m=+0.319349890 container start b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:22:33.689 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 ceph-mgr[91187]: -- 172.21.15.146:0/2951282529 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e007e0d00 con 0x559e007b5400 2024-09-20T20:22:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:33.520Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:33.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:22:33.945 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 bash[91150]: b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 2024-09-20T20:22:33.945 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 systemd[1]: Started Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:33.945 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:33.711+0000 7fa55c7ea040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T20:22:33.945 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:33.780+0000 7fa55c7ea040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T20:22:34.266 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:33.942+0000 7fa55c7ea040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T20:22:34.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:34.010+0000 7fa55c7ea040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T20:22:34.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:34.159+0000 7fa55c7ea040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:22:34.630 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:34.308+0000 7fa55c7ea040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T20:22:34.919 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:34.627+0000 7fa55c7ea040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T20:22:34.919 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:34.695+0000 7fa55c7ea040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T20:22:34.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:34.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []} : dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []} : dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:22:34.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []} : dispatch 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []} : dispatch 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []} : dispatch 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:34.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []} : dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []}]: dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []} : dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi146", "caps": []} : dispatch 2024-09-20T20:22:34.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:22:34.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:34.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:34.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:34.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []}]: dispatch 2024-09-20T20:22:34.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:34 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.agent.smithi078", "caps": []} : dispatch 2024-09-20T20:22:35.196 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:34 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:34.916+0000 7fa55c7ea040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:35.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:36.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:35 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.266 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:35.869+0000 7fa55c7ea040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T20:22:36.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:35 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:35.940+0000 7fa55c7ea040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:36 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.947 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:36.692+0000 7fa55c7ea040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T20:22:36.947 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:36.823+0000 7fa55c7ea040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T20:22:36.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T20:22:36.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:36.951 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:37.208 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:36.972+0000 7fa55c7ea040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T20:22:37.208 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.036+0000 7fa55c7ea040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T20:22:37.208 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.191+0000 7fa55c7ea040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T20:22:37.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.264+0000 7fa55c7ea040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T20:22:37.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.329+0000 7fa55c7ea040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T20:22:37.501 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.499+0000 7fa55c7ea040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T20:22:37.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.570+0000 7fa55c7ea040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T20:22:37.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.654+0000 7fa55c7ea040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T20:22:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:37 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.266 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.766+0000 7fa55c7ea040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T20:22:38.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.830+0000 7fa55c7ea040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T20:22:38.267 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:37.892+0000 7fa55c7ea040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T20:22:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:38.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:37 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:38.727+0000 7fa55c7ea040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T20:22:39.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: [20/Sep/2024:20:22:38] ENGINE Bus STARTING 2024-09-20T20:22:39.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: CherryPy Checker: 2024-09-20T20:22:39.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: The Application mounted at '' has an empty config. 2024-09-20T20:22:39.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: 2024-09-20T20:22:39.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: [20/Sep/2024:20:22:38] ENGINE Serving on http://:::9283 2024-09-20T20:22:39.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[91181]: [20/Sep/2024:20:22:38] ENGINE Bus STARTED 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:22:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:22:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:38 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Standby manager daemon x restarted 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: Standby manager daemon x started 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:22:39.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.? 172.21.15.146:0/3579986761' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:38 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: mgrmap e42: y(active, since 11s), standbys: x 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: mgrmap e42: y(active, since 11s), standbys: x 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:39.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:39 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: mgrmap e42: y(active, since 11s), standbys: x 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:39 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:40.815 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:22:40] "GET /metrics HTTP/1.1" 200 35855 "" "Prometheus/2.43.0" 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:22:41.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T20:22:41.148 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:22:40.983Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=7 2024-09-20T20:22:41.148 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:22:41.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:22:41.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:22:41.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:40 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T20:22:41.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: Upgrade: 1 mgr daemon(s) are 18.2.4-869-g0b2cc4d4 != target 19.3.0-5127-ge8baca1e 2024-09-20T20:22:41.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: Upgrade: It appears safe to stop mon.a 2024-09-20T20:22:41.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:41.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:41 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: Upgrade: 1 mgr daemon(s) are 18.2.4-869-g0b2cc4d4 != target 19.3.0-5127-ge8baca1e 2024-09-20T20:22:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: Upgrade: It appears safe to stop mon.a 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: Upgrade: 1 mgr daemon(s) are 18.2.4-869-g0b2cc4d4 != target 19.3.0-5127-ge8baca1e 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: Upgrade: It appears safe to stop mon.a 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:41 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[87942]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:42.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:42 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:43.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:42 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:43 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:44.239 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:43 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:44.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:43 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[87942]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:22:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:22:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:22:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[87942]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[30028]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:22:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:22:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:22:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:44 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:44 smithi146 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-20T20:22:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:44 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:22:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:44 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:22:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:44 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:45 smithi078 ceph-mon[30028]: Upgrade: Updating mon.a 2024-09-20T20:22:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:45 smithi078 ceph-mon[30028]: Deploying daemon mon.a on smithi078 2024-09-20T20:22:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:45 smithi078 ceph-mon[87942]: Upgrade: Updating mon.a 2024-09-20T20:22:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:45 smithi078 ceph-mon[87942]: Deploying daemon mon.a on smithi078 2024-09-20T20:22:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:45 smithi146 ceph-mon[25603]: Upgrade: Updating mon.a 2024-09-20T20:22:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:45 smithi146 ceph-mon[25603]: Deploying daemon mon.a on smithi078 2024-09-20T20:22:47.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:46 smithi078 ceph-mon[87942]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:22:47.115 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:46 smithi078 ceph-mon[30028]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:22:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:46 smithi146 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T20:22:47.848 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:22:48.382 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:22:48.382 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 13m - - 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 13m - - 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7m) 5s ago 8m 27.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (8m) 6s ago 8m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (56s) 5s ago 6m 54.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (15s) 6s ago 12m 439M - 18.2.4-869-g0b2cc4d4 4404209dfb71 b9f27a3e10d9 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (100s) 5s ago 14m 499M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (14m) 5s ago 14m 66.4M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 e445af1fa897 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (13m) 6s ago 13m 59.9M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (12m) 5s ago 12m 54.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (8m) 5s ago 8m 14.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (8m) 6s ago 8m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (12m) 5s ago 12m 53.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:22:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (11m) 5s ago 11m 61.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (11m) 5s ago 11m 58.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (11m) 5s ago 11m 59.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (10m) 6s ago 10m 66.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (10m) 6s ago 10m 59.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (9m) 6s ago 9m 50.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (9m) 6s ago 9m 62.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (69s) 6s ago 8m 40.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6m) 5s ago 6m 79.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6m) 6s ago 6m 79.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6m) 5s ago 6m 79.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:22:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6m) 6s ago 6m 82.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:22:48.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 15, 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:22:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:48 smithi078 ceph-mon[30028]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:48 smithi078 ceph-mon[30028]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:48 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3867809417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:48.991 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:22:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:22:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:22:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:22:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [], 2024-09-20T20:22:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T20:22:48.992 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T20:22:48.992 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:22:48.992 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:22:48.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:48 smithi078 ceph-mon[87942]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:48.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:48 smithi078 ceph-mon[87942]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:48.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:48 smithi078 ceph-mon[87942]: from='client.? 172.21.15.78:0/3867809417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:48 smithi146 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:48 smithi146 ceph-mon[25603]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:48 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3867809417' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:22:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:48 smithi078 systemd[1]: Stopping Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:49.337 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:49.337 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:49.337 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[87938]: 2024-09-20T20:22:49.346+0000 7fd63bcb0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:22:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[87938]: 2024-09-20T20:22:49.346+0000 7fd63bcb0640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T20:22:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:49 smithi078 podman[90808]: 2024-09-20 20:22:49.519645759 +0000 UTC m=+0.302359553 container died a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:22:50.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:49 smithi078 podman[90808]: 2024-09-20 20:22:49.897004105 +0000 UTC m=+0.679717903 container cleanup a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T20:22:50.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:49 smithi078 bash[90808]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a 2024-09-20T20:22:50.913 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:22:50] "GET /metrics HTTP/1.1" 200 38656 "" "Prometheus/2.43.0" 2024-09-20T20:22:50.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:50 smithi078 podman[90841]: 2024-09-20 20:22:50.567475233 +0000 UTC m=+1.048949422 container remove a8370246e1d5f34d5bd8de4fc5f184aa9432621ee15a34e4dfc6894680158129 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=) 2024-09-20T20:22:50.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:50 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T20:22:50.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:50 smithi078 systemd[1]: Stopped Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:50.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:50 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.a.service: Consumed 1.655s CPU time. 2024-09-20T20:22:51.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:50 smithi078 systemd[1]: Starting Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:22:51.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 podman[91091]: 2024-09-20 20:22:51.12723438 +0000 UTC m=+0.112170315 container create ddc9330d21c8c1be9d954cf0d47795042033703449ce1d3b5e86ee37f76b837c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:22:51.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 podman[91091]: 2024-09-20 20:22:51.036297908 +0000 UTC m=+0.021233859 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:22:51.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 podman[91091]: 2024-09-20 20:22:51.396711172 +0000 UTC m=+0.381647116 container init ddc9330d21c8c1be9d954cf0d47795042033703449ce1d3b5e86ee37f76b837c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T20:22:51.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 podman[91091]: 2024-09-20 20:22:51.400108404 +0000 UTC m=+0.385044339 container start ddc9330d21c8c1be9d954cf0d47795042033703449ce1d3b5e86ee37f76b837c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: pidfile_write: ignore empty --pid-file 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: load: jerasure load: lrc 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Git sha 0 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: DB SUMMARY 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: DB Session ID: I19OTH9KYU5NWTYAZ9AK 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: MANIFEST file: MANIFEST-000049 size: 554 Bytes 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000054.sst 2024-09-20T20:22:51.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000052.log size: 2444808 ; 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.env: 0x564a6b2fa120 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.info_log: 0x564a6c179700 2024-09-20T20:22:51.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.statistics: (nil) 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.use_fsync: 0 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:22:51.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.db_log_dir: 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.wal_dir: 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:22:51.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.write_buffer_manager: 0x564a6c17c1e0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.unordered_write: 0 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:22:51.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.row_cache: None 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.wal_filter: None 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.wal_compression: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:22:51.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:22:51.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_open_files: -1 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Compression algorithms supported: 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kZSTD supported: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:22:51.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000049 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.merge_operator: 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_filter: None 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564a6c179320) 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:22:51.507 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x564a6c16d610 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-09-20T20:22:51.508 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-09-20T20:22:51.509 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression: NoCompression 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.num_levels: 7 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:22:51.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:22:51.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:22:51.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:22:51.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:22:51.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:22:51.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:22:51.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:22:51.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:22:51.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:22:51.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:22:51.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.ttl: 2592000 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:22:51.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000049 succeeded,manifest_file_number is 49, next_file_number is 56, last_sequence is 24439, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 52 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d6e19c1c-7286-43f7-95ba-2c0353c411d2 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863771428360, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863771437593, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 2291318, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 24440, "largest_seqno": 25883, "table_properties": {"data_size": 2284894, "index_size": 3073, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2309, "raw_key_size": 19195, "raw_average_key_size": 21, "raw_value_size": 2269426, "raw_average_value_size": 2502, "num_data_blocks": 135, "num_entries": 907, "num_filter_entries": 907, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726863771, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d6e19c1c-7286-43f7-95ba-2c0353c411d2", "db_session_id": "I19OTH9KYU5NWTYAZ9AK", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863771437759, "job": 1, "event": "recovery_finished"} 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564a6c19ce00 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: DB pointer 0x564a6c29e000 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:22:51.517 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 1/0 2.19 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 586.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: L6 1/0 10.53 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 12.72 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 586.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:51.518 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 586.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 586.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:22:51.519 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 159.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 159.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x564a6c16d610#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.53 KB,0.000482798%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???) e3 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).mds e1 new map 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).mds e1 print_map 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T20:22:51.520 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 ceph-mon[91136]: mon.a@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 bash[91091]: ddc9330d21c8c1be9d954cf0d47795042033703449ce1d3b5e86ee37f76b837c 2024-09-20T20:22:51.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:51 smithi078 systemd[1]: Started Ceph mon.a for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:22:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: mon.a calling monitor election 2024-09-20T20:22:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:22:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: monmap epoch 3 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: min_mon_release 18 (reef) 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: election_strategy: 1 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: fsmap 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: mgrmap e42: y(active, since 25s), standbys: x 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='' 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='' 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:52 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: mon.a calling monitor election 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: monmap epoch 3 2024-09-20T20:22:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: min_mon_release 18 (reef) 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: election_strategy: 1 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: fsmap 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: mgrmap e42: y(active, since 25s), standbys: x 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='' 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='' 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:22:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: mon.a calling monitor election 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: monmap epoch 3 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: min_mon_release 18 (reef) 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: election_strategy: 1 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: fsmap 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:22:53.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: mgrmap e42: y(active, since 25s), standbys: x 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='' 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='' 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:52 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:53.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:53 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:54.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:53 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:54 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:55.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:54 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:55 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:22:56.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:56 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:57.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:56 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:22:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:22:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:22:59.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:22:59.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:22:59.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:22:59.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:22:59.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:22:58 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:23:00.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:23:00.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:00.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:00.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:00.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:00.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T20:23:00.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:23:00.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T20:23:00.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:23:00] "GET /metrics HTTP/1.1" 200 38650 "" "Prometheus/2.43.0" 2024-09-20T20:23:01.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:01 smithi146 ceph-mon[25603]: Upgrade: Updating mgr.x 2024-09-20T20:23:01.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:01 smithi146 ceph-mon[25603]: Deploying daemon mgr.x on smithi146 2024-09-20T20:23:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:01 smithi078 ceph-mon[91136]: Upgrade: Updating mgr.x 2024-09-20T20:23:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:01 smithi078 ceph-mon[91136]: Deploying daemon mgr.x on smithi146 2024-09-20T20:23:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:01 smithi078 ceph-mon[30028]: Upgrade: Updating mgr.x 2024-09-20T20:23:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:01 smithi078 ceph-mon[30028]: Deploying daemon mgr.x on smithi146 2024-09-20T20:23:01.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:01 smithi146 systemd[1]: Stopping Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:02 smithi146 ceph-mon[25603]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:02 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:02 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:02 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[91136]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.591 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[30028]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:02.591 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:02.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:02 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:03.016 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:02 smithi146 podman[94324]: 2024-09-20 20:23:02.797311311 +0000 UTC m=+0.130711641 container died b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T20:23:03.411 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 podman[94324]: 2024-09-20 20:23:03.173632048 +0000 UTC m=+0.507032367 container cleanup b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T20:23:03.411 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 bash[94324]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x 2024-09-20T20:23:03.755 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 podman[94336]: 2024-09-20 20:23:03.409429101 +0000 UTC m=+0.613178158 container remove b9f27a3e10d9041f6e74621569b8e5a3023f5536c70b01e31b4d314680a0a539 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-20T20:23:03.755 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T20:23:03.755 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T20:23:03.755 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 systemd[1]: Stopped Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:23:03.755 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mgr.x.service: Consumed 6.002s CPU time. 2024-09-20T20:23:03.755 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 systemd[1]: Starting Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:23:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:03 smithi146 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:04.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 podman[94465]: 2024-09-20 20:23:03.866619834 +0000 UTC m=+0.016438756 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:23:04.017 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:03 smithi146 podman[94465]: 2024-09-20 20:23:03.984681577 +0000 UTC m=+0.134500495 container create 38d4d95ffa4e7525980b31605d509477a5298c0c675b975cef2bdab1ac85289a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:23:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:03 smithi078 ceph-mon[30028]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:03 smithi078 ceph-mon[91136]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:04.479 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:04 smithi146 podman[94465]: 2024-09-20 20:23:04.178814745 +0000 UTC m=+0.328648992 container init 38d4d95ffa4e7525980b31605d509477a5298c0c675b975cef2bdab1ac85289a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T20:23:04.479 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:04 smithi146 podman[94465]: 2024-09-20 20:23:04.1818896 +0000 UTC m=+0.331708525 container start 38d4d95ffa4e7525980b31605d509477a5298c0c675b975cef2bdab1ac85289a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:23:04.479 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:04 smithi146 ceph-mgr[94484]: -- 172.21.15.146:0/3970186350 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583975af0e0 con 0x55839758d400 2024-09-20T20:23:04.479 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:04 smithi146 bash[94465]: 38d4d95ffa4e7525980b31605d509477a5298c0c675b975cef2bdab1ac85289a 2024-09-20T20:23:04.479 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:04 smithi146 systemd[1]: Started Ceph mgr.x for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:23:05.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:05.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:05.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:05 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:06 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:06.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:06.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:06.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:06 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:07 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:07.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:07 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:08 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:08.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.486+0000 7fc81e8f9040 -1 mgr[py] Module not found: 'dashboard' 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.486+0000 7fc81e8f9040 -1 mgr[py] Traceback (most recent call last): 2024-09-20T20:23:09.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: from .module import Module, StandbyModule # noqa: F401 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: File "/usr/share/ceph/mgr/dashboard/module.py", line 55, in 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: from .plugins import PLUGIN_MANAGER, debug, feature_toggles, motd # isort:skip # noqa E501 # pylint: disable=unused-import 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: File "/usr/share/ceph/mgr/dashboard/plugins/feature_toggles.py", line 15, in 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: from ..controllers.rgw import Rgw, RgwBucket, RgwDaemon, RgwUser 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 17, in 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: from ..services.rgw_client import _SYNC_GROUP_ID, NoRgwDaemonsException, RgwClient, RgwMultisite 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 251 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: logger.warning(f"Credentials not found for RGW Daemon '{daemon_name}'. 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: ^ 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: SyntaxError: EOL while scanning string literal 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.487+0000 7fc81e8f9040 -1 mgr[py] Class not found in module 'dashboard' 2024-09-20T20:23:09.520 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.487+0000 7fc81e8f9040 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:09.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:09 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.680+0000 7fc81e8f9040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: [20/Sep/2024:20:23:09] ENGINE Bus STARTING 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: CherryPy Checker: 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: The Application mounted at '' has an empty config. 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: [20/Sep/2024:20:23:09] ENGINE Serving on http://:::9283 2024-09-20T20:23:09.798 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Sep 20 20:23:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-x[94480]: [20/Sep/2024:20:23:09] ENGINE Bus STARTED 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.381 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Standby manager daemon x restarted 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Standby manager daemon x started 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:10 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Standby manager daemon x restarted 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Standby manager daemon x started 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Failed to load ceph-mgr modules: dashboard 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Standby manager daemon x restarted 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Standby manager daemon x started 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:10.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:23:10] "GET /metrics HTTP/1.1" 200 38650 "" "Prometheus/2.43.0" 2024-09-20T20:23:11.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: mgrmap e43: y(active, since 42s), standbys: x 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.581 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.582 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:11 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: mgrmap e43: y(active, since 42s), standbys: x 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: mgrmap e43: y(active, since 42s), standbys: x 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:11.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:11.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:11.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:12 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:12.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:12.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:12.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:12.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:12 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:13 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:13.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:13 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:14 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:14.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:14.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:14.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:14 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.621 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:15 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:15.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:15 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:16 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.798 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:16.799 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:16.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:16.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:16.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:16 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:17 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:17.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:17.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:17.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr.x"} : dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:23:18.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:18 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T20:23:18.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr.x"} : dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:23:18.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr.x"} : dispatch 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T20:23:18.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T20:23:18.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T20:23:19.640 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: Upgrade: It appears safe to stop mon.c 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.893 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.894 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:19 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: Upgrade: It appears safe to stop mon.c 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: Upgrade: It appears safe to stop mon.c 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:19.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:19 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:23:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 13m - - 2024-09-20T20:23:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 13m - - 2024-09-20T20:23:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (8m) 0s ago 9m 26.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:23:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (8m) 1s ago 8m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:23:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (88s) 0s ago 6m 55.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (16s) 1s ago 13m 447M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2m) 0s ago 14m 496M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (29s) 0s ago 14m 43.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (13m) 1s ago 13m 62.8M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (13m) 0s ago 13m 64.2M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 1c44732ec62c 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (8m) 0s ago 8m 14.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (8m) 1s ago 8m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (12m) 0s ago 12m 54.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (12m) 0s ago 12m 62.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (12m) 0s ago 12m 58.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (11m) 0s ago 11m 59.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (11m) 1s ago 11m 66.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (10m) 1s ago 10m 59.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:23:20.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (10m) 1s ago 10m 50.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:23:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (10m) 1s ago 10m 62.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:23:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (100s) 1s ago 9m 40.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:23:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7m) 0s ago 7m 80.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:23:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7m) 1s ago 7m 80.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:23:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7m) 0s ago 7m 80.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:23:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7m) 1s ago 7m 83.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:23:20.532 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:23:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:20.534 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:23:20.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 14, 2024-09-20T20:23:20.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:23:20.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:23:20.534 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:23:20.771 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:23:20] "GET /metrics HTTP/1.1" 200 38648 "" "Prometheus/2.43.0" 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "3/23 daemons upgraded", 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:23:20.819 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:23:20.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='client.25535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:20.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:20 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1341409248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='client.25535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/1341409248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='client.25535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 ' entity='mgr.y' 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T20:23:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1341409248' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T20:23:21.139 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:21.139 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:21.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[30028]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[30028]: Upgrade: Updating mon.c 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[30028]: Deploying daemon mon.c on smithi078 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[30028]: from='client.44115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[30028]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[30028]: from='client.? 172.21.15.78:0/3035972614' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:21.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[91136]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[91136]: Upgrade: Updating mon.c 2024-09-20T20:23:21.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[91136]: Deploying daemon mon.c on smithi078 2024-09-20T20:23:21.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[91136]: from='client.44115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[91136]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3035972614' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:21.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:21 smithi146 ceph-mon[25603]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:21 smithi146 ceph-mon[25603]: Upgrade: Updating mon.c 2024-09-20T20:23:21.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:21 smithi146 ceph-mon[25603]: Deploying daemon mon.c on smithi078 2024-09-20T20:23:21.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:21 smithi146 ceph-mon[25603]: from='client.44115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:21 smithi146 ceph-mon[25603]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:21.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:21 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/3035972614' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:22.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:22 smithi078 ceph-mon[30028]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:22.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:22 smithi078 ceph-mon[91136]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:23.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:22 smithi078 systemd[1]: Stopping Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:23:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:22 smithi146 ceph-mon[25603]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:23.536 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:23:23] ENGINE Bus STOPPING 2024-09-20T20:23:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c[30024]: 2024-09-20T20:23:23.225+0000 7fe73846d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:23:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c[30024]: 2024-09-20T20:23:23.225+0000 7fe73846d640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T20:23:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:23 smithi078 podman[96383]: 2024-09-20 20:23:23.347949568 +0000 UTC m=+0.255948343 container died 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T20:23:23.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:23 smithi078 podman[96383]: 2024-09-20 20:23:23.725315655 +0000 UTC m=+0.633314429 container cleanup 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T20:23:23.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:23 smithi078 bash[96383]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c 2024-09-20T20:23:23.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:23:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T20:23:23.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:23:23] ENGINE Bus STOPPED 2024-09-20T20:23:23.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:23:23] ENGINE Bus STARTING 2024-09-20T20:23:23.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:23:23] ENGINE Serving on http://:::9283 2024-09-20T20:23:23.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: [20/Sep/2024:20:23:23] ENGINE Bus STARTED 2024-09-20T20:23:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:23 smithi078 podman[96394]: 2024-09-20 20:23:23.993773957 +0000 UTC m=+0.642105328 container remove 1c44732ec62c82cffba004a3b7d887b7b1e95e09b9a4a56a4d3fd937790bc5ac (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T20:23:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-20T20:23:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 systemd[1]: Stopped Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:23:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.c.service: Consumed 12.468s CPU time. 2024-09-20T20:23:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 systemd[1]: Starting Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:23:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 podman[96533]: 2024-09-20 20:23:24.435987109 +0000 UTC m=+0.016301308 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:23:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 podman[96533]: 2024-09-20 20:23:24.536905646 +0000 UTC m=+0.117219833 container create 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T20:23:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 podman[96533]: 2024-09-20 20:23:24.88938837 +0000 UTC m=+0.469702562 container init 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD) 2024-09-20T20:23:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 podman[96533]: 2024-09-20 20:23:24.892863527 +0000 UTC m=+0.473177707 container start 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T20:23:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:23:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T20:23:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: pidfile_write: ignore empty --pid-file 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: load: jerasure load: lrc 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Git sha 0 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: DB SUMMARY 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: DB Session ID: 32ALALCAEY2T8MEMXYS0 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: MANIFEST file: MANIFEST-000010 size: 3121 Bytes 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000048.sst 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000046.log size: 736880 ; 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:23:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.env: 0x559383462120 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.info_log: 0x55938526b700 2024-09-20T20:23:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:23:25.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.statistics: (nil) 2024-09-20T20:23:25.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.use_fsync: 0 2024-09-20T20:23:25.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.db_log_dir: 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.wal_dir: 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:23:25.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.write_buffer_manager: 0x55938526e1e0 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:23:25.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.unordered_write: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.row_cache: None 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.wal_filter: None 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.wal_compression: 0 2024-09-20T20:23:25.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:23:25.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_open_files: -1 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:23:25.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Compression algorithms supported: 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kZSTD supported: 0 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.merge_operator: 2024-09-20T20:23:25.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_filter: None 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55938526b320) 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_type: 0 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: data_block_index_type: 0 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_shortening: 1 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:23:25.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: checksum: 4 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: no_block_cache: 0 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache: 0x55938525f610 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_options: 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: capacity : 536870912 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: num_shard_bits : 4 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: strict_capacity_limit : 0 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_compressed: (nil) 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: persistent_cache: (nil) 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_size: 4096 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_size_deviation: 10 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_restart_interval: 16 2024-09-20T20:23:25.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_block_restart_interval: 1 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: metadata_block_size: 4096 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: partition_filters: 0 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: use_delta_encoding: 1 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: filter_policy: bloomfilter 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: whole_key_filtering: 1 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: verify_compression: 0 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: format_version: 5 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: enable_index_compression: 1 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_align: 0 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: prepopulate_block_cache: 0 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:23:25.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression: NoCompression 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.num_levels: 7 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:23:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:23:25.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:23:25.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:23:25.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:23:25.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.ttl: 2592000 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:23:25.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 50, last_sequence is 31432, log_number is 46,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 46 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 46 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 02a0691f-2c78-405d-ad9e-04f6291722b7 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863804919995, "job": 1, "event": "recovery_started", "wal_files": [46]} 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #46 mode 2 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863804922591, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 51, "file_size": 467186, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 31437, "largest_seqno": 31940, "table_properties": {"data_size": 464333, "index_size": 1109, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 7013, "raw_average_key_size": 25, "raw_value_size": 459154, "raw_average_value_size": 1657, "num_data_blocks": 45, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 1, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726863804, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "02a0691f-2c78-405d-ad9e-04f6291722b7", "db_session_id": "32ALALCAEY2T8MEMXYS0", "orig_file_number": 51, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863804922689, "job": 1, "event": "recovery_finished"} 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/version_set.cc:5047] Creating manifest 53 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:23:25.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000046.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55938528ee00 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: DB pointer 0x559385390000 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** DB Stats ** 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:25.056 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: L0 1/0 456.24 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 309.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: L6 1/0 12.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: Sum 2/0 12.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 309.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 309.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 309.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:23:25.057 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative compaction: 0.00 GB write, 86.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval compaction: 0.00 GB write, 86.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Block cache BinnedLRUCache@0x55938525f610#2 capacity: 512.00 MB usage: 1.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.78 KB,0.000149012%) IndexBlock(1,1.20 KB,0.000229478%) Misc(1,0.00 KB,0%) 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:23:25.058 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: starting mon.c rank 2 at public addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] at bind addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???) e3 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).mds e1 new map 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).mds e1 print_map 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: e1 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: legacy client fscid: -1 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: No filesystems configured 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:23:25.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 ceph-mon[96551]: mon.c@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-09-20T20:23:25.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 bash[96533]: 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c 2024-09-20T20:23:25.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:24 smithi078 systemd[1]: Started Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:23:30.970 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:23:30] "GET /metrics HTTP/1.1" 200 38646 "" "Prometheus/2.43.0" 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: mon.c calling monitor election 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: mon.b calling monitor election 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: mon.a calling monitor election 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: monmap epoch 3 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:32.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: min_mon_release 18 (reef) 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: election_strategy: 1 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: fsmap 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: mgrmap e43: y(active, since 63s), standbys: x 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: mon.c calling monitor election 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: mon.b calling monitor election 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: mon.a calling monitor election 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: monmap epoch 3 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:23:32.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: min_mon_release 18 (reef) 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: election_strategy: 1 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: fsmap 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: mgrmap e43: y(active, since 63s), standbys: x 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:32 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: mon.c calling monitor election 2024-09-20T20:23:32.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: mon.b calling monitor election 2024-09-20T20:23:32.345 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: mon.a calling monitor election 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: monmap epoch 3 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: min_mon_release 18 (reef) 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: election_strategy: 1 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: fsmap 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: mgrmap e43: y(active, since 63s), standbys: x 2024-09-20T20:23:32.346 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:32.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:32.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:32 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:33.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:33.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:33.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:33.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:33 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.414 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:34.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:34 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:23:34.935+0000 7fcfc9d8b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-20T20:23:35.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:35.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:35 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:36 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:36.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:36 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:37.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:37 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:38 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:38.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:38.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:38.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:38.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:38 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.228 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:39 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:39.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:39.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:39.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:39.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:39 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: Upgrade: It appears safe to stop mon.c 2024-09-20T20:23:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:40 smithi146 ceph-mon[25603]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:40.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:40.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: Upgrade: It appears safe to stop mon.c 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[96551]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:40.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: Upgrade: It appears safe to stop mon.c 2024-09-20T20:23:40.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-mon[91136]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:23:40] "GET /metrics HTTP/1.1" 200 38646 "" "Prometheus/2.43.0" 2024-09-20T20:23:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:41 smithi146 ceph-mon[25603]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:41 smithi078 ceph-mon[96551]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:41 smithi078 ceph-mon[91136]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:43 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:43 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:23:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:43 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:23:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:43 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:43.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:23:43.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:43 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:44.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:44 smithi078 ceph-mon[91136]: Upgrade: Updating mon.c 2024-09-20T20:23:44.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:44 smithi078 ceph-mon[91136]: Deploying daemon mon.c on smithi078 2024-09-20T20:23:44.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:44 smithi078 ceph-mon[91136]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:44.479 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:44 smithi078 ceph-mon[96551]: Upgrade: Updating mon.c 2024-09-20T20:23:44.479 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:44 smithi078 ceph-mon[96551]: Deploying daemon mon.c on smithi078 2024-09-20T20:23:44.479 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:44 smithi078 ceph-mon[96551]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:44 smithi146 ceph-mon[25603]: Upgrade: Updating mon.c 2024-09-20T20:23:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:44 smithi146 ceph-mon[25603]: Deploying daemon mon.c on smithi078 2024-09-20T20:23:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:44 smithi146 ceph-mon[25603]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:46.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[96551]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:46.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:46 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:46 smithi078 systemd[1]: Stopping Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:23:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:47 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c[96547]: 2024-09-20T20:23:47.201+0000 7f4488e7d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:23:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:47 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c[96547]: 2024-09-20T20:23:47.201+0000 7f4488e7d640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T20:23:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:47 smithi078 podman[99797]: 2024-09-20 20:23:47.212561669 +0000 UTC m=+0.123556766 container died 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T20:23:47.830 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:47 smithi078 podman[99797]: 2024-09-20 20:23:47.640736449 +0000 UTC m=+0.551731540 container cleanup 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, ceph=True) 2024-09-20T20:23:47.830 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:47 smithi078 bash[99797]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c 2024-09-20T20:23:48.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:47 smithi078 podman[99809]: 2024-09-20 20:23:47.828461357 +0000 UTC m=+0.610608860 container remove 7fbd877786a35b82ea0e9ee64a365c08c2cac1e6946bec931dd88c4305a9985c (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T20:23:48.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-20T20:23:48.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 systemd[1]: Stopped Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:23:48.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.c.service: Consumed 1.277s CPU time. 2024-09-20T20:23:48.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 systemd[1]: Starting Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:23:48.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 podman[99941]: 2024-09-20 20:23:48.384466223 +0000 UTC m=+0.015672271 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:23:48.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 podman[99941]: 2024-09-20 20:23:48.487998709 +0000 UTC m=+0.119204740 container create ffd6b0f668a7924ab4104f5808f9e4e91d63b747b095bfbbe45083504590421b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T20:23:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 podman[99941]: 2024-09-20 20:23:48.757277262 +0000 UTC m=+0.388483303 container init ffd6b0f668a7924ab4104f5808f9e4e91d63b747b095bfbbe45083504590421b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:23:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 podman[99941]: 2024-09-20 20:23:48.760982821 +0000 UTC m=+0.392188852 container start ffd6b0f668a7924ab4104f5808f9e4e91d63b747b095bfbbe45083504590421b (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:23:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:23:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T20:23:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: pidfile_write: ignore empty --pid-file 2024-09-20T20:23:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: load: jerasure load: lrc 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Git sha 0 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: DB SUMMARY 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: DB Session ID: VBWTV7S7N9FL6IJWH169 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: MANIFEST file: MANIFEST-000053 size: 553 Bytes 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000058.sst 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000056.log size: 1079850 ; 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:23:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.env: 0x563aaf3fd120 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.info_log: 0x563ab19c9700 2024-09-20T20:23:49.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:23:49.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.statistics: (nil) 2024-09-20T20:23:49.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.use_fsync: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:23:49.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.db_log_dir: 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.wal_dir: 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.write_buffer_manager: 0x563ab19cc1e0 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:23:49.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.unordered_write: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.row_cache: None 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.wal_filter: None 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:23:49.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.wal_compression: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:23:49.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_open_files: -1 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:23:49.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Compression algorithms supported: 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kZSTD supported: 0 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:23:49.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000053 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.merge_operator: 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_filter: None 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563ab19c9320) 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:23:49.046 INFO:journalctl@ceph.mon.c.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_type: 0 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: data_block_index_type: 0 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_shortening: 1 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: checksum: 4 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: no_block_cache: 0 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache: 0x563ab19bd610 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_options: 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: capacity : 536870912 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: num_shard_bits : 4 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: strict_capacity_limit : 0 2024-09-20T20:23:49.047 INFO:journalctl@ceph.mon.c.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_cache_compressed: (nil) 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: persistent_cache: (nil) 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_size: 4096 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_size_deviation: 10 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_restart_interval: 16 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: index_block_restart_interval: 1 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: metadata_block_size: 4096 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: partition_filters: 0 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: use_delta_encoding: 1 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: filter_policy: bloomfilter 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: whole_key_filtering: 1 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: verify_compression: 0 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:23:49.048 INFO:journalctl@ceph.mon.c.smithi078.stdout: format_version: 5 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout: enable_index_compression: 1 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout: block_align: 0 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout: prepopulate_block_cache: 0 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression: NoCompression 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.num_levels: 7 2024-09-20T20:23:49.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:23:49.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:23:49.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:23:49.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:23:49.053 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.ttl: 2592000 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:23:49.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000053 succeeded,manifest_file_number is 53, next_file_number is 60, last_sequence is 33173, log_number is 56,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 56 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 56 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 02a0691f-2c78-405d-ad9e-04f6291722b7 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863828787388, "job": 1, "event": "recovery_started", "wal_files": [56]} 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #56 mode 2 2024-09-20T20:23:49.055 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863828790983, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 61, "file_size": 671639, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 33178, "largest_seqno": 33937, "table_properties": {"data_size": 668085, "index_size": 1490, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1029, "raw_key_size": 9974, "raw_average_key_size": 24, "raw_value_size": 660900, "raw_average_value_size": 1623, "num_data_blocks": 61, "num_entries": 407, "num_filter_entries": 407, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726863828, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "02a0691f-2c78-405d-ad9e-04f6291722b7", "db_session_id": "VBWTV7S7N9FL6IJWH169", "orig_file_number": 61, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:23:49.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863828791085, "job": 1, "event": "recovery_finished"} 2024-09-20T20:23:49.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/version_set.cc:5047] Creating manifest 63 2024-09-20T20:23:49.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:23:49.057 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000056.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:23:49.057 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563ab19ece00 2024-09-20T20:23:49.057 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: DB pointer 0x563ab1aee000 2024-09-20T20:23:49.057 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:23:49.057 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** DB Stats ** 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:23:49.058 INFO:journalctl@ceph.mon.c.smithi078.stdout: L0 1/0 655.90 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 344.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: L6 1/0 11.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: Sum 2/0 12.35 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 344.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 344.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 344.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:23:49.059 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: Cumulative compaction: 0.00 GB write, 109.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: Interval compaction: 0.00 GB write, 109.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: Block cache BinnedLRUCache@0x563ab19bd610#2 capacity: 512.00 MB usage: 2.73 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: starting mon.c rank 2 at public addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] at bind addrs [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???) e3 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).mds e1 new map 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).mds e1 print_map 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: e1 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:23:49.060 INFO:journalctl@ceph.mon.c.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout: legacy client fscid: -1 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout: 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout: No filesystems configured 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 ceph-mon[99959]: mon.c@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 bash[99941]: ffd6b0f668a7924ab4104f5808f9e4e91d63b747b095bfbbe45083504590421b 2024-09-20T20:23:49.061 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:48 smithi078 systemd[1]: Started Ceph mon.c for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:23:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:23:50] "GET /metrics HTTP/1.1" 200 38644 "" "Prometheus/2.43.0" 2024-09-20T20:23:54.458 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:23:55.027 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 14m - - 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 14m - - 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (8m) 9s ago 9m 26.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (9m) 16s ago 9m 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2m) 9s ago 7m 55.7M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (51s) 16s ago 13m 447M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2m) 9s ago 15m 500M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (63s) 9s ago 15m 48.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (14m) 16s ago 14m 62.3M 2048M 18.2.4-869-g0b2cc4d4 4404209dfb71 52abb78f8d40 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 starting - - - 2048M 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (9m) 9s ago 9m 14.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (9m) 16s ago 9m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:23:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (13m) 9s ago 13m 54.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (13m) 9s ago 13m 62.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (12m) 9s ago 12m 58.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (12m) 9s ago 12m 59.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (11m) 16s ago 11m 66.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (11m) 16s ago 11m 59.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (11m) 16s ago 11m 50.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (10m) 16s ago 10m 62.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2m) 16s ago 10m 45.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7m) 9s ago 7m 80.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7m) 16s ago 7m 80.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7m) 9s ago 7m 81.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:23:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7m) 16s ago 7m 83.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:55.418 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 13, 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:23:55.419 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:23:55.728 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:23:55.728 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "3/23 daemons upgraded", 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:23:55.729 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:23:56.099 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.100 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.100 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: mon.c calling monitor election 2024-09-20T20:23:56.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: mon.b calling monitor election 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: mon.a calling monitor election 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: monmap epoch 3 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: min_mon_release 18 (reef) 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: election_strategy: 1 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: fsmap 2024-09-20T20:23:56.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: mgrmap e43: y(active, since 87s), standbys: x 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: mon.c calling monitor election 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: mon.b calling monitor election 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: mon.a calling monitor election 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:56.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: monmap epoch 3 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: min_mon_release 18 (reef) 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: election_strategy: 1 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: fsmap 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: mgrmap e43: y(active, since 87s), standbys: x 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1451577780' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1451577780' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: mon.c calling monitor election 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: mon.b calling monitor election 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: mon.a calling monitor election 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: monmap epoch 3 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: last_changed 2024-09-20T20:09:54.489834+0000 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: min_mon_release 18 (reef) 2024-09-20T20:23:56.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: election_strategy: 1 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: fsmap 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: mgrmap e43: y(active, since 87s), standbys: x 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/1451577780' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.281 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:56.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:56.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:56.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:56.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:55 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2877831634' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2877831634' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='client.? 172.21.15.78:0/2877831634' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:57.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:56 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.063 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.063 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.064 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.068 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.070 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:58.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:57 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:58 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:23:59.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:00.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:23:59 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:00.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:24:00] "GET /metrics HTTP/1.1" 200 38645 "" "Prometheus/2.43.0" 2024-09-20T20:24:01.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:01.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:01.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:00 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:02.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:02.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:01 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:03.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:03.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:03.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:03.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:03.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:02 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:04.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:04.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:04.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:04.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:03 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:05.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:05.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:05.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:04 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:05 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:06.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:07.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[99959]: Upgrade: It appears safe to stop mon.b 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:07.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T20:24:07.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-20T20:24:07.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:07 smithi078 ceph-mon[91136]: Upgrade: It appears safe to stop mon.b 2024-09-20T20:24:07.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:07.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:07.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:07.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-20T20:24:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-20T20:24:07.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:07 smithi146 ceph-mon[25603]: Upgrade: It appears safe to stop mon.b 2024-09-20T20:24:08.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: Upgrade: Updating mon.b 2024-09-20T20:24:08.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:08.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:08.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:08.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:08.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: Deploying daemon mon.b on smithi146 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[91136]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: Upgrade: Updating mon.b 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: Deploying daemon mon.b on smithi146 2024-09-20T20:24:08.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:08 smithi078 ceph-mon[99959]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: Upgrade: Updating mon.b 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: Deploying daemon mon.b on smithi146 2024-09-20T20:24:08.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 ceph-mon[25603]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:09.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:08 smithi146 systemd[1]: Stopping Ceph mon.b for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:24:09.375 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b[25586]: 2024-09-20T20:24:09.063+0000 7fb013ff8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:24:09.375 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b[25586]: 2024-09-20T20:24:09.063+0000 7fb013ff8640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T20:24:09.375 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 podman[103118]: 2024-09-20 20:24:09.126363884 +0000 UTC m=+0.167606346 container died 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-20T20:24:09.376 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 podman[103118]: 2024-09-20 20:24:09.373949583 +0000 UTC m=+0.415192044 container exec_died 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, ceph=True, io.buildah.version=1.37.2) 2024-09-20T20:24:09.968 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 podman[103118]: 2024-09-20 20:24:09.665296351 +0000 UTC m=+0.706538815 container cleanup 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T20:24:09.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 bash[103118]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b 2024-09-20T20:24:10.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:09 smithi146 podman[103131]: 2024-09-20 20:24:09.966966646 +0000 UTC m=+0.835054579 container remove 52abb78f8d405e73fb9e05b3ec4b5bcc52ad907264e9d9507af9fec43dcf5949 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-20T20:24:10.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-20T20:24:10.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 systemd[1]: Stopped Ceph mon.b for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:24:10.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@mon.b.service: Consumed 15.687s CPU time. 2024-09-20T20:24:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 systemd[1]: Starting Ceph mon.b for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:24:10.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 podman[103259]: 2024-09-20 20:24:10.551135094 +0000 UTC m=+0.104053927 container create 9edca87d335c7efd38722bc4b7007bbb4a745692f23127515462947fa4cdc32d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-20T20:24:10.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 podman[103259]: 2024-09-20 20:24:10.46341923 +0000 UTC m=+0.016338077 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:24:10.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 podman[103259]: 2024-09-20 20:24:10.745218649 +0000 UTC m=+0.298137488 container init 9edca87d335c7efd38722bc4b7007bbb4a745692f23127515462947fa4cdc32d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 podman[103259]: 2024-09-20 20:24:10.748293147 +0000 UTC m=+0.301211995 container start 9edca87d335c7efd38722bc4b7007bbb4a745692f23127515462947fa4cdc32d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True) 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev), process ceph-mon, pid 2 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: pidfile_write: ignore empty --pid-file 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: load: jerasure load: lrc 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: RocksDB version: 7.9.2 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Git sha 0 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Compile date 2024-09-20 13:31:19 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: DB SUMMARY 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: DB Session ID: OKNDW7PMS0XL0V2OVFS0 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: CURRENT file: CURRENT 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: MANIFEST file: MANIFEST-000010 size: 3607 Bytes 2024-09-20T20:24:10.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-09-20T20:24:10.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 212676 ; 2024-09-20T20:24:10.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.error_if_exists: 0 2024-09-20T20:24:10.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.create_if_missing: 0 2024-09-20T20:24:10.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.paranoid_checks: 1 2024-09-20T20:24:10.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.env: 0x55779c487120 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.info_log: 0x55779d07f700 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.statistics: (nil) 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.use_fsync: 0 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_log_file_size: 0 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T20:24:10.825 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.allow_fallocate: 1 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.use_direct_reads: 0 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.db_log_dir: 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.wal_dir: 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T20:24:10.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.write_buffer_manager: 0x55779d0821e0 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T20:24:10.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.unordered_write: 0 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.row_cache: None 2024-09-20T20:24:10.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.wal_filter: None 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.two_write_queues: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.wal_compression: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.atomic_flush: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T20:24:10.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T20:24:10.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.log_readahead_size: 0 2024-09-20T20:24:10.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T20:24:10.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T20:24:10.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T20:24:10.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T20:24:10.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_background_jobs: 2 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_background_compactions: -1 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_subcompactions: 1 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T20:24:10.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_open_files: -1 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T20:24:10.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_background_flushes: -1 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Compression algorithms supported: 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kZSTD supported: 0 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kXpressCompression supported: 0 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kBZip2Compression supported: 0 2024-09-20T20:24:10.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kLZ4Compression supported: 1 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kZlibCompression supported: 1 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: kSnappyCompression supported: 1 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.merge_operator: 2024-09-20T20:24:10.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_filter: None 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55779d07f320) 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-09-20T20:24:10.835 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_type: 0 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_index_type: 0 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_shortening: 1 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: checksum: 4 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: no_block_cache: 0 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache: 0x55779d073610 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_options: 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: capacity : 536870912 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_shard_bits : 4 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: strict_capacity_limit : 0 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_compressed: (nil) 2024-09-20T20:24:10.836 INFO:journalctl@ceph.mon.b.smithi146.stdout: persistent_cache: (nil) 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size: 4096 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size_deviation: 10 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_restart_interval: 16 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_block_restart_interval: 1 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: metadata_block_size: 4096 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: partition_filters: 0 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_delta_encoding: 1 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: filter_policy: bloomfilter 2024-09-20T20:24:10.837 INFO:journalctl@ceph.mon.b.smithi146.stdout: whole_key_filtering: 1 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: verify_compression: 0 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: format_version: 5 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_index_compression: 1 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_align: 0 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: max_auto_readahead_size: 262144 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: prepopulate_block_cache: 0 2024-09-20T20:24:10.838 INFO:journalctl@ceph.mon.b.smithi146.stdout: initial_auto_readahead_size: 8192 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression: NoCompression 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.num_levels: 7 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T20:24:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T20:24:10.840 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T20:24:10.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T20:24:10.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T20:24:10.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T20:24:10.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T20:24:10.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T20:24:10.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T20:24:10.842 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T20:24:10.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T20:24:10.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.inplace_update_support: 0 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T20:24:10.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.bloom_locality: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.max_successive_merges: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.ttl: 2592000 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T20:24:10.846 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.enable_blob_files: false 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.min_blob_size: 0 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T20:24:10.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 56, last_sequence is 36516, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 52 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b6608849-9ec2-4625-b45b-41932e88bcb9 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863850775037, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-09-20T20:24:10.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863850776270, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 142005, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 36521, "largest_seqno": 36597, "table_properties": {"data_size": 140534, "index_size": 305, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 133, "raw_key_size": 1237, "raw_average_key_size": 26, "raw_value_size": 139360, "raw_average_value_size": 3029, "num_data_blocks": 13, "num_entries": 46, "num_filter_entries": 46, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726863850, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b6608849-9ec2-4625-b45b-41932e88bcb9", "db_session_id": "OKNDW7PMS0XL0V2OVFS0", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726863850776366, "job": 1, "event": "recovery_finished"} 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55779d0a2e00 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: DB pointer 0x55779d1a4000 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** DB Stats ** 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:24:10.849 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 138.68 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: L6 1/0 13.19 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 2/0 13.33 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:24:10.850 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T20:24:10.851 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 23.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 23.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache BinnedLRUCache@0x55779d073610#2 capacity: 512.00 MB usage: 0.56 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.17 KB,3.27826e-05%) IndexBlock(1,0.39 KB,7.45058e-05%) Misc(1,0.00 KB,0%) 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: starting mon.b rank 1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???) e3 preinit fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).mds e1 new map 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).mds e1 print_map 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: e1 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: legacy client fscid: -1 2024-09-20T20:24:10.852 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout: No filesystems configured 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 ceph-mon[103278]: mon.b@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 bash[103259]: 9edca87d335c7efd38722bc4b7007bbb4a745692f23127515462947fa4cdc32d 2024-09-20T20:24:10.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:10 smithi146 systemd[1]: Started Ceph mon.b for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:24:10.920 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:24:10] "GET /metrics HTTP/1.1" 200 38645 "" "Prometheus/2.43.0" 2024-09-20T20:24:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[91136]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:11 smithi078 ceph-mon[99959]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:11 smithi146 ceph-mon[103278]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:16.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:24:15.790+0000 7fcfc9d8b640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-20T20:24:17.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: mon.c calling monitor election 2024-09-20T20:24:17.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: mon.b calling monitor election 2024-09-20T20:24:17.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: last_changed 2024-09-20T20:24:11.283695+0000 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: min_mon_release 19 (squid) 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: election_strategy: 1 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: fsmap 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: mgrmap e43: y(active, since 108s), standbys: x 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: mon.a calling monitor election 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: monmap epoch 4 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: last_changed 2024-09-20T20:24:11.283695+0000 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: min_mon_release 19 (squid) 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: election_strategy: 1 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: fsmap 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:24:17.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: mgrmap e43: y(active, since 108s), standbys: x 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:17.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Reconfiguring mon.a (monmap changed)... 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[91136]: Reconfiguring daemon mon.a on smithi078 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: mon.c calling monitor election 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: mon.b calling monitor election 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: last_changed 2024-09-20T20:24:11.283695+0000 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:24:17.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: min_mon_release 19 (squid) 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: election_strategy: 1 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: fsmap 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: mgrmap e43: y(active, since 108s), standbys: x 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: mon.a calling monitor election 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:24:17.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: monmap epoch 4 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: last_changed 2024-09-20T20:24:11.283695+0000 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: min_mon_release 19 (squid) 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: election_strategy: 1 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: fsmap 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: mgrmap e43: y(active, since 108s), standbys: x 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Reconfiguring mon.a (monmap changed)... 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:17.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:17.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:17 smithi078 ceph-mon[99959]: Reconfiguring daemon mon.a on smithi078 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: mon.c calling monitor election 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: mon.b calling monitor election 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: last_changed 2024-09-20T20:24:11.283695+0000 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: min_mon_release 19 (squid) 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: election_strategy: 1 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: fsmap 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: mgrmap e43: y(active, since 108s), standbys: x 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: mon.a calling monitor election 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: monmap epoch 4 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: fsid f2df6e16-778b-11ef-baf5-efdc52797490 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: last_changed 2024-09-20T20:24:11.283695+0000 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: created 2024-09-20T20:08:17.222812+0000 2024-09-20T20:24:17.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: min_mon_release 19 (squid) 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: election_strategy: 1 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: 1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: 2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: fsmap 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: mgrmap e43: y(active, since 108s), standbys: x 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Health detail: HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Reconfiguring mon.a (monmap changed)... 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:17.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:17 smithi146 ceph-mon[103278]: Reconfiguring daemon mon.a on smithi078 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:18 smithi146 ceph-mon[103278]: Reconfiguring daemon mgr.y on smithi078 2024-09-20T20:24:19.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:19.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:19.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:19.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T20:24:19.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[99959]: Reconfiguring daemon mgr.y on smithi078 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:19.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:18 smithi078 ceph-mon[91136]: Reconfiguring daemon mgr.y on smithi078 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: Reconfiguring mon.c (monmap changed)... 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: Reconfiguring daemon mon.c on smithi078 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:20.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: Reconfiguring mon.c (monmap changed)... 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: Reconfiguring daemon mon.c on smithi078 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: Reconfiguring mon.c (monmap changed)... 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: Reconfiguring daemon mon.c on smithi078 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:21.022 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:24:20] "GET /metrics HTTP/1.1" 200 38643 "" "Prometheus/2.43.0" 2024-09-20T20:24:21.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:21.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:21.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:21 smithi146 ceph-mon[103278]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T20:24:21.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:21.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:21.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:21 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.0 on smithi078 2024-09-20T20:24:21.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[99959]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.0 on smithi078 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[91136]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:22.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:21 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.0 on smithi078 2024-09-20T20:24:22.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:22 smithi146 ceph-mon[103278]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:22.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:22.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:22.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:24:22.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[99959]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[91136]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:24:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:23 smithi146 ceph-mon[103278]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T20:24:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:23 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.1 on smithi078 2024-09-20T20:24:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T20:24:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:23.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[99959]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T20:24:23.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.1 on smithi078 2024-09-20T20:24:23.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T20:24:23.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:23.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[91136]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T20:24:23.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.1 on smithi078 2024-09-20T20:24:23.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:23.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T20:24:23.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:24.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:24 smithi078 ceph-mon[99959]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T20:24:24.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:24 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.2 on smithi078 2024-09-20T20:24:24.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:24 smithi078 ceph-mon[99959]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:24.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:24 smithi078 ceph-mon[91136]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T20:24:24.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:24 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.2 on smithi078 2024-09-20T20:24:24.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:24 smithi078 ceph-mon[91136]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:24 smithi146 ceph-mon[103278]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T20:24:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:24 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.2 on smithi078 2024-09-20T20:24:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:24 smithi146 ceph-mon[103278]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:26.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T20:24:25.847438745Z level=info msg="Completed cleanup jobs" duration=11.969063ms 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.3 on smithi078 2024-09-20T20:24:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:26 smithi146 ceph-mon[103278]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:26.386 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:24:26.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.3 on smithi078 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[99959]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:26.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.3 on smithi078 2024-09-20T20:24:26.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:26 smithi078 ceph-mon[91136]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 15m - - 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 14m - - 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (9m) 16s ago 10m 26.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (10m) 6s ago 10m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2m) 16s ago 8m 57.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (83s) 6s ago 14m 448M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3m) 16s ago 15m 502M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (95s) 16s ago 16m 52.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (16s) 6s ago 14m 18.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:24:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (38s) 16s ago 14m 36.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (9m) 16s ago 9m 14.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (9m) 6s ago 9m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (13m) 16s ago 13m 54.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (13m) 16s ago 13m 62.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (13m) 16s ago 13m 59.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (12m) 16s ago 12m 60.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (12m) 6s ago 12m 66.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (11m) 6s ago 11m 59.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (11m) 6s ago 11m 51.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (11m) 6s ago 11m 63.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2m) 6s ago 10m 51.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (8m) 16s ago 8m 80.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:24:26.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (8m) 6s ago 8m 80.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:24:26.981 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (8m) 16s ago 8m 81.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:24:26.981 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (8m) 6s ago 8m 83.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:24:27.298 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:24:27.298 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:24:27.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 12, 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 5 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:24:27.299 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:24:27.574 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:24:27.574 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:24:27.574 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:24:27.574 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:24:27.574 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "5/23 daemons upgraded", 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:24:27.575 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='client.64103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: Reconfiguring rgw.foo.smithi078.fewvbr (monmap changed)... 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: Reconfiguring daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:24:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='client.64109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3202459857' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='client.64103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: Reconfiguring rgw.foo.smithi078.fewvbr (monmap changed)... 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: Reconfiguring daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='client.64109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3202459857' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:27.886 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:27.886 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:27.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='client.64103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: Reconfiguring rgw.foo.smithi078.fewvbr (monmap changed)... 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: Reconfiguring daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='client.64109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3202459857' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4006609309' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4006609309' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4006609309' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: Reconfiguring rgw.smpl.smithi078.qphqws (monmap changed)... 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: Reconfiguring daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:29.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: Reconfiguring rgw.smpl.smithi078.qphqws (monmap changed)... 2024-09-20T20:24:29.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: Reconfiguring daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:24:29.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:29.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:29.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: Reconfiguring rgw.smpl.smithi078.qphqws (monmap changed)... 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: Reconfiguring daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-20T20:24:29.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[99959]: Reconfiguring mon.b (monmap changed)... 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[99959]: Reconfiguring daemon mon.b on smithi146 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[99959]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[91136]: Reconfiguring mon.b (monmap changed)... 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[91136]: Reconfiguring daemon mon.b on smithi146 2024-09-20T20:24:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-mon[91136]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:30.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:30.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:30 smithi146 ceph-mon[103278]: Reconfiguring mon.b (monmap changed)... 2024-09-20T20:24:30.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:30 smithi146 ceph-mon[103278]: Reconfiguring daemon mon.b on smithi146 2024-09-20T20:24:30.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:30 smithi146 ceph-mon[103278]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:30.880 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:24:30] "GET /metrics HTTP/1.1" 200 38639 "" "Prometheus/2.43.0" 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: Reconfiguring daemon mgr.x on smithi146 2024-09-20T20:24:32.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[99959]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: Reconfiguring daemon mgr.x on smithi146 2024-09-20T20:24:32.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:31 smithi078 ceph-mon[91136]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: Reconfiguring daemon mgr.x on smithi146 2024-09-20T20:24:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:31 smithi146 ceph-mon[103278]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:33.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:33.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:33.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:33 smithi146 ceph-mon[103278]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T20:24:33.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:24:33.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:33.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:33 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.4 on smithi146 2024-09-20T20:24:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[91136]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T20:24:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:24:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.4 on smithi146 2024-09-20T20:24:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[99959]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T20:24:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:24:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:33 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.4 on smithi146 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.5 on smithi146 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.5 on smithi146 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.5 on smithi146 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:35.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T20:24:36.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:24:36.370 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:36.370 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.6 on smithi146 2024-09-20T20:24:36.370 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:36 smithi146 ceph-mon[103278]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.6 on smithi146 2024-09-20T20:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[91136]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.6 on smithi146 2024-09-20T20:24:36.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:36 smithi078 ceph-mon[99959]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:37 smithi146 ceph-mon[103278]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T20:24:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:24:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:37 smithi146 ceph-mon[103278]: Reconfiguring daemon osd.7 on smithi146 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[91136]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[91136]: Reconfiguring daemon osd.7 on smithi146 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[99959]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T20:24:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:24:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:37 smithi078 ceph-mon[99959]: Reconfiguring daemon osd.7 on smithi146 2024-09-20T20:24:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:38 smithi146 ceph-mon[103278]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[99959]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[91136]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:38.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:38.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:38.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:38.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:39 smithi146 ceph-mon[103278]: Reconfiguring rgw.foo.smithi146.dulerg (monmap changed)... 2024-09-20T20:24:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:39 smithi146 ceph-mon[103278]: Reconfiguring daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:24:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[99959]: Reconfiguring rgw.foo.smithi146.dulerg (monmap changed)... 2024-09-20T20:24:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[99959]: Reconfiguring daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:24:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:39.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[91136]: Reconfiguring rgw.foo.smithi146.dulerg (monmap changed)... 2024-09-20T20:24:39.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[91136]: Reconfiguring daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:24:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:24:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:24:40] "GET /metrics HTTP/1.1" 200 38639 "" "Prometheus/2.43.0" 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: Reconfiguring rgw.smpl.smithi146.gfhcht (monmap changed)... 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: Reconfiguring daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: Reconfiguring rgw.smpl.smithi146.gfhcht (monmap changed)... 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: Reconfiguring daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: Reconfiguring rgw.smpl.smithi146.gfhcht (monmap changed)... 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: Reconfiguring daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:42.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T20:24:42.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T20:24:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: Upgrade: Setting container_image for all mon 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: Upgrade: Setting container_image for all crash 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: Upgrade: osd.0 is safe to restart 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:42.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: Upgrade: Setting container_image for all mon 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: Upgrade: Setting container_image for all crash 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: Upgrade: osd.0 is safe to restart 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:42.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: Upgrade: Setting container_image for all mon 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: Upgrade: Setting container_image for all crash 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: Upgrade: osd.0 is safe to restart 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:43.987 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:43 smithi078 ceph-mon[91136]: Upgrade: Updating osd.0 2024-09-20T20:24:43.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:43 smithi078 ceph-mon[91136]: Deploying daemon osd.0 on smithi078 2024-09-20T20:24:43.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:43.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:43 smithi078 ceph-mon[99959]: Upgrade: Updating osd.0 2024-09-20T20:24:43.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:43 smithi078 ceph-mon[99959]: Deploying daemon osd.0 on smithi078 2024-09-20T20:24:43.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:44.209 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:43 smithi146 ceph-mon[103278]: Upgrade: Updating osd.0 2024-09-20T20:24:44.209 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:43 smithi146 ceph-mon[103278]: Deploying daemon osd.0 on smithi078 2024-09-20T20:24:44.209 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:45.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:44 smithi078 ceph-mon[99959]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:45.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:44 smithi078 ceph-mon[91136]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:44 smithi146 ceph-mon[103278]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:24:45.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:45 smithi078 systemd[1]: Stopping Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:24:45.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[34319]: 2024-09-20T20:24:45.441+0000 7f4450ce6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:24:45.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[34319]: 2024-09-20T20:24:45.441+0000 7f4450ce6640 -1 osd.0 105 *** Got signal Terminated *** 2024-09-20T20:24:45.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[34319]: 2024-09-20T20:24:45.441+0000 7f4450ce6640 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:24:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:45 smithi146 ceph-mon[103278]: osd.0 marked itself down and dead 2024-09-20T20:24:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:45 smithi146 ceph-mon[103278]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-mon[91136]: osd.0 marked itself down and dead 2024-09-20T20:24:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-mon[91136]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-mon[99959]: osd.0 marked itself down and dead 2024-09-20T20:24:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:45 smithi078 ceph-mon[99959]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:24:46.652 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:46 smithi078 podman[105216]: 2024-09-20 20:24:46.360805071 +0000 UTC m=+1.020612460 container died 53a87889b1c530a73b346494547720cf165a2e4dd72bef812efc6d3504ebc6ce (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T20:24:47.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:46 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:24:47.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:46 smithi078 ceph-mon[91136]: osdmap e106: 8 total, 7 up, 8 in 2024-09-20T20:24:47.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:46 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:24:47.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:46 smithi078 ceph-mon[99959]: osdmap e106: 8 total, 7 up, 8 in 2024-09-20T20:24:47.020 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:46 smithi078 podman[105216]: 2024-09-20 20:24:46.749917637 +0000 UTC m=+1.409725028 container cleanup 53a87889b1c530a73b346494547720cf165a2e4dd72bef812efc6d3504ebc6ce (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:24:47.020 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:46 smithi078 bash[105216]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0 2024-09-20T20:24:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:46 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:24:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:46 smithi146 ceph-mon[103278]: osdmap e106: 8 total, 7 up, 8 in 2024-09-20T20:24:47.285 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105231]: 2024-09-20 20:24:47.018009679 +0000 UTC m=+0.654440400 container remove 53a87889b1c530a73b346494547720cf165a2e4dd72bef812efc6d3504ebc6ce (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-0b2cc4d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:24:47.636 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105321]: 2024-09-20 20:24:47.284519247 +0000 UTC m=+0.016278688 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:24:47.636 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105321]: 2024-09-20 20:24:47.402589497 +0000 UTC m=+0.134348935 container create 9bd5cae00a5430441318cafb32db2487516b893178b7f96ab9b2138fd164d9f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-20T20:24:47.636 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105321]: 2024-09-20 20:24:47.630434377 +0000 UTC m=+0.362193818 container init 9bd5cae00a5430441318cafb32db2487516b893178b7f96ab9b2138fd164d9f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T20:24:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:47 smithi078 ceph-mon[99959]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T20:24:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:47 smithi078 ceph-mon[99959]: pgmap v75: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:24:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:47 smithi078 ceph-mon[91136]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T20:24:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:47 smithi078 ceph-mon[91136]: pgmap v75: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:24:48.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105321]: 2024-09-20 20:24:47.634880578 +0000 UTC m=+0.366640020 container start 9bd5cae00a5430441318cafb32db2487516b893178b7f96ab9b2138fd164d9f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T20:24:48.037 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105321]: 2024-09-20 20:24:47.739031536 +0000 UTC m=+0.470790977 container attach 9bd5cae00a5430441318cafb32db2487516b893178b7f96ab9b2138fd164d9f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2) 2024-09-20T20:24:48.037 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:47 smithi078 podman[105321]: 2024-09-20 20:24:47.827929024 +0000 UTC m=+0.559688474 container died 9bd5cae00a5430441318cafb32db2487516b893178b7f96ab9b2138fd164d9f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-09-20T20:24:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:47 smithi146 ceph-mon[103278]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T20:24:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:47 smithi146 ceph-mon[103278]: pgmap v75: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:24:48.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 podman[105344]: 2024-09-20 20:24:48.401368269 +0000 UTC m=+0.562881568 container remove 9bd5cae00a5430441318cafb32db2487516b893178b7f96ab9b2138fd164d9f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T20:24:48.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T20:24:48.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 systemd[1]: Stopped Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:24:48.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.0.service: Consumed 7.280s CPU time. 2024-09-20T20:24:48.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 systemd[1]: Starting Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:24:49.166 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 podman[105444]: 2024-09-20 20:24:48.738568246 +0000 UTC m=+0.015686284 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:24:49.166 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:48 smithi078 podman[105444]: 2024-09-20 20:24:48.844659962 +0000 UTC m=+0.121777988 container create 45b645de73a8119632e899c638a73dc6c7d97f45d0f25ee37d94dcc1cc2dfbe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:24:49.166 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 podman[105444]: 2024-09-20 20:24:49.072931584 +0000 UTC m=+0.350049629 container init 45b645de73a8119632e899c638a73dc6c7d97f45d0f25ee37d94dcc1cc2dfbe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-20T20:24:49.166 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 podman[105444]: 2024-09-20 20:24:49.076392502 +0000 UTC m=+0.353510529 container start 45b645de73a8119632e899c638a73dc6c7d97f45d0f25ee37d94dcc1cc2dfbe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T20:24:49.166 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 podman[105444]: 2024-09-20 20:24:49.164389542 +0000 UTC m=+0.441507574 container attach 45b645de73a8119632e899c638a73dc6c7d97f45d0f25ee37d94dcc1cc2dfbe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T20:24:49.480 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:24:49.480 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:24:49.480 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:24:49.480 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:24:49.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:24:49.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:24:49.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:24:49.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:24:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:49 smithi146 ceph-mon[103278]: pgmap v76: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:49 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:24:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-mon[91136]: pgmap v76: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:24:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:24:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-mon[99959]: pgmap v76: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[105459]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T20:24:50.287 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:49 smithi078 bash[105444]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T20:24:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:24:50] "GET /metrics HTTP/1.1" 200 38655 "" "Prometheus/2.43.0" 2024-09-20T20:24:50.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:50 smithi078 podman[105444]: 2024-09-20 20:24:50.420453714 +0000 UTC m=+1.697571742 container died 45b645de73a8119632e899c638a73dc6c7d97f45d0f25ee37d94dcc1cc2dfbe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:24:51.418 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 podman[105621]: 2024-09-20 20:24:51.059845575 +0000 UTC m=+1.059134291 container remove 45b645de73a8119632e899c638a73dc6c7d97f45d0f25ee37d94dcc1cc2dfbe8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-20T20:24:51.418 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 podman[105682]: 2024-09-20 20:24:51.185706841 +0000 UTC m=+0.015914808 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:24:51.418 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 podman[105682]: 2024-09-20 20:24:51.294142649 +0000 UTC m=+0.124350603 container create fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-20T20:24:51.848 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 podman[105682]: 2024-09-20 20:24:51.538585614 +0000 UTC m=+0.368793581 container init fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T20:24:51.848 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 podman[105682]: 2024-09-20 20:24:51.542040023 +0000 UTC m=+0.372247976 container start fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T20:24:51.848 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 ceph-osd[105701]: -- 172.21.15.78:0/415834971 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b4f257da40 con 0x55b4f255bc00 2024-09-20T20:24:51.848 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 bash[105682]: fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc 2024-09-20T20:24:51.848 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:51 smithi078 systemd[1]: Started Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: pgmap v77: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: pgmap v77: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:52.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:52.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:52.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: pgmap v77: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:24:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:53.704 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[105697]: 2024-09-20T20:24:53.427+0000 7f57cff3c640 -1 Falling back to public interface 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T20:24:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:53 smithi146 ceph-mon[103278]: Upgrade: osd.0 is safe to restart 2024-09-20T20:24:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[91136]: Upgrade: osd.0 is safe to restart 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T20:24:54.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:53 smithi078 ceph-mon[99959]: Upgrade: osd.0 is safe to restart 2024-09-20T20:24:54.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:54 smithi078 ceph-mon[91136]: pgmap v78: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/633 objects degraded (10.585%) 2024-09-20T20:24:54.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:54 smithi078 ceph-mon[99959]: pgmap v78: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/633 objects degraded (10.585%) 2024-09-20T20:24:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:54 smithi146 ceph-mon[103278]: pgmap v78: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 67/633 objects degraded (10.585%) 2024-09-20T20:24:56.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:56.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:56.065 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:56.065 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-20T20:24:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:24:57.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-mon[91136]: Upgrade: Updating osd.0 2024-09-20T20:24:57.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-mon[91136]: Deploying daemon osd.0 on smithi078 2024-09-20T20:24:57.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-mon[91136]: pgmap v79: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 411 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:57.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-mon[99959]: Upgrade: Updating osd.0 2024-09-20T20:24:57.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-mon[99959]: Deploying daemon osd.0 on smithi078 2024-09-20T20:24:57.175 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-mon[99959]: pgmap v79: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 411 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:57.175 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:56 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[105697]: 2024-09-20T20:24:56.985+0000 7f57cff3c640 -1 osd.0 105 log_to_monitors true 2024-09-20T20:24:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:56 smithi146 ceph-mon[103278]: Upgrade: Updating osd.0 2024-09-20T20:24:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:56 smithi146 ceph-mon[103278]: Deploying daemon osd.0 on smithi078 2024-09-20T20:24:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:56 smithi146 ceph-mon[103278]: pgmap v79: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 411 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:58.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:57 smithi078 ceph-mon[99959]: from='osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:24:58.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:57 smithi078 ceph-mon[99959]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:24:58.020 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:24:57 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[105697]: 2024-09-20T20:24:57.850+0000 7f57c7ce6640 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:24:58.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:57 smithi078 ceph-mon[91136]: from='osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:24:58.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:57 smithi078 ceph-mon[91136]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:24:58.187 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:24:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:57 smithi146 ceph-mon[103278]: from='osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:24:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:57 smithi146 ceph-mon[103278]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:24:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:24:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 15m - - 2024-09-20T20:24:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 15m - - 2024-09-20T20:24:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (9m) 6s ago 10m 26.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:24:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (10m) 18s ago 10m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:24:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3m) 6s ago 8m 59.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (114s) 18s ago 14m 448M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3m) 6s ago 16m 506M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2m) 6s ago 16m 56.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (48s) 18s ago 15m 35.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (70s) 6s ago 15m 41.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (10m) 6s ago 10m 13.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (10m) 18s ago 10m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (14m) 6s ago 14m 54.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 53a87889b1c5 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (14m) 6s ago 14m 62.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (13m) 6s ago 13m 59.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (13m) 6s ago 13m 60.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:24:58.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (12m) 18s ago 12m 66.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (12m) 18s ago 12m 59.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (12m) 18s ago 12m 51.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (11m) 18s ago 11m 63.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3m) 18s ago 11m 51.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (8m) 6s ago 8m 81.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (8m) 18s ago 8m 80.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (8m) 6s ago 8m 81.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:24:58.730 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (8m) 18s ago 8m 83.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: from='osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: pgmap v81: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: from='osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: pgmap v81: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:59.073 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:24:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:24:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:24:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:24:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7, 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 11, 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:24:59.074 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: from='osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: pgmap v81: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:24:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:24:59.349 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:24:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:24:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "5/23 daemons upgraded", 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:24:59.350 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:24:59.689 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded 2024-09-20T20:24:59.689 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:59.689 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-20T20:24:59.690 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-09-20T20:24:59.691 INFO:teuthology.orchestra.run.smithi078.stdout: pg 5.c is active+undersized+degraded, acting [1,4] 2024-09-20T20:25:00.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: from='client.64157 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413] boot 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/198182493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1946367967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[99959]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: from='client.64157 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413] boot 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/198182493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1946367967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:00.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:24:59 smithi078 ceph-mon[91136]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: from='client.64157 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: osd.0 [v2:172.21.15.78:6802/997271413,v1:172.21.15.78:6803/997271413] boot 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/198182493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1946367967' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:24:59 smithi146 ceph-mon[103278]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T20:25:00.841 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:00 smithi078 systemd[1]: Stopping Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:25:00.841 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:25:00] "GET /metrics HTTP/1.1" 200 38641 "" "Prometheus/2.43.0" 2024-09-20T20:25:01.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:01.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: from='client.64169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:01.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: pgmap v83: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:01.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 31/633 objects degraded (4.897%), 10 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: from='client.64169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: pgmap v83: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:00 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 31/633 objects degraded (4.897%), 10 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: from='client.64169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: pgmap v83: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:00 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 31/633 objects degraded (4.897%), 10 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:01.459 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[105697]: 2024-09-20T20:25:01.099+0000 7f57cced4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:25:01.459 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[105697]: 2024-09-20T20:25:01.099+0000 7f57cced4640 -1 osd.0 110 *** Got signal Terminated *** 2024-09-20T20:25:01.459 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[105697]: 2024-09-20T20:25:01.099+0000 7f57cced4640 -1 osd.0 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:25:02.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:01 smithi146 ceph-mon[103278]: osd.0 marked itself down and dead 2024-09-20T20:25:02.159 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:01 smithi078 ceph-mon[99959]: osd.0 marked itself down and dead 2024-09-20T20:25:02.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:01 smithi078 ceph-mon[91136]: osd.0 marked itself down and dead 2024-09-20T20:25:02.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:02 smithi078 podman[107385]: 2024-09-20 20:25:02.46929174 +0000 UTC m=+1.499144002 container died fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:25:03.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:02 smithi146 ceph-mon[103278]: pgmap v85: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:03.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:02 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:25:03.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:02 smithi146 ceph-mon[103278]: osdmap e111: 8 total, 7 up, 8 in 2024-09-20T20:25:03.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:02 smithi078 ceph-mon[91136]: pgmap v85: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:03.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:02 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:25:03.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:02 smithi078 ceph-mon[91136]: osdmap e111: 8 total, 7 up, 8 in 2024-09-20T20:25:03.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:02 smithi078 ceph-mon[99959]: pgmap v85: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:03.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:02 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:25:03.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:02 smithi078 ceph-mon[99959]: osdmap e111: 8 total, 7 up, 8 in 2024-09-20T20:25:03.187 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:02 smithi078 podman[107385]: 2024-09-20 20:25:02.971347112 +0000 UTC m=+2.001199384 container cleanup fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-20T20:25:03.187 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:02 smithi078 bash[107385]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0 2024-09-20T20:25:03.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:03 smithi078 podman[107522]: 2024-09-20 20:25:03.185098983 +0000 UTC m=+0.943638802 container remove fce6fb98f76557fbed4a65651e03f1b73840d950b4964bd913032bf411fbb4bc (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-20T20:25:04.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:03 smithi078 ceph-mon[91136]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T20:25:04.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:03 smithi078 ceph-mon[99959]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T20:25:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:03 smithi146 ceph-mon[103278]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T20:25:04.453 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:04 smithi078 podman[107736]: 2024-09-20 20:25:04.083923858 +0000 UTC m=+0.018334420 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:25:04.453 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:04 smithi078 podman[107736]: 2024-09-20 20:25:04.215100494 +0000 UTC m=+0.149511051 container create 56526ee2571decb557df58f96027a63444958aa6add20708442457e160a5a7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:25:04.453 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:04 smithi078 podman[107736]: 2024-09-20 20:25:04.451372703 +0000 UTC m=+0.385783273 container init 56526ee2571decb557df58f96027a63444958aa6add20708442457e160a5a7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T20:25:04.714 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:04 smithi078 podman[107736]: 2024-09-20 20:25:04.455452879 +0000 UTC m=+0.389863443 container start 56526ee2571decb557df58f96027a63444958aa6add20708442457e160a5a7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T20:25:04.714 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:04 smithi078 podman[107736]: 2024-09-20 20:25:04.534784048 +0000 UTC m=+0.469194606 container attach 56526ee2571decb557df58f96027a63444958aa6add20708442457e160a5a7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T20:25:04.714 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:04 smithi078 podman[107736]: 2024-09-20 20:25:04.667779319 +0000 UTC m=+0.602189876 container died 56526ee2571decb557df58f96027a63444958aa6add20708442457e160a5a7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-20T20:25:04.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[99959]: pgmap v88: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:04.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:04.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:04.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:04.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[91136]: pgmap v88: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:04.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:04.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:04.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:04 smithi146 ceph-mon[103278]: pgmap v88: 161 pgs: 30 peering, 18 active+undersized, 10 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 31/633 objects degraded (4.897%) 2024-09-20T20:25:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:05.619 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:05 smithi078 podman[107890]: 2024-09-20 20:25:05.339244647 +0000 UTC m=+0.662537881 container remove 56526ee2571decb557df58f96027a63444958aa6add20708442457e160a5a7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:25:05.619 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:05 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T20:25:05.619 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:05 smithi078 systemd[1]: Stopped Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:25:05.619 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:05 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.0.service: Consumed 1.764s CPU time. 2024-09-20T20:25:05.620 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:05 smithi078 systemd[1]: Starting Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:25:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:05 smithi146 ceph-mon[103278]: pgmap v89: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:05 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:05 smithi078 ceph-mon[91136]: pgmap v89: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:05 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:05 smithi078 ceph-mon[99959]: pgmap v89: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:05 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:06.586 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 podman[108020]: 2024-09-20 20:25:06.236360352 +0000 UTC m=+0.017272886 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:25:06.586 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 podman[108020]: 2024-09-20 20:25:06.336625126 +0000 UTC m=+0.117537686 container create e07af08192ab134b5974e53708364b2df5c7903becb038c3f78cb1f7372a85be (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T20:25:06.586 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 podman[108020]: 2024-09-20 20:25:06.58105025 +0000 UTC m=+0.361962784 container init e07af08192ab134b5974e53708364b2df5c7903becb038c3f78cb1f7372a85be (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T20:25:06.586 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 podman[108020]: 2024-09-20 20:25:06.584477624 +0000 UTC m=+0.365390139 container start e07af08192ab134b5974e53708364b2df5c7903becb038c3f78cb1f7372a85be (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T20:25:07.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 podman[108020]: 2024-09-20 20:25:06.697776017 +0000 UTC m=+0.478688547 container attach e07af08192ab134b5974e53708364b2df5c7903becb038c3f78cb1f7372a85be (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T20:25:07.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:07.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 bash[108020]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:07.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:07.036 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:06 smithi078 bash[108020]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:07.493 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:25:07.493 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:25:07.494 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:25:07.494 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:25:07.494 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:25:07.494 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:25:07.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:25:07.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:25:07.786 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T20:25:07.787 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T20:25:07.787 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:25:07.787 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T20:25:07.787 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate[108048]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T20:25:07.787 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 bash[108020]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T20:25:07.787 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:07 smithi078 podman[108020]: 2024-09-20 20:25:07.517022109 +0000 UTC m=+1.297934628 container died e07af08192ab134b5974e53708364b2df5c7903becb038c3f78cb1f7372a85be (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:25:08.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-mon[99959]: pgmap v90: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:08.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:07 smithi078 ceph-mon[91136]: pgmap v90: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:07 smithi146 ceph-mon[103278]: pgmap v90: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:08.891 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:08 smithi078 podman[108323]: 2024-09-20 20:25:08.564308504 +0000 UTC m=+1.034453684 container remove e07af08192ab134b5974e53708364b2df5c7903becb038c3f78cb1f7372a85be (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0-activate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:25:08.891 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:08 smithi078 podman[108456]: 2024-09-20 20:25:08.765309773 +0000 UTC m=+0.104941108 container create d07702be168701ff74680b7921c83178513b30e92b820be2b2e5db43d72203f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-20T20:25:08.891 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:08 smithi078 podman[108456]: 2024-09-20 20:25:08.677896142 +0000 UTC m=+0.017527502 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:25:09.457 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:09 smithi078 podman[108456]: 2024-09-20 20:25:09.186549798 +0000 UTC m=+0.526181144 container init d07702be168701ff74680b7921c83178513b30e92b820be2b2e5db43d72203f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T20:25:09.457 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:09 smithi078 podman[108456]: 2024-09-20 20:25:09.189916662 +0000 UTC m=+0.529548003 container start d07702be168701ff74680b7921c83178513b30e92b820be2b2e5db43d72203f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T20:25:09.457 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:09 smithi078 ceph-osd[108520]: -- 172.21.15.78:0/894090728 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d800f61a40 con 0x55d800f3fc00 2024-09-20T20:25:09.457 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:09 smithi078 bash[108456]: d07702be168701ff74680b7921c83178513b30e92b820be2b2e5db43d72203f5 2024-09-20T20:25:09.457 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:09 smithi078 systemd[1]: Started Ceph osd.0 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:25:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: pgmap v91: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: pgmap v91: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: pgmap v91: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:10.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:10.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:25:10] "GET /metrics HTTP/1.1" 200 38641 "" "Prometheus/2.43.0" 2024-09-20T20:25:11.276 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[108490]: 2024-09-20T20:25:11.077+0000 7f1586042640 -1 Falling back to public interface 2024-09-20T20:25:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:11.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:11.874 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:25:11.601+0000 7fcfc9d8b640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: pgmap v92: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[99959]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: pgmap v92: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:12 smithi078 ceph-mon[91136]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: pgmap v92: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:12.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:12.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:12 smithi146 ceph-mon[103278]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:25:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:13 smithi146 ceph-mon[103278]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.0 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:13 smithi078 ceph-mon[91136]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.0 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:13 smithi078 ceph-mon[99959]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.0 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:14 smithi078 ceph-mon[91136]: pgmap v93: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:14 smithi078 ceph-mon[99959]: pgmap v93: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:14 smithi146 ceph-mon[103278]: pgmap v93: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:14.873 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[108490]: 2024-09-20T20:25:14.653+0000 7f1586042640 -1 osd.0 110 log_to_monitors true 2024-09-20T20:25:15.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:15 smithi078 ceph-mon[91136]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:25:15.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:15 smithi078 ceph-mon[99959]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:25:15.743 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 20 20:25:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-0[108490]: 2024-09-20T20:25:15.375+0000 7f157ddec640 -1 osd.0 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:25:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:15 smithi146 ceph-mon[103278]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-20T20:25:16.650 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[99959]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[99959]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[99959]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[99959]: pgmap v95: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[91136]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[91136]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[91136]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:16.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:16 smithi078 ceph-mon[91136]: pgmap v95: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:16 smithi146 ceph-mon[103278]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T20:25:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:16 smithi146 ceph-mon[103278]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T20:25:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:16 smithi146 ceph-mon[103278]: from='osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:16 smithi146 ceph-mon[103278]: pgmap v95: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[99959]: osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897] boot 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[99959]: osdmap e114: 8 total, 8 up, 8 in 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[91136]: osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897] boot 2024-09-20T20:25:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[91136]: osdmap e114: 8 total, 8 up, 8 in 2024-09-20T20:25:17.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T20:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:17 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:17 smithi146 ceph-mon[103278]: osd.0 [v2:172.21.15.78:6802/626969897,v1:172.21.15.78:6803/626969897] boot 2024-09-20T20:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:17 smithi146 ceph-mon[103278]: osdmap e114: 8 total, 8 up, 8 in 2024-09-20T20:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-20T20:25:18.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:18 smithi078 ceph-mon[99959]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T20:25:18.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:18 smithi078 ceph-mon[99959]: pgmap v98: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:18.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:18 smithi078 ceph-mon[91136]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T20:25:18.642 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:18 smithi078 ceph-mon[91136]: pgmap v98: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:18 smithi146 ceph-mon[103278]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T20:25:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:18 smithi146 ceph-mon[103278]: pgmap v98: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:18.964 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:25:18 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:25:18.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:25:18.964 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:25:18 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:25:18.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:25:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:19 smithi146 ceph-mon[103278]: pgmap v99: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[91136]: pgmap v99: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[99959]: pgmap v99: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s; 67/633 objects degraded (10.585%) 2024-09-20T20:25:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:20 smithi146 ceph-mon[103278]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.0 on smithi078 is in unknown state']) 2024-09-20T20:25:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:25:20] "GET /metrics HTTP/1.1" 200 38643 "" "Prometheus/2.43.0" 2024-09-20T20:25:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[91136]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.0 on smithi078 is in unknown state']) 2024-09-20T20:25:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[99959]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.0 on smithi078 is in unknown state']) 2024-09-20T20:25:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:21 smithi146 ceph-mon[103278]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-09-20T20:25:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:21 smithi078 ceph-mon[91136]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-09-20T20:25:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:21 smithi078 ceph-mon[99959]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-09-20T20:25:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:22 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded) 2024-09-20T20:25:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:22 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded) 2024-09-20T20:25:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:22 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/633 objects degraded (10.585%), 20 pgs degraded) 2024-09-20T20:25:23.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:23 smithi146 ceph-mon[103278]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 726 B/s rd, 0 op/s 2024-09-20T20:25:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:23 smithi078 ceph-mon[91136]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 726 B/s rd, 0 op/s 2024-09-20T20:25:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:23 smithi078 ceph-mon[99959]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 726 B/s rd, 0 op/s 2024-09-20T20:25:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:25 smithi146 ceph-mon[103278]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:25 smithi078 ceph-mon[91136]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:25 smithi078 ceph-mon[99959]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:26.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:26.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:27 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:27 smithi146 ceph-mon[103278]: Upgrade: osd.1 is safe to restart 2024-09-20T20:25:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:27 smithi146 ceph-mon[103278]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:25:28.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:27 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:28.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:27 smithi078 ceph-mon[99959]: Upgrade: osd.1 is safe to restart 2024-09-20T20:25:28.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:27 smithi078 ceph-mon[99959]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:25:28.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:27 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:28.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:27 smithi078 ceph-mon[91136]: Upgrade: osd.1 is safe to restart 2024-09-20T20:25:28.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:27 smithi078 ceph-mon[91136]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:25:28.761 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:25:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:25:28.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:25:28.761 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:25:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:25:28.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:25:28.761 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:25:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:25:28.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: Upgrade: Updating osd.1 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: Deploying daemon osd.1 on smithi078 2024-09-20T20:25:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:29 smithi146 ceph-mon[103278]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: Upgrade: Updating osd.1 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: Deploying daemon osd.1 on smithi078 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[91136]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: Upgrade: Updating osd.1 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: Deploying daemon osd.1 on smithi078 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:29 smithi078 ceph-mon[99959]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-09-20T20:25:29.981 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:25:30.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:30.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:30.527 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:25:30.527 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 16m - - 2024-09-20T20:25:30.527 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 15m - - 2024-09-20T20:25:30.527 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (10m) 11s ago 11m 26.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:25:30.527 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (11m) 10s ago 11m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:25:30.527 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3m) 11s ago 9m 61.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2m) 10s ago 15m 449M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4m) 11s ago 16m 508M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2m) 11s ago 17m 58.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (80s) 10s ago 15m 38.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (102s) 11s ago 15m 44.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (10m) 11s ago 10m 13.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (10m) 10s ago 10m 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (21s) 11s ago 14m 13.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (14m) 11s ago 14m 62.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7b4a18cf32ee 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (14m) 11s ago 14m 60.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (13m) 11s ago 13m 61.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (13m) 10s ago 13m 67.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:25:30.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (13m) 10s ago 13m 61.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (12m) 10s ago 12m 52.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (12m) 10s ago 12m 66.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3m) 10s ago 11m 50.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (9m) 11s ago 9m 81.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (9m) 10s ago 9m 81.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (9m) 11s ago 9m 81.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:25:30.529 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (9m) 10s ago 9m 84.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:25:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:25:30] "GET /metrics HTTP/1.1" 200 38784 "" "Prometheus/2.43.0" 2024-09-20T20:25:30.842 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7, 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 1 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:25:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:25:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 11, 2024-09-20T20:25:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T20:25:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:25:30.844 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:25:31.128 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:25:31.128 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "6/23 daemons upgraded", 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:25:31.129 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:25:31.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[99959]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[99959]: from='client.64193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[99959]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[99959]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:31.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2291840960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:31.475 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:31 smithi078 systemd[1]: Stopping Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:25:31.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[91136]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[91136]: from='client.64193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[91136]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[91136]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:31.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2291840960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:31.476 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:31.476 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:25:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:31 smithi146 ceph-mon[103278]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:31 smithi146 ceph-mon[103278]: from='client.64193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:31 smithi146 ceph-mon[103278]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:31 smithi146 ceph-mon[103278]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:25:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2291840960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:32.138 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[37698]: 2024-09-20T20:25:31.828+0000 7f79ade6f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:25:32.138 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[37698]: 2024-09-20T20:25:31.828+0000 7f79ade6f640 -1 osd.1 115 *** Got signal Terminated *** 2024-09-20T20:25:32.138 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[37698]: 2024-09-20T20:25:31.828+0000 7f79ade6f640 -1 osd.1 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:25:32.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: from='client.64211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:32.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3761445669' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:32.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-09-20T20:25:32.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:32 smithi146 ceph-mon[103278]: osd.1 marked itself down and dead 2024-09-20T20:25:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: from='client.64211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3761445669' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[99959]: osd.1 marked itself down and dead 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: from='client.64211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3761445669' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:32.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:32 smithi078 ceph-mon[91136]: osd.1 marked itself down and dead 2024-09-20T20:25:33.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:33 smithi078 ceph-mon[99959]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:33.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:33 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:25:33.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:33 smithi078 ceph-mon[99959]: osdmap e116: 8 total, 7 up, 8 in 2024-09-20T20:25:33.383 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:33 smithi078 podman[110843]: 2024-09-20 20:25:33.080762018 +0000 UTC m=+1.539194435 container died 7b4a18cf32ee79e5ed45fec202d5a777d41fac97d94b92c521249793dbc20f82 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T20:25:33.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:33 smithi078 ceph-mon[91136]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:33.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:33 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:25:33.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:33 smithi078 ceph-mon[91136]: osdmap e116: 8 total, 7 up, 8 in 2024-09-20T20:25:33.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:33 smithi146 ceph-mon[103278]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:33.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:33 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:25:33.442 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:33 smithi146 ceph-mon[103278]: osdmap e116: 8 total, 7 up, 8 in 2024-09-20T20:25:33.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:33 smithi078 podman[110843]: 2024-09-20 20:25:33.562905608 +0000 UTC m=+2.021338069 container cleanup 7b4a18cf32ee79e5ed45fec202d5a777d41fac97d94b92c521249793dbc20f82 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T20:25:33.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:33 smithi078 bash[110843]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1 2024-09-20T20:25:34.240 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:33 smithi078 podman[110909]: 2024-09-20 20:25:33.823075075 +0000 UTC m=+0.812454010 container remove 7b4a18cf32ee79e5ed45fec202d5a777d41fac97d94b92c521249793dbc20f82 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-20T20:25:34.529 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:34 smithi078 podman[110998]: 2024-09-20 20:25:34.142315754 +0000 UTC m=+0.017522808 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:25:34.529 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:34 smithi078 podman[110998]: 2024-09-20 20:25:34.274127175 +0000 UTC m=+0.149334209 container create a53e82b6806fad518c3b4af68ce170b788726ec712aaf420d560aba4736339dd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T20:25:34.529 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:34 smithi078 podman[110998]: 2024-09-20 20:25:34.527643844 +0000 UTC m=+0.402850895 container init a53e82b6806fad518c3b4af68ce170b788726ec712aaf420d560aba4736339dd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T20:25:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:34 smithi078 ceph-mon[91136]: pgmap v108: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-09-20T20:25:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:34 smithi078 ceph-mon[91136]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T20:25:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:34 smithi078 ceph-mon[99959]: pgmap v108: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-09-20T20:25:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:34 smithi078 ceph-mon[99959]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T20:25:34.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:34 smithi078 podman[110998]: 2024-09-20 20:25:34.531205377 +0000 UTC m=+0.406412413 container start a53e82b6806fad518c3b4af68ce170b788726ec712aaf420d560aba4736339dd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:25:34.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:34 smithi078 podman[110998]: 2024-09-20 20:25:34.627901438 +0000 UTC m=+0.503108491 container attach a53e82b6806fad518c3b4af68ce170b788726ec712aaf420d560aba4736339dd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-09-20T20:25:34.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:34 smithi078 podman[110998]: 2024-09-20 20:25:34.726452331 +0000 UTC m=+0.601659382 container died a53e82b6806fad518c3b4af68ce170b788726ec712aaf420d560aba4736339dd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T20:25:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:34 smithi146 ceph-mon[103278]: pgmap v108: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-09-20T20:25:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:34 smithi146 ceph-mon[103278]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T20:25:35.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 podman[111021]: 2024-09-20 20:25:35.306915567 +0000 UTC m=+0.574724878 container remove a53e82b6806fad518c3b4af68ce170b788726ec712aaf420d560aba4736339dd (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T20:25:35.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T20:25:35.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 systemd[1]: Stopped Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:25:35.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.1.service: Consumed 8.289s CPU time. 2024-09-20T20:25:35.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:35 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 85/633 objects degraded (13.428%), 22 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:35.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:35 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 85/633 objects degraded (13.428%), 22 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:35.801 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 systemd[1]: Starting Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:25:35.801 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 podman[111121]: 2024-09-20 20:25:35.667102076 +0000 UTC m=+0.019635562 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:25:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:35 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 85/633 objects degraded (13.428%), 22 pgs degraded (PG_DEGRADED) 2024-09-20T20:25:36.121 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:35 smithi078 podman[111121]: 2024-09-20 20:25:35.799934531 +0000 UTC m=+0.152468000 container create 1483e1ef9a20b246c643c0f3c03b016646b4e43ad7c08581bb4507f614438d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-20T20:25:36.121 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 podman[111121]: 2024-09-20 20:25:36.036236804 +0000 UTC m=+0.388770274 container init 1483e1ef9a20b246c643c0f3c03b016646b4e43ad7c08581bb4507f614438d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:25:36.121 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 podman[111121]: 2024-09-20 20:25:36.039520704 +0000 UTC m=+0.392054169 container start 1483e1ef9a20b246c643c0f3c03b016646b4e43ad7c08581bb4507f614438d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-20T20:25:36.121 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 podman[111121]: 2024-09-20 20:25:36.119336182 +0000 UTC m=+0.471869669 container attach 1483e1ef9a20b246c643c0f3c03b016646b4e43ad7c08581bb4507f614438d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-20T20:25:36.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:36.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:36.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:36.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:25:36.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-mon[99959]: pgmap v110: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:36.937 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:25:36.937 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:25:36.937 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:25:36.938 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:25:36.938 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:25:36.938 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:25:36.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-mon[91136]: pgmap v110: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:36 smithi146 ceph-mon[103278]: pgmap v110: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:37.258 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:25:37.258 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:25:37.258 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T20:25:37.258 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T20:25:37.259 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:25:37.259 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:25:37.259 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[111144]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T20:25:37.259 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 bash[111121]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T20:25:37.259 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:36 smithi078 podman[111121]: 2024-09-20 20:25:36.962906848 +0000 UTC m=+1.315440321 container died 1483e1ef9a20b246c643c0f3c03b016646b4e43ad7c08581bb4507f614438d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T20:25:37.990 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:37 smithi078 podman[111291]: 2024-09-20 20:25:37.66544462 +0000 UTC m=+0.696243480 container remove 1483e1ef9a20b246c643c0f3c03b016646b4e43ad7c08581bb4507f614438d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:25:37.990 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:37 smithi078 podman[111349]: 2024-09-20 20:25:37.776193874 +0000 UTC m=+0.015559392 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:25:37.990 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:37 smithi078 podman[111349]: 2024-09-20 20:25:37.883442308 +0000 UTC m=+0.122807895 container create ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T20:25:38.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:38 smithi078 podman[111349]: 2024-09-20 20:25:38.102473156 +0000 UTC m=+0.341838675 container init ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T20:25:38.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:38 smithi078 podman[111349]: 2024-09-20 20:25:38.105750951 +0000 UTC m=+0.345116465 container start ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:25:38.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-osd[111369]: -- 172.21.15.78:0/930612234 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d5b9d6ba40 con 0x55d5bab3a400 2024-09-20T20:25:38.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:38 smithi078 bash[111349]: ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a 2024-09-20T20:25:38.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:38 smithi078 systemd[1]: Started Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:25:38.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:25:38.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: pgmap v111: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: pgmap v111: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:38.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:38.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:38.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:38.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: pgmap v111: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: pgmap v112: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: pgmap v112: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state']) 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state']) 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:39.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: pgmap v112: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state']) 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.179 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:39 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[111365]: 2024-09-20T20:25:39.990+0000 7f50706ee640 -1 Falling back to public interface 2024-09-20T20:25:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:25:40] "GET /metrics HTTP/1.1" 200 38784 "" "Prometheus/2.43.0" 2024-09-20T20:25:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:40.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: pgmap v113: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: pgmap v113: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:41.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: pgmap v113: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.716 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[111365]: 2024-09-20T20:25:43.559+0000 7f50706ee640 -1 osd.1 115 log_to_monitors true 2024-09-20T20:25:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: pgmap v114: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.774 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.775 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:43.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:43.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:43.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:43.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: pgmap v114: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: pgmap v114: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.015 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[111365]: 2024-09-20T20:25:44.549+0000 7f5064490640 -1 osd.1 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:25:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:44.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: pgmap v115: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: pgmap v115: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802] boot 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802] boot 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: pgmap v115: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 85/633 objects degraded (13.428%) 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: osd.1 [v2:172.21.15.78:6810/2908028802,v1:172.21.15.78:6811/2908028802] boot 2024-09-20T20:25:45.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:45.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:46.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:46.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:46.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:46.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T20:25:47.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: pgmap v118: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: pgmap v118: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:25:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: pgmap v118: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:48.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:49.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: pgmap v120: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: pgmap v120: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: pgmap v120: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.030 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:50.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:50.644 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:25:50] "GET /metrics HTTP/1.1" 200 38716 "" "Prometheus/2.43.0" 2024-09-20T20:25:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: pgmap v121: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: pgmap v121: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:51.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: pgmap v121: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[99959]: Upgrade: osd.1 is safe to restart 2024-09-20T20:25:52.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:52.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:52.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:52 smithi078 ceph-mon[91136]: Upgrade: osd.1 is safe to restart 2024-09-20T20:25:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T20:25:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:52 smithi146 ceph-mon[103278]: Upgrade: osd.1 is safe to restart 2024-09-20T20:25:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:53 smithi146 ceph-mon[103278]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:53 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded) 2024-09-20T20:25:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:53 smithi078 ceph-mon[91136]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:54.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:53 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded) 2024-09-20T20:25:54.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:53 smithi078 ceph-mon[99959]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:25:54.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:53 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded) 2024-09-20T20:25:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:54 smithi146 ceph-mon[103278]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 618 B/s rd, 0 op/s 2024-09-20T20:25:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:54 smithi078 ceph-mon[91136]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 618 B/s rd, 0 op/s 2024-09-20T20:25:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:54 smithi078 ceph-mon[99959]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 618 B/s rd, 0 op/s 2024-09-20T20:25:57.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:56 smithi078 ceph-mon[99959]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:25:57.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:25:57.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:57.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:56 smithi078 ceph-mon[91136]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:25:57.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:25:57.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:56 smithi146 ceph-mon[103278]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:25:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-20T20:25:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:25:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:57 smithi078 ceph-mon[91136]: Upgrade: Updating osd.1 2024-09-20T20:25:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:57 smithi078 ceph-mon[91136]: Deploying daemon osd.1 on smithi078 2024-09-20T20:25:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:57 smithi078 ceph-mon[99959]: Upgrade: Updating osd.1 2024-09-20T20:25:58.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:57 smithi078 ceph-mon[99959]: Deploying daemon osd.1 on smithi078 2024-09-20T20:25:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:57 smithi146 ceph-mon[103278]: Upgrade: Updating osd.1 2024-09-20T20:25:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:57 smithi146 ceph-mon[103278]: Deploying daemon osd.1 on smithi078 2024-09-20T20:25:58.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-09-20T20:25:58.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:58.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:58.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:25:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[91136]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[91136]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state']) 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[99959]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:25:59 smithi078 ceph-mon[99959]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state']) 2024-09-20T20:26:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:59 smithi146 ceph-mon[103278]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:26:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:25:59 smithi146 ceph-mon[103278]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.1 on smithi078 is in unknown state']) 2024-09-20T20:26:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:26:00] "GET /metrics HTTP/1.1" 200 38780 "" "Prometheus/2.43.0" 2024-09-20T20:26:01.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:01.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:01.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:01.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:01 smithi078 systemd[1]: Stopping Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:26:01.801 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:26:02.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:01 smithi146 ceph-mon[103278]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T20:26:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:01 smithi078 ceph-mon[99959]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T20:26:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:01 smithi078 ceph-mon[91136]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-20T20:26:02.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[111365]: 2024-09-20T20:26:01.923+0000 7f506d686640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:26:02.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[111365]: 2024-09-20T20:26:01.923+0000 7f506d686640 -1 osd.1 120 *** Got signal Terminated *** 2024-09-20T20:26:02.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[111365]: 2024-09-20T20:26:01.923+0000 7f506d686640 -1 osd.1 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:26:02.343 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:26:02.343 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 16m - - 2024-09-20T20:26:02.343 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 16m - - 2024-09-20T20:26:02.343 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (10m) 3s ago 11m 25.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:26:02.343 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (11m) 2s ago 11m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4m) 3s ago 9m 61.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2m) 2s ago 15m 449M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4m) 3s ago 17m 511M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3m) 3s ago 17m 59.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (111s) 2s ago 16m 37.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2m) 3s ago 16m 42.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (11m) 3s ago 11m 14.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (11m) 2s ago 11m 15.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (53s) 3s ago 15m 43.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (24s) 3s ago 15m 43.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ae34b6256d4e 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (14m) 3s ago 14m 61.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (14m) 3s ago 14m 62.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (13m) 2s ago 13m 67.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:26:02.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (13m) 2s ago 13m 61.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (13m) 2s ago 13m 53.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (12m) 2s ago 12m 66.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4m) 2s ago 12m 51.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (9m) 3s ago 9m 81.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (10m) 2s ago 10m 81.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (9m) 3s ago 9m 82.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:26:02.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (9m) 2s ago 9m 84.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:26:02.659 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:26:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:26:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:26:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 10, 2024-09-20T20:26:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T20:26:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:26:02.661 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:26:02.941 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "7/23 daemons upgraded", 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:26:02.942 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:26:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:26:02.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:02 smithi078 ceph-mon[99959]: from='client.64229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:02.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:02 smithi078 ceph-mon[99959]: osd.1 marked itself down and dead 2024-09-20T20:26:02.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/447749567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:02.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:02 smithi078 ceph-mon[91136]: from='client.64229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:02.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:02 smithi078 ceph-mon[91136]: osd.1 marked itself down and dead 2024-09-20T20:26:02.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/447749567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:03.256 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down 2024-09-20T20:26:03.256 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:03.256 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:26:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: osd.1 (root=default,host=smithi078) is down 2024-09-20T20:26:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:02 smithi146 ceph-mon[103278]: from='client.64229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:02 smithi146 ceph-mon[103278]: osd.1 marked itself down and dead 2024-09-20T20:26:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/447749567' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:03.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:03 smithi078 podman[116030]: 2024-09-20 20:26:03.315143689 +0000 UTC m=+1.493537115 container died ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: from='client.64235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: from='client.64241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: osdmap e121: 8 total, 7 up, 8 in 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: from='client.64253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2352418544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:04.037 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:03 smithi078 podman[116030]: 2024-09-20 20:26:03.835232795 +0000 UTC m=+2.013626222 container cleanup ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-09-20T20:26:04.037 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:03 smithi078 bash[116030]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: from='client.64235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: from='client.64241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: osdmap e121: 8 total, 7 up, 8 in 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: from='client.64253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2352418544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:04.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T20:26:04.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: from='client.64235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: from='client.64241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:26:04.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: osdmap e121: 8 total, 7 up, 8 in 2024-09-20T20:26:04.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: from='client.64253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:04.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2352418544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:04.498 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:04 smithi078 podman[116155]: 2024-09-20 20:26:04.050715827 +0000 UTC m=+0.726121935 container remove ae34b6256d4eab22f1eb3e3944c3a951bac0caea9b83c1865bf9aaf74477a92a (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:26:04.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:04 smithi078 podman[116243]: 2024-09-20 20:26:04.399567455 +0000 UTC m=+0.015849266 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:26:04.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:04 smithi078 podman[116243]: 2024-09-20 20:26:04.510206405 +0000 UTC m=+0.126488249 container create 73369104764f81071bedd635117a34c0a1898ae166c5a3e4582f7946257caed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T20:26:05.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:04 smithi146 ceph-mon[103278]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T20:26:05.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:04 smithi146 ceph-mon[103278]: pgmap v131: 161 pgs: 41 peering, 17 stale+active+clean, 103 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T20:26:05.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:04 smithi078 ceph-mon[99959]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T20:26:05.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:04 smithi078 ceph-mon[99959]: pgmap v131: 161 pgs: 41 peering, 17 stale+active+clean, 103 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T20:26:05.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:04 smithi078 ceph-mon[91136]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T20:26:05.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:04 smithi078 ceph-mon[91136]: pgmap v131: 161 pgs: 41 peering, 17 stale+active+clean, 103 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 118 op/s 2024-09-20T20:26:05.253 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:04 smithi078 podman[116243]: 2024-09-20 20:26:04.813046982 +0000 UTC m=+0.429328787 container init 73369104764f81071bedd635117a34c0a1898ae166c5a3e4582f7946257caed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2) 2024-09-20T20:26:05.253 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:04 smithi078 podman[116243]: 2024-09-20 20:26:04.816943741 +0000 UTC m=+0.433225547 container start 73369104764f81071bedd635117a34c0a1898ae166c5a3e4582f7946257caed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T20:26:05.253 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:04 smithi078 podman[116243]: 2024-09-20 20:26:04.913060263 +0000 UTC m=+0.529342079 container attach 73369104764f81071bedd635117a34c0a1898ae166c5a3e4582f7946257caed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-20T20:26:05.253 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:05 smithi078 podman[116243]: 2024-09-20 20:26:05.009189435 +0000 UTC m=+0.625471244 container died 73369104764f81071bedd635117a34c0a1898ae166c5a3e4582f7946257caed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T20:26:05.966 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:05 smithi078 podman[116266]: 2024-09-20 20:26:05.73381588 +0000 UTC m=+0.715355611 container remove 73369104764f81071bedd635117a34c0a1898ae166c5a3e4582f7946257caed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T20:26:05.966 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:05 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T20:26:05.966 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:05 smithi078 systemd[1]: Stopped Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:26:05.967 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:05 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.1.service: Consumed 1.914s CPU time. 2024-09-20T20:26:05.967 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:05 smithi078 systemd[1]: Starting Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:26:06.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:06 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:26:06.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:06 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:26:06.229 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 podman[116375]: 2024-09-20 20:26:06.080013707 +0000 UTC m=+0.016566588 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:26:06.229 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 podman[116375]: 2024-09-20 20:26:06.210375219 +0000 UTC m=+0.146928096 container create 57de41944ea45aa1d18394c93036b01c69fd7e803edb5b773001b59b44a7c02d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T20:26:06.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:06 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:26:06.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 podman[116375]: 2024-09-20 20:26:06.447338459 +0000 UTC m=+0.383891326 container init 57de41944ea45aa1d18394c93036b01c69fd7e803edb5b773001b59b44a7c02d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T20:26:06.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 podman[116375]: 2024-09-20 20:26:06.450894569 +0000 UTC m=+0.387447444 container start 57de41944ea45aa1d18394c93036b01c69fd7e803edb5b773001b59b44a7c02d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T20:26:07.036 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 podman[116375]: 2024-09-20 20:26:06.558338595 +0000 UTC m=+0.494891471 container attach 57de41944ea45aa1d18394c93036b01c69fd7e803edb5b773001b59b44a7c02d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:26:07.036 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:07.036 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 bash[116375]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:07.036 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:07.036 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:06 smithi078 bash[116375]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:07.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-mon[99959]: pgmap v132: 161 pgs: 5 active+undersized, 41 peering, 16 stale+active+clean, 1 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:26:07.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-mon[91136]: pgmap v132: 161 pgs: 5 active+undersized, 41 peering, 16 stale+active+clean, 1 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:26:07.339 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:26:07.339 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:26:07.339 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:26:07.339 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:26:07.339 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:26:07.339 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:26:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:07 smithi146 ceph-mon[103278]: pgmap v132: 161 pgs: 5 active+undersized, 41 peering, 16 stale+active+clean, 1 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate[116391]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 bash[116375]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T20:26:07.737 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:07 smithi078 podman[116375]: 2024-09-20 20:26:07.36651556 +0000 UTC m=+1.303068435 container died 57de41944ea45aa1d18394c93036b01c69fd7e803edb5b773001b59b44a7c02d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-20T20:26:08.467 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 podman[116538]: 2024-09-20 20:26:08.14307331 +0000 UTC m=+0.769516844 container remove 57de41944ea45aa1d18394c93036b01c69fd7e803edb5b773001b59b44a7c02d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:26:08.467 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 podman[116596]: 2024-09-20 20:26:08.250967055 +0000 UTC m=+0.016350035 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:26:08.467 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 podman[116596]: 2024-09-20 20:26:08.369117968 +0000 UTC m=+0.134500935 container create 0520846b86c322cd10457a45bc0dbaa6164c3431f0e5ad48a678900e3727f70f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T20:26:08.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 podman[116596]: 2024-09-20 20:26:08.588536174 +0000 UTC m=+0.353919138 container init 0520846b86c322cd10457a45bc0dbaa6164c3431f0e5ad48a678900e3727f70f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-20T20:26:08.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 podman[116596]: 2024-09-20 20:26:08.591900784 +0000 UTC m=+0.357283742 container start 0520846b86c322cd10457a45bc0dbaa6164c3431f0e5ad48a678900e3727f70f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T20:26:08.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 ceph-osd[116615]: -- 172.21.15.78:0/958511604 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e66098fa40 con 0x55e66096dc00 2024-09-20T20:26:08.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 bash[116596]: 0520846b86c322cd10457a45bc0dbaa6164c3431f0e5ad48a678900e3727f70f 2024-09-20T20:26:08.786 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:08 smithi078 systemd[1]: Started Ceph osd.1 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: pgmap v133: 161 pgs: 5 active+undersized, 41 peering, 16 stale+active+clean, 1 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.843 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: pgmap v133: 161 pgs: 5 active+undersized, 41 peering, 16 stale+active+clean, 1 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.844 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.846 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.848 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:09.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: pgmap v133: 161 pgs: 5 active+undersized, 41 peering, 16 stale+active+clean, 1 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s; 6/633 objects degraded (0.948%) 2024-09-20T20:26:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:10.611 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[116611]: 2024-09-20T20:26:10.478+0000 7f64c5b6e640 -1 Falling back to public interface 2024-09-20T20:26:11.003 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:26:10] "GET /metrics HTTP/1.1" 200 38780 "" "Prometheus/2.43.0" 2024-09-20T20:26:11.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:11.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:11.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:11.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: pgmap v134: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s rd, 0 B/s wr, 238 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 85/633 objects degraded (13.428%), 22 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: pgmap v134: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s rd, 0 B/s wr, 238 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: pgmap v134: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s rd, 0 B/s wr, 238 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:12.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 85/633 objects degraded (13.428%), 22 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 85/633 objects degraded (13.428%), 22 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:12.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:12.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:12.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:12.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:13.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: pgmap v135: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 0 B/s wr, 223 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: pgmap v135: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 0 B/s wr, 223 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: pgmap v135: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 0 B/s wr, 223 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:14.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.376 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[116611]: 2024-09-20T20:26:14.037+0000 7f64c5b6e640 -1 osd.1 120 log_to_monitors true 2024-09-20T20:26:14.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:14.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:15.286 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-1[116611]: 2024-09-20T20:26:15.018+0000 7f64b9910640 -1 osd.1 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:26:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: pgmap v136: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 200 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:16.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: pgmap v136: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 200 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:26:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: pgmap v136: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 0 B/s wr, 200 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749] boot 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: osdmap e124: 8 total, 8 up, 8 in 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749] boot 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: osdmap e124: 8 total, 8 up, 8 in 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: osd.1 [v2:172.21.15.78:6810/3891336749,v1:172.21.15.78:6811/3891336749] boot 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: osdmap e124: 8 total, 8 up, 8 in 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: pgmap v139: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 152 KiB/s rd, 0 B/s wr, 252 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T20:26:18.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: pgmap v139: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 152 KiB/s rd, 0 B/s wr, 252 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: pgmap v139: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 152 KiB/s rd, 0 B/s wr, 252 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T20:26:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:18.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:18.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:18.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:18.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:19.285 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:26:19.140+0000 7fcfc9d8b640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T20:26:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: pgmap v141: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: pgmap v141: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: pgmap v141: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 85/633 objects degraded (13.428%) 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T20:26:20.863 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:26:20] "GET /metrics HTTP/1.1" 200 38782 "" "Prometheus/2.43.0" 2024-09-20T20:26:20.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T20:26:20.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T20:26:20.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:20.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:21 smithi078 ceph-mon[91136]: pgmap v142: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s; 6/633 objects degraded (0.948%); 0 B/s, 8 objects/s recovering 2024-09-20T20:26:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:21 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:26:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:21 smithi078 ceph-mon[99959]: pgmap v142: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s; 6/633 objects degraded (0.948%); 0 B/s, 8 objects/s recovering 2024-09-20T20:26:22.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:21 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:26:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:21 smithi146 ceph-mon[103278]: pgmap v142: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s; 6/633 objects degraded (0.948%); 0 B/s, 8 objects/s recovering 2024-09-20T20:26:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:21 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded (PG_DEGRADED) 2024-09-20T20:26:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:22 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded) 2024-09-20T20:26:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:22 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded) 2024-09-20T20:26:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:22 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/633 objects degraded (0.948%), 1 pg degraded) 2024-09-20T20:26:24.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:23 smithi078 ceph-mon[99959]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s; 0 B/s, 8 objects/s recovering 2024-09-20T20:26:24.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:23 smithi078 ceph-mon[91136]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s; 0 B/s, 8 objects/s recovering 2024-09-20T20:26:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:23 smithi146 ceph-mon[103278]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s; 0 B/s, 8 objects/s recovering 2024-09-20T20:26:25.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T20:26:25.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:25.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:25.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:25.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:25.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:25.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T20:26:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:26.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:27 smithi146 ceph-mon[103278]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:26:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:27 smithi078 ceph-mon[91136]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:26:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:27 smithi078 ceph-mon[99959]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:26:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[91136]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:26:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[99959]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:26:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:29 smithi146 ceph-mon[103278]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T20:26:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:26:30] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.43.0" 2024-09-20T20:26:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:31 smithi146 ceph-mon[103278]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T20:26:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:31 smithi078 ceph-mon[91136]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T20:26:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:31 smithi078 ceph-mon[99959]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T20:26:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:33 smithi146 ceph-mon[103278]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:26:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:33 smithi078 ceph-mon[91136]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:26:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:33 smithi078 ceph-mon[99959]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T20:26:33.559 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:26:34.088 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 17m - - 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 16m - - 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (11m) 9s ago 12m 25.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (12m) 13s ago 12m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4m) 9s ago 10m 61.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3m) 13s ago 16m 449M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5m) 9s ago 17m 512M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3m) 9s ago 18m 61.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2m) 13s ago 16m 40.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2m) 9s ago 16m 45.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (11m) 9s ago 11m 14.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (11m) 13s ago 11m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:26:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (85s) 9s ago 16m 46.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (25s) 9s ago 15m 44.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (15m) 9s ago 15m 64.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 77678ed758a5 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (14m) 9s ago 14m 65.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (14m) 13s ago 14m 70.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (14m) 13s ago 14m 64.8M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (13m) 13s ago 13m 56.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (13m) 13s ago 13m 68.2M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4m) 13s ago 12m 54.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (10m) 9s ago 10m 81.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (10m) 13s ago 10m 81.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (10m) 9s ago 10m 82.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:26:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (10m) 13s ago 10m 84.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:26:34.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:34 smithi078 ceph-mon[91136]: from='client.44343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:34.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:34 smithi078 ceph-mon[91136]: from='client.44349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:34.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:34 smithi078 ceph-mon[99959]: from='client.44343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:34.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:34 smithi078 ceph-mon[99959]: from='client.44349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:26:34.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:26:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:26:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:26:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 10, 2024-09-20T20:26:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 7 2024-09-20T20:26:34.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:26:34.432 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:26:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:34 smithi146 ceph-mon[103278]: from='client.44343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:34 smithi146 ceph-mon[103278]: from='client.44349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:26:34.709 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "7/23 daemons upgraded", 2024-09-20T20:26:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:26:34.710 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:26:34.710 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:26:35.027 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:35.027 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:35.027 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: Upgrade: osd.2 is safe to restart 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4021999739' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: from='client.44367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3606652811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: Upgrade: osd.2 is safe to restart 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4021999739' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: from='client.44367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3606652811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: Upgrade: osd.2 is safe to restart 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4021999739' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: from='client.44367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:26:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3606652811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:26:37.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[99959]: Upgrade: Updating osd.2 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[99959]: Deploying daemon osd.2 on smithi078 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[91136]: Upgrade: Updating osd.2 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T20:26:37.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:37.139 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:36 smithi078 ceph-mon[91136]: Deploying daemon osd.2 on smithi078 2024-09-20T20:26:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:36 smithi146 ceph-mon[103278]: Upgrade: Updating osd.2 2024-09-20T20:26:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-20T20:26:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:36 smithi146 ceph-mon[103278]: Deploying daemon osd.2 on smithi078 2024-09-20T20:26:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:37 smithi078 ceph-mon[91136]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:37 smithi078 ceph-mon[99959]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:37 smithi146 ceph-mon[103278]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:38.950 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:38 smithi078 systemd[1]: Stopping Ceph osd.2 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:26:39.202 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[41209]: 2024-09-20T20:26:38.972+0000 7f4e93394640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:26:39.202 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[41209]: 2024-09-20T20:26:38.972+0000 7f4e93394640 -1 osd.2 125 *** Got signal Terminated *** 2024-09-20T20:26:39.202 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[41209]: 2024-09-20T20:26:38.972+0000 7f4e93394640 -1 osd.2 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:26:40.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:39 smithi078 ceph-mon[91136]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:40.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:39 smithi078 ceph-mon[91136]: osd.2 marked itself down and dead 2024-09-20T20:26:40.096 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:39 smithi078 ceph-mon[99959]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:40.096 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:39 smithi078 ceph-mon[99959]: osd.2 marked itself down and dead 2024-09-20T20:26:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:39 smithi146 ceph-mon[103278]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:26:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:39 smithi146 ceph-mon[103278]: osd.2 marked itself down and dead 2024-09-20T20:26:40.358 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:40 smithi078 podman[120614]: 2024-09-20 20:26:40.335076977 +0000 UTC m=+1.464523905 container died 77678ed758a5a5da15758df9abc09402799962076f0b38fb26747a64b02d7497 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-20T20:26:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:26:40] "GET /metrics HTTP/1.1" 200 38781 "" "Prometheus/2.43.0" 2024-09-20T20:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[91136]: osdmap e126: 8 total, 7 up, 8 in 2024-09-20T20:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[99959]: osdmap e126: 8 total, 7 up, 8 in 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.038 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:40 smithi078 podman[120614]: 2024-09-20 20:26:40.830129786 +0000 UTC m=+1.959576722 container cleanup 77678ed758a5a5da15758df9abc09402799962076f0b38fb26747a64b02d7497 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:26:41.038 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:40 smithi078 bash[120614]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2 2024-09-20T20:26:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:40 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:26:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:40 smithi146 ceph-mon[103278]: osdmap e126: 8 total, 7 up, 8 in 2024-09-20T20:26:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:41.452 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:41 smithi078 podman[120752]: 2024-09-20 20:26:41.090151703 +0000 UTC m=+0.747670773 container remove 77678ed758a5a5da15758df9abc09402799962076f0b38fb26747a64b02d7497 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:26:42.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:41 smithi078 ceph-mon[91136]: pgmap v153: 161 pgs: 6 peering, 10 stale+active+clean, 145 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:26:42.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:41 smithi078 ceph-mon[91136]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T20:26:42.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:41 smithi078 ceph-mon[99959]: pgmap v153: 161 pgs: 6 peering, 10 stale+active+clean, 145 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:26:42.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:41 smithi078 ceph-mon[99959]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T20:26:42.197 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:42 smithi078 podman[120967]: 2024-09-20 20:26:41.987251713 +0000 UTC m=+0.017942135 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:26:42.197 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:42 smithi078 podman[120967]: 2024-09-20 20:26:42.094566887 +0000 UTC m=+0.125257358 container create 463a77eeabcaf42a89569f93e9f7496b1eab98579ac55bf45ed39172d166769f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-09-20T20:26:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:41 smithi146 ceph-mon[103278]: pgmap v153: 161 pgs: 6 peering, 10 stale+active+clean, 145 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:26:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:41 smithi146 ceph-mon[103278]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T20:26:42.528 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:42 smithi078 podman[120967]: 2024-09-20 20:26:42.324335881 +0000 UTC m=+0.355026302 container init 463a77eeabcaf42a89569f93e9f7496b1eab98579ac55bf45ed39172d166769f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-20T20:26:42.528 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:42 smithi078 podman[120967]: 2024-09-20 20:26:42.327739105 +0000 UTC m=+0.358429528 container start 463a77eeabcaf42a89569f93e9f7496b1eab98579ac55bf45ed39172d166769f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T20:26:42.528 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:42 smithi078 podman[120967]: 2024-09-20 20:26:42.424206518 +0000 UTC m=+0.454896959 container attach 463a77eeabcaf42a89569f93e9f7496b1eab98579ac55bf45ed39172d166769f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T20:26:42.786 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:42 smithi078 podman[120967]: 2024-09-20 20:26:42.527541698 +0000 UTC m=+0.558232124 container died 463a77eeabcaf42a89569f93e9f7496b1eab98579ac55bf45ed39172d166769f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:26:43.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:42 smithi078 ceph-mon[99959]: pgmap v155: 161 pgs: 13 peering, 8 stale+active+clean, 140 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:26:43.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:42 smithi078 ceph-mon[99959]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T20:26:43.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:42 smithi078 ceph-mon[91136]: pgmap v155: 161 pgs: 13 peering, 8 stale+active+clean, 140 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:26:43.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:42 smithi078 ceph-mon[91136]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T20:26:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:42 smithi146 ceph-mon[103278]: pgmap v155: 161 pgs: 13 peering, 8 stale+active+clean, 140 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:26:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:42 smithi146 ceph-mon[103278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-20T20:26:43.536 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 podman[121007]: 2024-09-20 20:26:43.153248728 +0000 UTC m=+0.619847814 container remove 463a77eeabcaf42a89569f93e9f7496b1eab98579ac55bf45ed39172d166769f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-20T20:26:43.536 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-20T20:26:43.536 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 systemd[1]: Stopped Ceph osd.2 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:26:43.536 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.2.service: Consumed 7.741s CPU time. 2024-09-20T20:26:43.536 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 systemd[1]: Starting Ceph osd.2 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:26:43.900 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 podman[121159]: 2024-09-20 20:26:43.504627219 +0000 UTC m=+0.016647639 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:26:43.900 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 podman[121159]: 2024-09-20 20:26:43.646143502 +0000 UTC m=+0.158163957 container create 151019d8508b4ad6434a61546a0f8c4a2c57e3abc0cbc667e87d923e4c2e90d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-20T20:26:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 podman[121159]: 2024-09-20 20:26:43.898938358 +0000 UTC m=+0.410958776 container init 151019d8508b4ad6434a61546a0f8c4a2c57e3abc0cbc667e87d923e4c2e90d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:43 smithi078 podman[121159]: 2024-09-20 20:26:43.902736161 +0000 UTC m=+0.414756583 container start 151019d8508b4ad6434a61546a0f8c4a2c57e3abc0cbc667e87d923e4c2e90d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux ) 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 podman[121159]: 2024-09-20 20:26:44.007272497 +0000 UTC m=+0.519292910 container attach 151019d8508b4ad6434a61546a0f8c4a2c57e3abc0cbc667e87d923e4c2e90d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:44.287 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:26:44.373 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:44.373 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:44.786 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T20:26:44.787 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T20:26:44.787 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:26:44.787 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:26:45.105 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:26:45.105 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:26:45.105 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:26:45.105 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:26:45.105 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T20:26:45.106 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T20:26:45.106 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T20:26:45.106 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T20:26:45.106 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate[121189]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T20:26:45.106 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 bash[121159]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T20:26:45.106 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:44 smithi078 podman[121159]: 2024-09-20 20:26:44.949279986 +0000 UTC m=+1.461300406 container died 151019d8508b4ad6434a61546a0f8c4a2c57e3abc0cbc667e87d923e4c2e90d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2) 2024-09-20T20:26:45.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:45 smithi146 ceph-mon[103278]: pgmap v156: 161 pgs: 19 active+undersized, 13 peering, 13 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:26:45.415 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:45 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 34/633 objects degraded (5.371%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:45 smithi078 ceph-mon[99959]: pgmap v156: 161 pgs: 19 active+undersized, 13 peering, 13 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:26:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:45 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 34/633 objects degraded (5.371%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:45 smithi078 ceph-mon[91136]: pgmap v156: 161 pgs: 19 active+undersized, 13 peering, 13 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:26:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:45 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 34/633 objects degraded (5.371%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:45.977 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:45 smithi078 podman[121400]: 2024-09-20 20:26:45.624520242 +0000 UTC m=+0.772738353 container remove 151019d8508b4ad6434a61546a0f8c4a2c57e3abc0cbc667e87d923e4c2e90d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T20:26:45.977 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:45 smithi078 podman[121506]: 2024-09-20 20:26:45.736106647 +0000 UTC m=+0.016919012 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:26:46.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.264 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:45 smithi078 podman[121506]: 2024-09-20 20:26:45.975376117 +0000 UTC m=+0.256188470 container create ea01eae519aa896626182a640003c4344c938f472caa23dafc8f0dc6df1d5fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T20:26:46.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:46.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.113 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:46 smithi078 podman[121506]: 2024-09-20 20:26:46.832007846 +0000 UTC m=+1.112820214 container init ea01eae519aa896626182a640003c4344c938f472caa23dafc8f0dc6df1d5fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:26:47.113 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:46 smithi078 podman[121506]: 2024-09-20 20:26:46.835366971 +0000 UTC m=+1.116179323 container start ea01eae519aa896626182a640003c4344c938f472caa23dafc8f0dc6df1d5fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:26:47.113 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:46 smithi078 ceph-osd[121602]: -- 172.21.15.78:0/3334280070 <== mon.2 v2:172.21.15.78:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564eb903ba40 con 0x564eb9e0a400 2024-09-20T20:26:47.113 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:46 smithi078 bash[121506]: ea01eae519aa896626182a640003c4344c938f472caa23dafc8f0dc6df1d5fd1 2024-09-20T20:26:47.113 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:46 smithi078 systemd[1]: Started Ceph osd.2 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:26:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: pgmap v157: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-20T20:26:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.390 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: pgmap v157: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: pgmap v157: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-20T20:26:47.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:47.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:47.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:47.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:47.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:48.785 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:48 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[121539]: 2024-09-20T20:26:48.723+0000 7fa9dc001640 -1 Falling back to public interface 2024-09-20T20:26:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: pgmap v158: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: pgmap v158: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: pgmap v158: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:49.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:49.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:49 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:26:49.416+0000 7fcfc9d8b640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T20:26:50.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: pgmap v159: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T20:26:50.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T20:26:50.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:50 smithi146 ceph-mon[103278]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T20:26:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: pgmap v159: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T20:26:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[91136]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: pgmap v159: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T20:26:50.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-mon[99959]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T20:26:51.016 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:26:50] "GET /metrics HTTP/1.1" 200 38849 "" "Prometheus/2.43.0" 2024-09-20T20:26:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:51 smithi078 ceph-mon[91136]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:26:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:51 smithi078 ceph-mon[99959]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:26:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:51 smithi146 ceph-mon[103278]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:26:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:52 smithi078 ceph-mon[91136]: pgmap v160: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:52 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 58/633 objects degraded (9.163%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:52.239 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:52 smithi078 ceph-mon[99959]: pgmap v160: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:52.239 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:52 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 58/633 objects degraded (9.163%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:52 smithi146 ceph-mon[103278]: pgmap v160: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:52 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 58/633 objects degraded (9.163%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:52.536 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[121539]: 2024-09-20T20:26:52.289+0000 7fa9dc001640 -1 osd.2 125 log_to_monitors true 2024-09-20T20:26:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:53 smithi078 ceph-mon[91136]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T20:26:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:53 smithi078 ceph-mon[99959]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T20:26:53.286 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 20 20:26:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-2[121539]: 2024-09-20T20:26:53.157+0000 7fa9cfda3640 -1 osd.2 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:53 smithi146 ceph-mon[103278]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-20T20:26:54.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[99959]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T20:26:54.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[99959]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T20:26:54.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[99959]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:54.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:54.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[99959]: pgmap v162: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:54.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[91136]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T20:26:54.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[91136]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T20:26:54.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[91136]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:54.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:54.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:54 smithi078 ceph-mon[91136]: pgmap v162: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:54 smithi146 ceph-mon[103278]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T20:26:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:54 smithi146 ceph-mon[103278]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T20:26:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:54 smithi146 ceph-mon[103278]: from='osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:26:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:54 smithi146 ceph-mon[103278]: pgmap v162: 161 pgs: 29 active+undersized, 16 active+undersized+degraded, 116 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 548 B/s rd, 0 op/s; 58/633 objects degraded (9.163%) 2024-09-20T20:26:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:55 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:26:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:55 smithi146 ceph-mon[103278]: osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811] boot 2024-09-20T20:26:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:55 smithi146 ceph-mon[103278]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T20:26:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[91136]: osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811] boot 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[91136]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[99959]: osd.2 [v2:172.21.15.78:6818/3346652811,v1:172.21.15.78:6819/3346652811] boot 2024-09-20T20:26:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[99959]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T20:26:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-20T20:26:56.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:56 smithi078 ceph-mon[99959]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T20:26:56.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:56 smithi078 ceph-mon[99959]: pgmap v165: 161 pgs: 4 peering, 27 active+undersized, 14 active+undersized+degraded, 116 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:26:56.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:56 smithi078 ceph-mon[91136]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T20:26:56.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:56 smithi078 ceph-mon[91136]: pgmap v165: 161 pgs: 4 peering, 27 active+undersized, 14 active+undersized+degraded, 116 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:26:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:56 smithi146 ceph-mon[103278]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T20:26:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:56 smithi146 ceph-mon[103278]: pgmap v165: 161 pgs: 4 peering, 27 active+undersized, 14 active+undersized+degraded, 116 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 39/633 objects degraded (6.161%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: pgmap v166: 161 pgs: 4 peering, 23 active+undersized, 13 active+undersized+degraded, 121 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 39/633 objects degraded (6.161%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: pgmap v166: 161 pgs: 4 peering, 23 active+undersized, 13 active+undersized+degraded, 121 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 39/633 objects degraded (6.161%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: pgmap v166: 161 pgs: 4 peering, 23 active+undersized, 13 active+undersized+degraded, 121 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:58 smithi146 ceph-mon[103278]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi078 is in unknown state']) 2024-09-20T20:26:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:58 smithi078 ceph-mon[91136]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi078 is in unknown state']) 2024-09-20T20:26:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:58 smithi078 ceph-mon[99959]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi078 is in unknown state']) 2024-09-20T20:26:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:26:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:26:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:26:59 smithi146 ceph-mon[103278]: pgmap v167: 161 pgs: 4 peering, 13 active+undersized, 11 active+undersized+degraded, 133 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s; 32/633 objects degraded (5.055%) 2024-09-20T20:26:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:26:59 smithi078 ceph-mon[91136]: pgmap v167: 161 pgs: 4 peering, 13 active+undersized, 11 active+undersized+degraded, 133 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s; 32/633 objects degraded (5.055%) 2024-09-20T20:26:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:26:59 smithi078 ceph-mon[99959]: pgmap v167: 161 pgs: 4 peering, 13 active+undersized, 11 active+undersized+degraded, 133 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s; 32/633 objects degraded (5.055%) 2024-09-20T20:27:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:27:00] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.43.0" 2024-09-20T20:27:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:01 smithi146 ceph-mon[103278]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:27:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:01 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/633 objects degraded (5.055%), 11 pgs degraded) 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[91136]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/633 objects degraded (5.055%), 11 pgs degraded) 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[99959]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:02.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:01 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/633 objects degraded (5.055%), 11 pgs degraded) 2024-09-20T20:27:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:03 smithi146 ceph-mon[103278]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-09-20T20:27:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:03 smithi078 ceph-mon[99959]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-09-20T20:27:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:03 smithi078 ceph-mon[91136]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-09-20T20:27:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T20:27:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T20:27:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-20T20:27:05.321 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 17m - - 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 17m - - 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (11m) 9s ago 12m 25.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (12m) 5s ago 12m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5m) 9s ago 10m 61.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4m) 5s ago 16m 449M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:27:05.853 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5m) 9s ago 18m 514M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4m) 9s ago 18m 62.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2m) 5s ago 17m 41.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3m) 9s ago 17m 46.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (12m) 9s ago 12m 13.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (12m) 5s ago 12m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (117s) 9s ago 16m 46.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (57s) 9s ago 16m 45.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (20s) 9s ago 15m 41.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (15m) 9s ago 15m 66.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 0f4b1d92a0e2 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (15m) 5s ago 15m 71.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (14m) 5s ago 14m 65.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (14m) 5s ago 14m 57.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:27:05.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (13m) 5s ago 13m 69.4M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:27:05.855 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5m) 5s ago 13m 54.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:27:05.855 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (11m) 9s ago 11m 82.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:27:05.855 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (11m) 5s ago 11m 81.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:27:05.855 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (10m) 9s ago 10m 82.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:27:05.855 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (10m) 5s ago 10m 84.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:27:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:05 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T20:27:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:05 smithi146 ceph-mon[103278]: Upgrade: osd.3 is safe to restart 2024-09-20T20:27:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:05 smithi146 ceph-mon[103278]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:27:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:05 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T20:27:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:05 smithi078 ceph-mon[91136]: Upgrade: osd.3 is safe to restart 2024-09-20T20:27:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:05 smithi078 ceph-mon[91136]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:27:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:05 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T20:27:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:05 smithi078 ceph-mon[99959]: Upgrade: osd.3 is safe to restart 2024-09-20T20:27:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:05 smithi078 ceph-mon[99959]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:27:06.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:27:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:27:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:27:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:27:06.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 5, 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 9, 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:27:06.180 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "8/23 daemons upgraded", 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:27:06.452 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:27:06.773 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:06.773 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:06.773 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:06.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='client.64313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:06.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='client.44391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:06.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:06.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:06.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T20:27:06.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:06.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3037567807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='client.64313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='client.44391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3037567807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='client.64313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='client.44391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-20T20:27:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:07.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3037567807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:07.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[99959]: Upgrade: Updating osd.3 2024-09-20T20:27:07.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[99959]: Deploying daemon osd.3 on smithi078 2024-09-20T20:27:07.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[99959]: from='client.44409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[99959]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-09-20T20:27:07.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4164966630' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:07.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[91136]: Upgrade: Updating osd.3 2024-09-20T20:27:07.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[91136]: Deploying daemon osd.3 on smithi078 2024-09-20T20:27:07.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[91136]: from='client.44409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[91136]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-09-20T20:27:07.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4164966630' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:07.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:07 smithi146 ceph-mon[103278]: Upgrade: Updating osd.3 2024-09-20T20:27:07.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:07 smithi146 ceph-mon[103278]: Deploying daemon osd.3 on smithi078 2024-09-20T20:27:07.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:07 smithi146 ceph-mon[103278]: from='client.44409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:07.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:07 smithi146 ceph-mon[103278]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-09-20T20:27:07.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4164966630' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:09.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:09 smithi078 ceph-mon[91136]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:09.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:09 smithi078 ceph-mon[99959]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:09.950 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:09 smithi078 systemd[1]: Stopping Ceph osd.3 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:27:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:09 smithi146 ceph-mon[103278]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:27:10.286 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[44763]: 2024-09-20T20:27:09.947+0000 7fb32db2f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:27:10.286 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[44763]: 2024-09-20T20:27:09.947+0000 7fb32db2f640 -1 osd.3 130 *** Got signal Terminated *** 2024-09-20T20:27:10.286 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[44763]: 2024-09-20T20:27:09.947+0000 7fb32db2f640 -1 osd.3 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:27:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:10 smithi146 ceph-mon[103278]: osd.3 marked itself down and dead 2024-09-20T20:27:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:27:10] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.43.0" 2024-09-20T20:27:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:10 smithi078 ceph-mon[99959]: osd.3 marked itself down and dead 2024-09-20T20:27:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:10 smithi078 ceph-mon[91136]: osd.3 marked itself down and dead 2024-09-20T20:27:11.339 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:11 smithi078 podman[124049]: 2024-09-20 20:27:11.09149506 +0000 UTC m=+1.272609107 container died 0f4b1d92a0e21f60efa7969aba3117a8f31a7e7d36a5bf20e6413b684b4f0ace (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T20:27:11.718 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:11 smithi078 podman[124049]: 2024-09-20 20:27:11.523078943 +0000 UTC m=+1.704192976 container cleanup 0f4b1d92a0e21f60efa7969aba3117a8f31a7e7d36a5bf20e6413b684b4f0ace (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-20T20:27:11.718 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:11 smithi078 bash[124049]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3 2024-09-20T20:27:11.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:11 smithi078 ceph-mon[99959]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:11.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:11 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:27:11.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:11 smithi078 ceph-mon[99959]: osdmap e131: 8 total, 7 up, 8 in 2024-09-20T20:27:11.992 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:11 smithi078 podman[124064]: 2024-09-20 20:27:11.716112323 +0000 UTC m=+0.616849588 container remove 0f4b1d92a0e21f60efa7969aba3117a8f31a7e7d36a5bf20e6413b684b4f0ace (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3, ceph=True, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T20:27:11.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:11 smithi078 ceph-mon[91136]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:11.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:11 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:27:11.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:11 smithi078 ceph-mon[91136]: osdmap e131: 8 total, 7 up, 8 in 2024-09-20T20:27:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:11 smithi146 ceph-mon[103278]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:27:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:11 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:27:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:11 smithi146 ceph-mon[103278]: osdmap e131: 8 total, 7 up, 8 in 2024-09-20T20:27:12.286 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:12 smithi078 podman[124265]: 2024-09-20 20:27:12.191930859 +0000 UTC m=+0.111491967 container create e2d24dd87b09a84b63ae76e0264fe1e0e05298c4adb542cf862bff78d6ef6fb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:27:12.286 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:12 smithi078 podman[124265]: 2024-09-20 20:27:12.099472779 +0000 UTC m=+0.019033899 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:27:12.621 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:12 smithi078 podman[124265]: 2024-09-20 20:27:12.427972664 +0000 UTC m=+0.347533792 container init e2d24dd87b09a84b63ae76e0264fe1e0e05298c4adb542cf862bff78d6ef6fb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:27:12.621 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:12 smithi078 podman[124265]: 2024-09-20 20:27:12.431409575 +0000 UTC m=+0.350970692 container start e2d24dd87b09a84b63ae76e0264fe1e0e05298c4adb542cf862bff78d6ef6fb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:27:12.621 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:12 smithi078 podman[124265]: 2024-09-20 20:27:12.527836829 +0000 UTC m=+0.447397950 container attach e2d24dd87b09a84b63ae76e0264fe1e0e05298c4adb542cf862bff78d6ef6fb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-20T20:27:12.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:12 smithi078 ceph-mon[99959]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T20:27:12.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:12 smithi078 ceph-mon[91136]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T20:27:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:12 smithi146 ceph-mon[103278]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T20:27:13.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:13 smithi078 podman[124265]: 2024-09-20 20:27:13.248772383 +0000 UTC m=+1.168333516 container died e2d24dd87b09a84b63ae76e0264fe1e0e05298c4adb542cf862bff78d6ef6fb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T20:27:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:13 smithi078 ceph-mon[99959]: pgmap v176: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:27:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:13.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:13 smithi078 ceph-mon[91136]: pgmap v176: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:27:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:13.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:14.251 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:13 smithi078 podman[124327]: 2024-09-20 20:27:13.785765169 +0000 UTC m=+1.158096932 container remove e2d24dd87b09a84b63ae76e0264fe1e0e05298c4adb542cf862bff78d6ef6fb5 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-20T20:27:14.251 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:13 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-20T20:27:14.251 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:13 smithi078 systemd[1]: Stopped Ceph osd.3 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:27:14.251 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:13 smithi078 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.3.service: Consumed 9.703s CPU time. 2024-09-20T20:27:14.251 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 systemd[1]: Starting Ceph osd.3 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:27:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:13 smithi146 ceph-mon[103278]: pgmap v176: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T20:27:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:14.513 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 podman[124535]: 2024-09-20 20:27:14.153306758 +0000 UTC m=+0.018409426 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:27:14.514 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 podman[124535]: 2024-09-20 20:27:14.270516323 +0000 UTC m=+0.135618988 container create ce7db64d58abef3809263623f39f50540e8e261e93ce81855bcd1eccde64c2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T20:27:14.514 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 podman[124535]: 2024-09-20 20:27:14.512096496 +0000 UTC m=+0.377199159 container init ce7db64d58abef3809263623f39f50540e8e261e93ce81855bcd1eccde64c2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-20T20:27:14.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 podman[124535]: 2024-09-20 20:27:14.515549934 +0000 UTC m=+0.380652593 container start ce7db64d58abef3809263623f39f50540e8e261e93ce81855bcd1eccde64c2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T20:27:14.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 podman[124535]: 2024-09-20 20:27:14.638372123 +0000 UTC m=+0.503474787 container attach ce7db64d58abef3809263623f39f50540e8e261e93ce81855bcd1eccde64c2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-20T20:27:14.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:14.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 bash[124535]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:14.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:14.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:14 smithi078 bash[124535]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:15.094 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:14 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 39/633 objects degraded (6.161%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:15.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:14 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 39/633 objects degraded (6.161%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:14 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 39/633 objects degraded (6.161%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:15.413 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T20:27:15.413 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T20:27:15.413 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:27:15.413 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:27:15.413 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:27:15.413 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:27:15.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate[124579]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 bash[124535]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T20:27:15.787 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:15 smithi078 podman[124535]: 2024-09-20 20:27:15.544887689 +0000 UTC m=+1.409990351 container died ce7db64d58abef3809263623f39f50540e8e261e93ce81855bcd1eccde64c2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:27:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:15 smithi146 ceph-mon[103278]: pgmap v177: 161 pgs: 26 active+undersized, 12 stale+active+clean, 13 active+undersized+degraded, 110 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 39/633 objects degraded (6.161%) 2024-09-20T20:27:16.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-mon[99959]: pgmap v177: 161 pgs: 26 active+undersized, 12 stale+active+clean, 13 active+undersized+degraded, 110 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 39/633 objects degraded (6.161%) 2024-09-20T20:27:16.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:15 smithi078 ceph-mon[91136]: pgmap v177: 161 pgs: 26 active+undersized, 12 stale+active+clean, 13 active+undersized+degraded, 110 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 39/633 objects degraded (6.161%) 2024-09-20T20:27:16.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:16 smithi078 podman[124832]: 2024-09-20 20:27:16.257265939 +0000 UTC m=+0.810243890 container remove ce7db64d58abef3809263623f39f50540e8e261e93ce81855bcd1eccde64c2f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T20:27:16.892 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:16 smithi078 podman[124935]: 2024-09-20 20:27:16.555749582 +0000 UTC m=+0.195833206 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:27:16.893 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:16 smithi078 podman[124935]: 2024-09-20 20:27:16.785277349 +0000 UTC m=+0.425360960 container create b2d0b8b0eb4edbe06f0ee1145354de63e0b0bf7883a8e8928450bde819f936b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-20T20:27:17.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:17 smithi078 podman[124935]: 2024-09-20 20:27:17.340214419 +0000 UTC m=+0.980298036 container init b2d0b8b0eb4edbe06f0ee1145354de63e0b0bf7883a8e8928450bde819f936b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-20T20:27:17.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:17 smithi078 podman[124935]: 2024-09-20 20:27:17.343928745 +0000 UTC m=+0.984012356 container start b2d0b8b0eb4edbe06f0ee1145354de63e0b0bf7883a8e8928450bde819f936b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T20:27:17.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-osd[125047]: -- 172.21.15.78:0/1953666739 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56339c5bfa40 con 0x56339c59dc00 2024-09-20T20:27:17.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:17 smithi078 bash[124935]: b2d0b8b0eb4edbe06f0ee1145354de63e0b0bf7883a8e8928450bde819f936b9 2024-09-20T20:27:17.786 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:17 smithi078 systemd[1]: Started Ceph osd.3 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:27:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: pgmap v178: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: pgmap v178: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: pgmap v178: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:18.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:18.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:18.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.036 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[124997]: 2024-09-20T20:27:18.732+0000 7fc473ac9640 -1 Falling back to public interface 2024-09-20T20:27:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:19.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:19.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:19.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:19.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:27:19.447+0000 7fcfc9d8b640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:27:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: pgmap v179: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:20.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: pgmap v180: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.277 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:27:20.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:27:20.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:27:20.278 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:19 smithi146 ceph-mon[103278]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.3 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:27:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: pgmap v179: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: pgmap v179: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: pgmap v180: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:20.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:27:20.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:27:20.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:27:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[91136]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.3 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:27:20.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:20.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:20.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:20.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: pgmap v180: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:20.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:20.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:20.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:20.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:27:20.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:27:20.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:27:20.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:19 smithi078 ceph-mon[99959]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.3 on smithi078 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:27:20.762 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:27:20] "GET /metrics HTTP/1.1" 200 38846 "" "Prometheus/2.43.0" 2024-09-20T20:27:22.086 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[124997]: 2024-09-20T20:27:22.049+0000 7fc473ac9640 -1 osd.3 130 log_to_monitors true 2024-09-20T20:27:22.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[91136]: pgmap v181: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:22.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[91136]: pgmap v182: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:22.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 78/633 objects degraded (12.322%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:22.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[99959]: pgmap v181: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:22.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[99959]: pgmap v182: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:22.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:21 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 78/633 objects degraded (12.322%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:21 smithi146 ceph-mon[103278]: pgmap v181: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:21 smithi146 ceph-mon[103278]: pgmap v182: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 78/633 objects degraded (12.322%) 2024-09-20T20:27:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:21 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 78/633 objects degraded (12.322%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:23.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:22 smithi078 ceph-mon[99959]: from='osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T20:27:23.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:22 smithi078 ceph-mon[99959]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T20:27:23.028 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 20 20:27:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-3[124997]: 2024-09-20T20:27:22.864+0000 7fc46b873640 -1 osd.3 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:27:23.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:22 smithi078 ceph-mon[91136]: from='osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T20:27:23.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:22 smithi078 ceph-mon[91136]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T20:27:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:22 smithi146 ceph-mon[103278]: from='osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T20:27:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:22 smithi146 ceph-mon[103278]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-20T20:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[91136]: pgmap v183: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 369 B/s rd, 0 op/s; 78/633 objects degraded (12.322%) 2024-09-20T20:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[91136]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T20:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[91136]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T20:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[91136]: from='osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[91136]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:27:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[99959]: pgmap v183: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 369 B/s rd, 0 op/s; 78/633 objects degraded (12.322%) 2024-09-20T20:27:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[99959]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T20:27:24.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[99959]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T20:27:24.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[99959]: from='osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:27:24.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:23 smithi078 ceph-mon[99959]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:27:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:23 smithi146 ceph-mon[103278]: pgmap v183: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 369 B/s rd, 0 op/s; 78/633 objects degraded (12.322%) 2024-09-20T20:27:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:23 smithi146 ceph-mon[103278]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T20:27:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:23 smithi146 ceph-mon[103278]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T20:27:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:23 smithi146 ceph-mon[103278]: from='osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:27:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:23 smithi146 ceph-mon[103278]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]} : dispatch 2024-09-20T20:27:25.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[91136]: osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327] boot 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[91136]: osdmap e134: 8 total, 8 up, 8 in 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[99959]: osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327] boot 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[99959]: osdmap e134: 8 total, 8 up, 8 in 2024-09-20T20:27:25.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T20:27:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:24 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:27:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:24 smithi146 ceph-mon[103278]: osd.3 [v2:172.21.15.78:6826/3113282327,v1:172.21.15.78:6827/3113282327] boot 2024-09-20T20:27:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:24 smithi146 ceph-mon[103278]: osdmap e134: 8 total, 8 up, 8 in 2024-09-20T20:27:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-20T20:27:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:25 smithi146 ceph-mon[103278]: pgmap v186: 161 pgs: 18 peering, 32 active+undersized, 16 active+undersized+degraded, 95 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 57/633 objects degraded (9.005%) 2024-09-20T20:27:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:25 smithi146 ceph-mon[103278]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T20:27:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:25 smithi078 ceph-mon[91136]: pgmap v186: 161 pgs: 18 peering, 32 active+undersized, 16 active+undersized+degraded, 95 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 57/633 objects degraded (9.005%) 2024-09-20T20:27:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:25 smithi078 ceph-mon[91136]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T20:27:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:25 smithi078 ceph-mon[99959]: pgmap v186: 161 pgs: 18 peering, 32 active+undersized, 16 active+undersized+degraded, 95 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 57/633 objects degraded (9.005%) 2024-09-20T20:27:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:25 smithi078 ceph-mon[99959]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T20:27:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:26 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 57/633 objects degraded (9.005%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:27.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:26 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 57/633 objects degraded (9.005%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:27.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:26 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 57/633 objects degraded (9.005%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: pgmap v188: 161 pgs: 18 peering, 23 active+undersized, 10 active+undersized+degraded, 110 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: pgmap v188: 161 pgs: 18 peering, 23 active+undersized, 10 active+undersized+degraded, 110 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: pgmap v188: 161 pgs: 18 peering, 23 active+undersized, 10 active+undersized+degraded, 110 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:28 smithi146 ceph-mon[103278]: pgmap v189: 161 pgs: 18 peering, 23 active+undersized, 10 active+undersized+degraded, 110 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:27:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:28 smithi146 ceph-mon[103278]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.3 on smithi078 is in unknown state']) 2024-09-20T20:27:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[91136]: pgmap v189: 161 pgs: 18 peering, 23 active+undersized, 10 active+undersized+degraded, 110 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[91136]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.3 on smithi078 is in unknown state']) 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[99959]: pgmap v189: 161 pgs: 18 peering, 23 active+undersized, 10 active+undersized+degraded, 110 active+clean; 585 KiB data, 260 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 34/633 objects degraded (5.371%) 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[99959]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.3 on smithi078 is in unknown state']) 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:29 smithi146 ceph-mon[103278]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:29 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/633 objects degraded (5.371%), 10 pgs degraded) 2024-09-20T20:27:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:29 smithi078 ceph-mon[91136]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:29 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/633 objects degraded (5.371%), 10 pgs degraded) 2024-09-20T20:27:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:29 smithi078 ceph-mon[99959]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:29 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/633 objects degraded (5.371%), 10 pgs degraded) 2024-09-20T20:27:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:27:30] "GET /metrics HTTP/1.1" 200 38848 "" "Prometheus/2.43.0" 2024-09-20T20:27:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:32 smithi146 ceph-mon[103278]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:32 smithi078 ceph-mon[91136]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:32 smithi078 ceph-mon[99959]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:34 smithi146 ceph-mon[103278]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:34 smithi078 ceph-mon[99959]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:34 smithi078 ceph-mon[91136]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:27:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:35 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:27:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:35 smithi146 ceph-mon[103278]: Upgrade: osd.4 is safe to restart 2024-09-20T20:27:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:27:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:35 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:27:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:35 smithi078 ceph-mon[99959]: Upgrade: osd.4 is safe to restart 2024-09-20T20:27:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-20T20:27:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:35 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T20:27:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:35 smithi078 ceph-mon[91136]: Upgrade: osd.4 is safe to restart 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[99959]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[99959]: Upgrade: Updating osd.4 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[99959]: Deploying daemon osd.4 on smithi146 2024-09-20T20:27:37.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[91136]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:37.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[91136]: Upgrade: Updating osd.4 2024-09-20T20:27:37.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:37.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:27:37.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:37.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:36 smithi078 ceph-mon[91136]: Deploying daemon osd.4 on smithi146 2024-09-20T20:27:37.072 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:27:37.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:36 smithi146 ceph-mon[103278]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:37.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:36 smithi146 ceph-mon[103278]: Upgrade: Updating osd.4 2024-09-20T20:27:37.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:37.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-20T20:27:37.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:37.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:36 smithi146 ceph-mon[103278]: Deploying daemon osd.4 on smithi146 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 18m - - 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 18m - - 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (12m) 10s ago 13m 25.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (13m) 16s ago 13m 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5m) 10s ago 11m 61.7M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4m) 16s ago 17m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6m) 10s ago 19m 516M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:27:37.597 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4m) 10s ago 19m 65.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3m) 16s ago 17m 43.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3m) 10s ago 17m 49.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (12m) 10s ago 12m 13.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (12m) 16s ago 12m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2m) 10s ago 17m 48.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (89s) 10s ago 16m 46.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (51s) 10s ago 16m 43.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (21s) 10s ago 15m 43.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (15m) 16s ago 15m 72.1M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 7d356e7eb5e7 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (15m) 16s ago 15m 65.7M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (14m) 16s ago 14m 57.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (14m) 16s ago 14m 70.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5m) 16s ago 13m 51.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:27:37.598 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (11m) 10s ago 11m 82.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:27:37.599 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (11m) 16s ago 11m 82.1M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:27:37.599 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (11m) 10s ago 11m 82.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:27:37.599 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (11m) 16s ago 11m 85.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:27:37.918 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4, 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:27:37.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:27:37.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:27:37.920 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:27:37.920 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 8, 2024-09-20T20:27:37.920 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 9 2024-09-20T20:27:37.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:27:37.920 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "9/23 daemons upgraded", 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:27:38.197 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:27:38.510 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:38.510 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:38.510 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:27:38.999 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:38 smithi146 systemd[1]: Stopping Ceph osd.4 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:27:38.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:38.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: from='client.64355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:38.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: from='client.64361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: from='client.64367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2997004440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:39.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:39.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/831396783' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: from='client.64355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: from='client.64361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: from='client.64367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2997004440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/831396783' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: from='client.64355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: from='client.64361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: from='client.64367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:39.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2997004440' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:27:39.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:39.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/831396783' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:27:39.267 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[31825]: 2024-09-20T20:27:38.996+0000 7fe29d796640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:27:39.267 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[31825]: 2024-09-20T20:27:38.996+0000 7fe29d796640 -1 osd.4 135 *** Got signal Terminated *** 2024-09-20T20:27:39.267 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:38 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[31825]: 2024-09-20T20:27:38.996+0000 7fe29d796640 -1 osd.4 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:27:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:39 smithi146 ceph-mon[103278]: from='client.44451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:39 smithi146 ceph-mon[103278]: osd.4 marked itself down and dead 2024-09-20T20:27:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:39 smithi078 ceph-mon[91136]: from='client.44451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:39 smithi078 ceph-mon[91136]: osd.4 marked itself down and dead 2024-09-20T20:27:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:39 smithi078 ceph-mon[99959]: from='client.44451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:27:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:39 smithi078 ceph-mon[99959]: osd.4 marked itself down and dead 2024-09-20T20:27:40.728 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:27:40] "GET /metrics HTTP/1.1" 200 38848 "" "Prometheus/2.43.0" 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[99959]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[99959]: osdmap e136: 8 total, 7 up, 8 in 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[91136]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:40 smithi078 ceph-mon[91136]: osdmap e136: 8 total, 7 up, 8 in 2024-09-20T20:27:41.152 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:40 smithi146 podman[112911]: 2024-09-20 20:27:40.862084372 +0000 UTC m=+1.983577910 container died 7d356e7eb5e70c146e2f2c33aa272bbdf0eec7a93f822f82bc264495d79dc852 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T20:27:41.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:40 smithi146 ceph-mon[103278]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-20T20:27:41.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:41.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:40 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:27:41.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:40 smithi146 ceph-mon[103278]: osdmap e136: 8 total, 7 up, 8 in 2024-09-20T20:27:41.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:41 smithi146 podman[112911]: 2024-09-20 20:27:41.358838258 +0000 UTC m=+2.480331791 container cleanup 7d356e7eb5e70c146e2f2c33aa272bbdf0eec7a93f822f82bc264495d79dc852 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T20:27:41.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:41 smithi146 bash[112911]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4 2024-09-20T20:27:41.834 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:41 smithi146 podman[113107]: 2024-09-20 20:27:41.535513462 +0000 UTC m=+1.206664845 container remove 7d356e7eb5e70c146e2f2c33aa272bbdf0eec7a93f822f82bc264495d79dc852 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-0b2cc4d, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:27:42.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:41 smithi146 ceph-mon[103278]: osdmap e137: 8 total, 7 up, 8 in 2024-09-20T20:27:42.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.085 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:41 smithi146 podman[113289]: 2024-09-20 20:27:41.87234188 +0000 UTC m=+0.017402405 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:27:42.085 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:41 smithi146 podman[113289]: 2024-09-20 20:27:41.969461629 +0000 UTC m=+0.114522145 container create 86e2f4e48afc377e04250f1e5114bd685cf28c420f7caa82193af6f20e91a12d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[91136]: osdmap e137: 8 total, 7 up, 8 in 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[99959]: osdmap e137: 8 total, 7 up, 8 in 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:42.387 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:42 smithi146 podman[113289]: 2024-09-20 20:27:42.189094512 +0000 UTC m=+0.334155034 container init 86e2f4e48afc377e04250f1e5114bd685cf28c420f7caa82193af6f20e91a12d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T20:27:42.387 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:42 smithi146 podman[113289]: 2024-09-20 20:27:42.192843335 +0000 UTC m=+0.337903852 container start 86e2f4e48afc377e04250f1e5114bd685cf28c420f7caa82193af6f20e91a12d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:27:42.387 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:42 smithi146 podman[113289]: 2024-09-20 20:27:42.288510169 +0000 UTC m=+0.433570691 container attach 86e2f4e48afc377e04250f1e5114bd685cf28c420f7caa82193af6f20e91a12d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:27:42.766 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:42 smithi146 podman[113289]: 2024-09-20 20:27:42.564914456 +0000 UTC m=+0.709974970 container died 86e2f4e48afc377e04250f1e5114bd685cf28c420f7caa82193af6f20e91a12d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:27:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:42 smithi146 ceph-mon[103278]: pgmap v198: 161 pgs: 20 peering, 19 stale+active+clean, 122 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:27:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:42 smithi146 ceph-mon[103278]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T20:27:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:42 smithi078 ceph-mon[91136]: pgmap v198: 161 pgs: 20 peering, 19 stale+active+clean, 122 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:27:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:42 smithi078 ceph-mon[91136]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T20:27:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:42 smithi078 ceph-mon[99959]: pgmap v198: 161 pgs: 20 peering, 19 stale+active+clean, 122 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T20:27:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:42 smithi078 ceph-mon[99959]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-20T20:27:44.018 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:43 smithi146 podman[113355]: 2024-09-20 20:27:43.72080354 +0000 UTC m=+1.324054241 container remove 86e2f4e48afc377e04250f1e5114bd685cf28c420f7caa82193af6f20e91a12d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T20:27:44.018 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:43 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.4.service: Deactivated successfully. 2024-09-20T20:27:44.018 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:43 smithi146 systemd[1]: Stopped Ceph osd.4 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:27:44.018 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:43 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.4.service: Consumed 10.044s CPU time. 2024-09-20T20:27:44.018 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:43 smithi146 systemd[1]: Starting Ceph osd.4 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:27:44.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:44 smithi146 ceph-mon[103278]: pgmap v199: 161 pgs: 8 active+undersized, 20 peering, 13 stale+active+clean, 2 active+undersized+degraded, 118 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 5/633 objects degraded (0.790%) 2024-09-20T20:27:44.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:44.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:44.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:44 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 5/633 objects degraded (0.790%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[91136]: pgmap v199: 161 pgs: 8 active+undersized, 20 peering, 13 stale+active+clean, 2 active+undersized+degraded, 118 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 5/633 objects degraded (0.790%) 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 5/633 objects degraded (0.790%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[99959]: pgmap v199: 161 pgs: 8 active+undersized, 20 peering, 13 stale+active+clean, 2 active+undersized+degraded, 118 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 5/633 objects degraded (0.790%) 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:44.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:44 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 5/633 objects degraded (0.790%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:45.071 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:44 smithi146 podman[113552]: 2024-09-20 20:27:44.716152325 +0000 UTC m=+0.645390598 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:27:45.071 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:44 smithi146 podman[113552]: 2024-09-20 20:27:44.81707407 +0000 UTC m=+0.746312334 container create d9b9a2b6fd90f7b5dc6d209a867b4aee6eb5490b9aec6bc7058c2ba2d24fcbed (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T20:27:45.364 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 podman[113552]: 2024-09-20 20:27:45.263021967 +0000 UTC m=+1.192260237 container init d9b9a2b6fd90f7b5dc6d209a867b4aee6eb5490b9aec6bc7058c2ba2d24fcbed (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:27:45.364 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 podman[113552]: 2024-09-20 20:27:45.266506411 +0000 UTC m=+1.195744681 container start d9b9a2b6fd90f7b5dc6d209a867b4aee6eb5490b9aec6bc7058c2ba2d24fcbed (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T20:27:45.768 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 podman[113552]: 2024-09-20 20:27:45.362905819 +0000 UTC m=+1.292144098 container attach d9b9a2b6fd90f7b5dc6d209a867b4aee6eb5490b9aec6bc7058c2ba2d24fcbed (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T20:27:45.768 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:45.768 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 bash[113552]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:45.768 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:45.768 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 bash[113552]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:27:46.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:46 smithi146 ceph-mon[103278]: pgmap v200: 161 pgs: 37 active+undersized, 9 peering, 23 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:27:46.151 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:27:46.151 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 bash[113552]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:27:46.151 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:27:46.151 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:45 smithi146 bash[113552]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T20:27:46.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:27:46.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 bash[113552]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T20:27:46.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:27:46.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 bash[113552]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T20:27:46.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T20:27:46.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 bash[113552]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T20:27:46.518 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:27:46.518 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 bash[113552]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T20:27:46.518 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate[113626]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T20:27:46.518 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 bash[113552]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T20:27:46.518 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 podman[113552]: 2024-09-20 20:27:46.269467605 +0000 UTC m=+2.198705867 container died d9b9a2b6fd90f7b5dc6d209a867b4aee6eb5490b9aec6bc7058c2ba2d24fcbed (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T20:27:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:46 smithi078 ceph-mon[91136]: pgmap v200: 161 pgs: 37 active+undersized, 9 peering, 23 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:27:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:46 smithi078 ceph-mon[99959]: pgmap v200: 161 pgs: 37 active+undersized, 9 peering, 23 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:27:47.162 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:46 smithi146 podman[113862]: 2024-09-20 20:27:46.80323522 +0000 UTC m=+0.606489049 container remove d9b9a2b6fd90f7b5dc6d209a867b4aee6eb5490b9aec6bc7058c2ba2d24fcbed (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-20T20:27:47.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:47 smithi146 ceph-mon[103278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-20T20:27:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:47 smithi078 ceph-mon[91136]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-20T20:27:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:47 smithi078 ceph-mon[99959]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-20T20:27:48.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 podman[114014]: 2024-09-20 20:27:48.311284695 +0000 UTC m=+0.015965926 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:27:48.517 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 podman[114014]: 2024-09-20 20:27:48.412075211 +0000 UTC m=+0.116756436 container create f18ce9968a7dd4c02e13fa4fb13bd5e6e0e387e2ec87ca3774cbd9534671e177 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: pgmap v201: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 109/633 objects degraded (17.220%) 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.780 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 podman[114014]: 2024-09-20 20:27:48.622571125 +0000 UTC m=+0.327252352 container init f18ce9968a7dd4c02e13fa4fb13bd5e6e0e387e2ec87ca3774cbd9534671e177 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:27:48.780 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 podman[114014]: 2024-09-20 20:27:48.625683481 +0000 UTC m=+0.330364717 container start f18ce9968a7dd4c02e13fa4fb13bd5e6e0e387e2ec87ca3774cbd9534671e177 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T20:27:48.781 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 ceph-osd[114032]: -- 172.21.15.146:0/3410742980 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad775e1a40 con 0x55ad783b0400 2024-09-20T20:27:48.781 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 bash[114014]: f18ce9968a7dd4c02e13fa4fb13bd5e6e0e387e2ec87ca3774cbd9534671e177 2024-09-20T20:27:48.781 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:48 smithi146 systemd[1]: Started Ceph osd.4 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:27:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: pgmap v201: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 109/633 objects degraded (17.220%) 2024-09-20T20:27:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: pgmap v201: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 109/633 objects degraded (17.220%) 2024-09-20T20:27:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:48.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: pgmap v202: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 503 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: pgmap v203: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: pgmap v202: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 503 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: pgmap v203: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: pgmap v202: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 503 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: pgmap v203: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.649 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[114028]: 2024-09-20T20:27:50.536+0000 7f3fc481d640 -1 Falling back to public interface 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.905 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:50.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:50.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:50.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:27:50] "GET /metrics HTTP/1.1" 200 38841 "" "Prometheus/2.43.0" 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.048 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: pgmap v204: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 109/633 objects degraded (17.220%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:51.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: pgmap v204: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: pgmap v204: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 109/633 objects degraded (17.220%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 109/633 objects degraded (17.220%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:52.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:53.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: pgmap v205: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 314 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: pgmap v205: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 314 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: pgmap v205: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 314 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:54.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:54.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:54.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:54.518 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[114028]: 2024-09-20T20:27:54.152+0000 7f3fc481d640 -1 osd.4 135 log_to_monitors true 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:27:55.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.516 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-4[114028]: 2024-09-20T20:27:55.140+0000 7f3fb85bf640 -1 osd.4 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:27:55.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: pgmap v206: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: pgmap v206: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: osdmap e138: 8 total, 7 up, 8 in 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:27:55.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: pgmap v206: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: osdmap e138: 8 total, 7 up, 8 in 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:55.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: osdmap e138: 8 total, 7 up, 8 in 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:56.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708] boot 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708] boot 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: osd.4 [v2:172.21.15.146:6800/2277474708,v1:172.21.15.146:6801/2277474708] boot 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T20:27:57.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: pgmap v209: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:57.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:57.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:57.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:57.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: pgmap v209: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: pgmap v209: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 109/633 objects degraded (17.220%) 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:58.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:27:59.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.118 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:27:59.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: pgmap v211: 161 pgs: 35 active+undersized, 27 active+undersized+degraded, 99 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 105/633 objects degraded (16.588%) 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:27:59 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 105/633 objects degraded (16.588%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: pgmap v211: 161 pgs: 35 active+undersized, 27 active+undersized+degraded, 99 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 105/633 objects degraded (16.588%) 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: pgmap v211: 161 pgs: 35 active+undersized, 27 active+undersized+degraded, 99 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 105/633 objects degraded (16.588%) 2024-09-20T20:28:00.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 105/633 objects degraded (16.588%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:00.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:27:59 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 105/633 objects degraded (16.588%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:28:00] "GET /metrics HTTP/1.1" 200 38697 "" "Prometheus/2.43.0" 2024-09-20T20:28:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:01.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:02.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:01 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 105/633 objects degraded (16.588%), 27 pgs degraded) 2024-09-20T20:28:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 105/633 objects degraded (16.588%), 27 pgs degraded) 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:02.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:02.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:02.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:02.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:01 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 105/633 objects degraded (16.588%), 27 pgs degraded) 2024-09-20T20:28:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.024 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.027 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.027 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.027 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.027 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:03.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-09-20T20:28:04.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-09-20T20:28:04.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:04.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.307 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.311 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:04.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:04.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:04.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:04.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:05.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:06.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:07.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: Upgrade: osd.5 is safe to restart 2024-09-20T20:28:07.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: Upgrade: osd.5 is safe to restart 2024-09-20T20:28:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: Upgrade: osd.5 is safe to restart 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:08.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:28:08.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:08.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:28:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:08.801 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:28:08.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:08 smithi146 ceph-mon[103278]: Upgrade: Updating osd.5 2024-09-20T20:28:08.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:08 smithi146 ceph-mon[103278]: Deploying daemon osd.5 on smithi146 2024-09-20T20:28:08.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:09.120 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:08 smithi078 ceph-mon[99959]: Upgrade: Updating osd.5 2024-09-20T20:28:09.120 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:08 smithi078 ceph-mon[99959]: Deploying daemon osd.5 on smithi146 2024-09-20T20:28:09.120 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:09.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:08 smithi078 ceph-mon[91136]: Upgrade: Updating osd.5 2024-09-20T20:28:09.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:08 smithi078 ceph-mon[91136]: Deploying daemon osd.5 on smithi146 2024-09-20T20:28:09.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:09.334 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:28:09.334 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 18m - - 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 18m - - 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (12m) 1s ago 13m 25.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (13m) 3s ago 13m 87.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6m) 1s ago 11m 61.9M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5m) 3s ago 17m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7m) 1s ago 19m 518M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5m) 1s ago 19m 69.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3m) 3s ago 18m 46.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:28:09.335 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4m) 1s ago 18m 54.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (13m) 1s ago 13m 13.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (13m) 3s ago 13m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3m) 1s ago 17m 49.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2m) 1s ago 17m 47.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (83s) 1s ago 16m 45.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (52s) 1s ago 16m 46.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (21s) 3s ago 16m 47.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (15m) 3s ago 15m 66.6M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 b64a73dd22e7 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (15m) 3s ago 15m 58.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (14m) 3s ago 14m 71.9M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6m) 3s ago 14m 53.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:28:09.336 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (12m) 1s ago 12m 82.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:28:09.337 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (12m) 3s ago 12m 82.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:28:09.337 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (12m) 1s ago 12m 83.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:28:09.337 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (11m) 3s ago 11m 85.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 3, 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 5 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 7, 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 10 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:28:09.645 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:28:09.917 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "10/23 daemons upgraded", 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:28:09.918 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:28:09.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:09 smithi078 ceph-mon[99959]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:09.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:09 smithi078 ceph-mon[99959]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:28:09.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2780436132' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:09.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:09 smithi078 ceph-mon[91136]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:09.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:09 smithi078 ceph-mon[91136]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:28:09.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2780436132' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:10.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:09 smithi146 ceph-mon[103278]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:10.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:09 smithi146 ceph-mon[103278]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:28:10.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2780436132' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:10.230 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:10.230 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:10.230 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:10.865 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:28:10] "GET /metrics HTTP/1.1" 200 38697 "" "Prometheus/2.43.0" 2024-09-20T20:28:11.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:10 smithi146 ceph-mon[103278]: from='client.44475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:11.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:11.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:10 smithi146 ceph-mon[103278]: from='client.64409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:11.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1189173043' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[99959]: from='client.44475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[99959]: from='client.64409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1189173043' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[91136]: from='client.44475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[91136]: from='client.64409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1189173043' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:11.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:11 smithi146 systemd[1]: Stopping Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:28:11.866 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:11 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[35277]: 2024-09-20T20:28:11.607+0000 7f485fb40640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:28:11.867 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:11 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[35277]: 2024-09-20T20:28:11.607+0000 7f485fb40640 -1 osd.5 140 *** Got signal Terminated *** 2024-09-20T20:28:11.867 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:11 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[35277]: 2024-09-20T20:28:11.607+0000 7f485fb40640 -1 osd.5 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:28:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:11 smithi078 ceph-mon[91136]: from='client.44493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:11 smithi078 ceph-mon[91136]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:11 smithi078 ceph-mon[91136]: osd.5 marked itself down and dead 2024-09-20T20:28:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:11 smithi078 ceph-mon[99959]: from='client.44493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:11 smithi078 ceph-mon[99959]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:12.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:11 smithi078 ceph-mon[99959]: osd.5 marked itself down and dead 2024-09-20T20:28:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:11 smithi146 ceph-mon[103278]: from='client.44493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:11 smithi146 ceph-mon[103278]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:11 smithi146 ceph-mon[103278]: osd.5 marked itself down and dead 2024-09-20T20:28:12.594 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:12 smithi146 podman[119362]: 2024-09-20 20:28:12.315765149 +0000 UTC m=+0.856535080 container died b64a73dd22e7b688d9356fd29b90229b574361564e796adb60891c09f56e4a82 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2) 2024-09-20T20:28:13.144 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:12 smithi146 podman[119362]: 2024-09-20 20:28:12.924883145 +0000 UTC m=+1.465653075 container cleanup b64a73dd22e7b688d9356fd29b90229b574361564e796adb60891c09f56e4a82 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2) 2024-09-20T20:28:13.145 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:12 smithi146 bash[119362]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5 2024-09-20T20:28:13.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:12 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:28:13.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:12 smithi146 ceph-mon[103278]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T20:28:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:12 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:28:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:12 smithi078 ceph-mon[91136]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T20:28:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:12 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:28:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:12 smithi078 ceph-mon[99959]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T20:28:13.467 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119378]: 2024-09-20 20:28:13.143005943 +0000 UTC m=+0.818966354 container remove b64a73dd22e7b688d9356fd29b90229b574361564e796adb60891c09f56e4a82 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:28:13.756 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119467]: 2024-09-20 20:28:13.368968508 +0000 UTC m=+0.015910874 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:28:13.756 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119467]: 2024-09-20 20:28:13.485669963 +0000 UTC m=+0.132612320 container create 9f9f0664690e6a6b23cb53e5de24feb54048a55b1c1415276d18ae8735b39083 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-20T20:28:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:13 smithi146 ceph-mon[103278]: pgmap v219: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:13 smithi146 ceph-mon[103278]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T20:28:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:14.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119467]: 2024-09-20 20:28:13.75475607 +0000 UTC m=+0.401698436 container init 9f9f0664690e6a6b23cb53e5de24feb54048a55b1c1415276d18ae8735b39083 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T20:28:14.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119467]: 2024-09-20 20:28:13.758133372 +0000 UTC m=+0.405075729 container start 9f9f0664690e6a6b23cb53e5de24feb54048a55b1c1415276d18ae8735b39083 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-20T20:28:14.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119467]: 2024-09-20 20:28:13.854852522 +0000 UTC m=+0.501794893 container attach 9f9f0664690e6a6b23cb53e5de24feb54048a55b1c1415276d18ae8735b39083 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T20:28:14.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:13 smithi146 podman[119467]: 2024-09-20 20:28:13.954986829 +0000 UTC m=+0.601929188 container died 9f9f0664690e6a6b23cb53e5de24feb54048a55b1c1415276d18ae8735b39083 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[91136]: pgmap v219: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[91136]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[99959]: pgmap v219: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[99959]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T20:28:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:14.934 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:14 smithi146 podman[119490]: 2024-09-20 20:28:14.575413817 +0000 UTC m=+0.614844094 container remove 9f9f0664690e6a6b23cb53e5de24feb54048a55b1c1415276d18ae8735b39083 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T20:28:14.935 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:14 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.5.service: Deactivated successfully. 2024-09-20T20:28:14.935 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:14 smithi146 systemd[1]: Stopped Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:28:14.935 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:14 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.5.service: Consumed 9.727s CPU time. 2024-09-20T20:28:14.935 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:14 smithi146 systemd[1]: Starting Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:28:15.263 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 podman[119589]: 2024-09-20 20:28:14.924974664 +0000 UTC m=+0.015973465 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:28:15.263 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 podman[119589]: 2024-09-20 20:28:15.026673349 +0000 UTC m=+0.117672127 container create 117ebbc7dfd528b1e9274778e35d4824f024e4c0db017db1829b4ed05b4202ee (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:28:15.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:14 smithi146 ceph-mon[103278]: pgmap v221: 161 pgs: 12 active+undersized, 10 peering, 10 stale+active+clean, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 19/633 objects degraded (3.002%) 2024-09-20T20:28:15.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:14 smithi146 ceph-mon[103278]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-20T20:28:15.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:14 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 19/633 objects degraded (3.002%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:14 smithi078 ceph-mon[99959]: pgmap v221: 161 pgs: 12 active+undersized, 10 peering, 10 stale+active+clean, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 19/633 objects degraded (3.002%) 2024-09-20T20:28:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:14 smithi078 ceph-mon[99959]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-20T20:28:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:14 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 19/633 objects degraded (3.002%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:14 smithi078 ceph-mon[91136]: pgmap v221: 161 pgs: 12 active+undersized, 10 peering, 10 stale+active+clean, 8 active+undersized+degraded, 121 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 19/633 objects degraded (3.002%) 2024-09-20T20:28:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:14 smithi078 ceph-mon[91136]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-20T20:28:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:14 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 19/633 objects degraded (3.002%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 podman[119589]: 2024-09-20 20:28:15.262084454 +0000 UTC m=+0.353083246 container init 117ebbc7dfd528b1e9274778e35d4824f024e4c0db017db1829b4ed05b4202ee (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 podman[119589]: 2024-09-20 20:28:15.265537567 +0000 UTC m=+0.356536349 container start 117ebbc7dfd528b1e9274778e35d4824f024e4c0db017db1829b4ed05b4202ee (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 podman[119589]: 2024-09-20 20:28:15.34574109 +0000 UTC m=+0.436739874 container attach 117ebbc7dfd528b1e9274778e35d4824f024e4c0db017db1829b4ed05b4202ee (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 bash[119589]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:15.517 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 bash[119589]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:16.157 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:16.157 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 bash[119589]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:16.157 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:16.157 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:15 smithi146 bash[119589]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:16.157 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:16.467 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 bash[119589]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:16.467 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 bash[119589]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 bash[119589]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 bash[119589]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[119604]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 bash[119589]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:28:16.468 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 podman[119589]: 2024-09-20 20:28:16.183543969 +0000 UTC m=+1.274542762 container died 117ebbc7dfd528b1e9274778e35d4824f024e4c0db017db1829b4ed05b4202ee (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:28:17.199 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:16 smithi146 podman[119740]: 2024-09-20 20:28:16.849623708 +0000 UTC m=+0.660014589 container remove 117ebbc7dfd528b1e9274778e35d4824f024e4c0db017db1829b4ed05b4202ee (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:28:17.199 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 podman[119798]: 2024-09-20 20:28:16.959292328 +0000 UTC m=+0.015981201 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:28:17.199 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 podman[119798]: 2024-09-20 20:28:17.092338616 +0000 UTC m=+0.149027476 container create 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T20:28:17.713 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 podman[119798]: 2024-09-20 20:28:17.395095425 +0000 UTC m=+0.451784279 container init 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:28:17.713 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 podman[119798]: 2024-09-20 20:28:17.39878428 +0000 UTC m=+0.455473140 container start 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-09-20T20:28:17.713 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-osd[119818]: -- 172.21.15.146:0/3306788774 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557888e09a40 con 0x557889bd8400 2024-09-20T20:28:17.713 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 bash[119798]: 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 2024-09-20T20:28:17.713 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:17 smithi146 systemd[1]: Started Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: pgmap v222: 161 pgs: 32 active+undersized, 10 peering, 18 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55/633 objects degraded (8.689%) 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: pgmap v222: 161 pgs: 32 active+undersized, 10 peering, 18 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55/633 objects degraded (8.689%) 2024-09-20T20:28:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: pgmap v222: 161 pgs: 32 active+undersized, 10 peering, 18 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55/633 objects degraded (8.689%) 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:19.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:19.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:19.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:19.559 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[119814]: 2024-09-20T20:28:19.302+0000 7fed91a3c640 -1 Falling back to public interface 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: pgmap v223: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: pgmap v223: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: pgmap v223: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-20T20:28:20.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:20.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:20.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:28:20] "GET /metrics HTTP/1.1" 200 38840 "" "Prometheus/2.43.0" 2024-09-20T20:28:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:21.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:21.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:21.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: pgmap v224: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: pgmap v224: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: pgmap v224: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:22.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:22.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:22.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:22.890 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[119814]: 2024-09-20T20:28:22.635+0000 7fed91a3c640 -1 osd.5 140 log_to_monitors true 2024-09-20T20:28:23.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: pgmap v225: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: pgmap v225: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: pgmap v225: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%) 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:23.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.016 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[119814]: 2024-09-20T20:28:23.650+0000 7fed857de640 -1 osd.5 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.321 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714] boot 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: pgmap v228: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 75/633 objects degraded (11.848%) 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.333 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:28:25.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714] boot 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: osd.5 [v2:172.21.15.146:6808/1449785714,v1:172.21.15.146:6809/1449785714] boot 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: pgmap v228: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 75/633 objects degraded (11.848%) 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: pgmap v228: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 75/633 objects degraded (11.848%) 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:25.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:25.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:25.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:25.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:25.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:25 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.286 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:28:26.127+0000 7fcfc9d8b640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:28:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T20:28:26.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:26.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:25 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:28:27.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:26 smithi146 ceph-mon[103278]: pgmap v230: 161 pgs: 20 peering, 23 active+undersized, 17 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:28:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[91136]: pgmap v230: 161 pgs: 20 peering, 23 active+undersized, 17 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:27.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T20:28:27.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:26 smithi078 ceph-mon[99959]: pgmap v230: 161 pgs: 20 peering, 23 active+undersized, 17 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:28:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:27 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:29.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:29.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:29.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:29 smithi146 ceph-mon[103278]: pgmap v231: 161 pgs: 20 peering, 5 active+undersized, 4 active+undersized+degraded, 132 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 12/633 objects degraded (1.896%) 2024-09-20T20:28:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:29 smithi078 ceph-mon[99959]: pgmap v231: 161 pgs: 20 peering, 5 active+undersized, 4 active+undersized+degraded, 132 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 12/633 objects degraded (1.896%) 2024-09-20T20:28:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:29 smithi078 ceph-mon[91136]: pgmap v231: 161 pgs: 20 peering, 5 active+undersized, 4 active+undersized+degraded, 132 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 12/633 objects degraded (1.896%) 2024-09-20T20:28:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:28:30] "GET /metrics HTTP/1.1" 200 38837 "" "Prometheus/2.43.0" 2024-09-20T20:28:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:31 smithi146 ceph-mon[103278]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-09-20T20:28:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:31 smithi078 ceph-mon[99959]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-09-20T20:28:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:31 smithi078 ceph-mon[91136]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-09-20T20:28:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:32 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/633 objects degraded (1.896%), 4 pgs degraded) 2024-09-20T20:28:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:32 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/633 objects degraded (1.896%), 4 pgs degraded) 2024-09-20T20:28:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:32 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/633 objects degraded (1.896%), 4 pgs degraded) 2024-09-20T20:28:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:33 smithi146 ceph-mon[103278]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[99959]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:33 smithi078 ceph-mon[91136]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-09-20T20:28:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:35 smithi146 ceph-mon[103278]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:35 smithi078 ceph-mon[91136]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:35 smithi078 ceph-mon[99959]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:38.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:37 smithi146 ceph-mon[103278]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-20T20:28:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:37 smithi078 ceph-mon[91136]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-20T20:28:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:37 smithi078 ceph-mon[99959]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-20T20:28:39.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:39.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:39 smithi146 ceph-mon[103278]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:39 smithi078 ceph-mon[99959]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:39 smithi078 ceph-mon[91136]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:40.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:40.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:40.524 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:28:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:28:40] "GET /metrics HTTP/1.1" 200 38837 "" "Prometheus/2.43.0" 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 19m - - 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 19m - - 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (13m) 13s ago 14m 25.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (14m) 8s ago 14m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6m) 13s ago 12m 61.9M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:28:41.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5m) 8s ago 18m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7m) 13s ago 20m 519M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5m) 13s ago 20m 68.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4m) 8s ago 18m 47.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4m) 13s ago 18m 53.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (14m) 13s ago 14m 14.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (13m) 8s ago 13m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3m) 13s ago 18m 50.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2m) 13s ago 17m 48.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (115s) 13s ago 17m 44.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (84s) 13s ago 17m 45.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (52s) 8s ago 16m 48.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (24s) 8s ago 16m 45.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 94bccacae1d6 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (15m) 8s ago 15m 59.3M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:28:41.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (15m) 8s ago 15m 72.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:28:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7m) 8s ago 14m 51.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:28:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (12m) 13s ago 12m 82.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:28:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (12m) 8s ago 12m 82.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:28:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (12m) 13s ago 12m 83.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:28:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (12m) 8s ago 12m 85.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:28:41.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 2, 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:28:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:28:41.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 6, 2024-09-20T20:28:41.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 11 2024-09-20T20:28:41.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:28:41.410 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:28:41.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[99959]: from='client.64439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:41.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[99959]: from='client.64445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:41.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[99959]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:41.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:41.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[91136]: from='client.64439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:41.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[91136]: from='client.64445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:41.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[91136]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:41.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:41.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:41 smithi146 ceph-mon[103278]: from='client.64439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:41.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:41 smithi146 ceph-mon[103278]: from='client.64445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:41.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:41 smithi146 ceph-mon[103278]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:41.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16} : dispatch 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:28:41.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:28:41.694 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:28:41.694 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "11/23 daemons upgraded", 2024-09-20T20:28:41.694 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:28:41.694 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:28:41.694 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:28:42.018 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:42.018 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:42.018 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[99959]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[99959]: Upgrade: osd.5 is safe to restart 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3317969765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[99959]: from='client.44535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2274962479' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[91136]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[91136]: Upgrade: osd.5 is safe to restart 2024-09-20T20:28:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3317969765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[91136]: from='client.44535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2274962479' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:42 smithi146 ceph-mon[103278]: from='client.44523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:42 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T20:28:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:42 smithi146 ceph-mon[103278]: Upgrade: osd.5 is safe to restart 2024-09-20T20:28:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3317969765' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:42 smithi146 ceph-mon[103278]: from='client.44535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:28:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2274962479' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:28:43.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: Upgrade: Updating osd.5 2024-09-20T20:28:43.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:28:43.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:43.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: Deploying daemon osd.5 on smithi146 2024-09-20T20:28:43.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:43.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:43.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: Upgrade: Updating osd.5 2024-09-20T20:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: Deploying daemon osd.5 on smithi146 2024-09-20T20:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: Upgrade: Updating osd.5 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: Deploying daemon osd.5 on smithi146 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:45.669 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:45 smithi146 systemd[1]: Stopping Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:28:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:45 smithi146 ceph-mon[103278]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:45 smithi146 ceph-mon[103278]: osd.5 marked itself down and dead 2024-09-20T20:28:46.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[119814]: 2024-09-20T20:28:45.667+0000 7fed8e9d4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:28:46.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[119814]: 2024-09-20T20:28:45.667+0000 7fed8e9d4640 -1 osd.5 145 *** Got signal Terminated *** 2024-09-20T20:28:46.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[119814]: 2024-09-20T20:28:45.667+0000 7fed8e9d4640 -1 osd.5 145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:28:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:45 smithi078 ceph-mon[91136]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:45 smithi078 ceph-mon[91136]: osd.5 marked itself down and dead 2024-09-20T20:28:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:45 smithi078 ceph-mon[99959]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:28:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:45 smithi078 ceph-mon[99959]: osd.5 marked itself down and dead 2024-09-20T20:28:46.727 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:46 smithi146 podman[123460]: 2024-09-20 20:28:46.406530271 +0000 UTC m=+0.831982393 container died 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-20T20:28:46.727 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:46 smithi146 podman[123460]: 2024-09-20 20:28:46.72551213 +0000 UTC m=+1.150964244 container cleanup 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T20:28:47.016 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:46 smithi146 bash[123460]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5 2024-09-20T20:28:47.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:46 smithi146 podman[123476]: 2024-09-20 20:28:46.926995106 +0000 UTC m=+0.519561917 container remove 94bccacae1d6c0a1546511c7423773e8e2957afbc7c6e737fdfbb56190d97138 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-20T20:28:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:46 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:28:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:46 smithi146 ceph-mon[103278]: osdmap e146: 8 total, 7 up, 8 in 2024-09-20T20:28:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:46 smithi146 ceph-mon[103278]: pgmap v241: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:46 smithi146 ceph-mon[103278]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[91136]: osdmap e146: 8 total, 7 up, 8 in 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[91136]: pgmap v241: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[91136]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[99959]: osdmap e146: 8 total, 7 up, 8 in 2024-09-20T20:28:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[99959]: pgmap v241: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:28:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:46 smithi078 ceph-mon[99959]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T20:28:47.732 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:47 smithi146 podman[123568]: 2024-09-20 20:28:47.241434268 +0000 UTC m=+0.016347887 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:28:47.732 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:47 smithi146 podman[123568]: 2024-09-20 20:28:47.36955572 +0000 UTC m=+0.144469374 container create 45f8890e133e84f865cc1d7344e74fd101d064a55d3c8f533cc6fed5b26ae920 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T20:28:48.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:47 smithi146 podman[123568]: 2024-09-20 20:28:47.730631639 +0000 UTC m=+0.505545261 container init 45f8890e133e84f865cc1d7344e74fd101d064a55d3c8f533cc6fed5b26ae920 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:28:48.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:47 smithi146 podman[123568]: 2024-09-20 20:28:47.734256973 +0000 UTC m=+0.509170585 container start 45f8890e133e84f865cc1d7344e74fd101d064a55d3c8f533cc6fed5b26ae920 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T20:28:48.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:47 smithi146 podman[123568]: 2024-09-20 20:28:47.813768827 +0000 UTC m=+0.588682451 container attach 45f8890e133e84f865cc1d7344e74fd101d064a55d3c8f533cc6fed5b26ae920 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T20:28:48.017 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:47 smithi146 podman[123568]: 2024-09-20 20:28:47.927571618 +0000 UTC m=+0.702485232 container died 45f8890e133e84f865cc1d7344e74fd101d064a55d3c8f533cc6fed5b26ae920 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:28:48.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:48.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:48.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:48.791 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:48 smithi146 podman[123640]: 2024-09-20 20:28:48.534046567 +0000 UTC m=+0.595477360 container remove 45f8890e133e84f865cc1d7344e74fd101d064a55d3c8f533cc6fed5b26ae920 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T20:28:48.791 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:48 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.5.service: Deactivated successfully. 2024-09-20T20:28:48.791 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:48 smithi146 systemd[1]: Stopped Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:28:48.791 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:48 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.5.service: Consumed 1.815s CPU time. 2024-09-20T20:28:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:49.202 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:48 smithi146 systemd[1]: Starting Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:28:49.202 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:49 smithi146 podman[123846]: 2024-09-20 20:28:48.910196357 +0000 UTC m=+0.017229040 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:28:49.203 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:49 smithi146 podman[123846]: 2024-09-20 20:28:49.010022476 +0000 UTC m=+0.117055200 container create 34c93bb8f27804c4cb9c66d3dd3eb3b004ac5938d1faf81925209c11a03b12d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T20:28:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:49 smithi078 ceph-mon[99959]: pgmap v243: 161 pgs: 9 active+undersized, 15 stale+active+clean, 7 active+undersized+degraded, 130 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 22/633 objects degraded (3.476%) 2024-09-20T20:28:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:49 smithi078 ceph-mon[91136]: pgmap v243: 161 pgs: 9 active+undersized, 15 stale+active+clean, 7 active+undersized+degraded, 130 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 22/633 objects degraded (3.476%) 2024-09-20T20:28:50.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:49 smithi146 ceph-mon[103278]: pgmap v243: 161 pgs: 9 active+undersized, 15 stale+active+clean, 7 active+undersized+degraded, 130 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 22/633 objects degraded (3.476%) 2024-09-20T20:28:50.082 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:49 smithi146 podman[123846]: 2024-09-20 20:28:49.940563796 +0000 UTC m=+1.047596482 container init 34c93bb8f27804c4cb9c66d3dd3eb3b004ac5938d1faf81925209c11a03b12d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:28:50.082 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:49 smithi146 podman[123846]: 2024-09-20 20:28:49.945096818 +0000 UTC m=+1.052129501 container start 34c93bb8f27804c4cb9c66d3dd3eb3b004ac5938d1faf81925209c11a03b12d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T20:28:50.082 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 podman[123846]: 2024-09-20 20:28:50.015622694 +0000 UTC m=+1.122655386 container attach 34c93bb8f27804c4cb9c66d3dd3eb3b004ac5938d1faf81925209c11a03b12d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T20:28:50.345 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:50.345 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:50.346 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:50.346 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:28:50.767 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:50.767 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:50.767 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:50.767 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:28:50] "GET /metrics HTTP/1.1" 200 38840 "" "Prometheus/2.43.0" 2024-09-20T20:28:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:50 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 22/633 objects degraded (3.476%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:50 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 22/633 objects degraded (3.476%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:51.058 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:28:51.058 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 22/633 objects degraded (3.476%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T20:28:51.058 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:51.059 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T20:28:51.059 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate[123884]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:28:51.059 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:50 smithi146 bash[123846]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T20:28:51.351 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:51 smithi146 podman[123846]: 2024-09-20 20:28:51.093163381 +0000 UTC m=+2.200196064 container died 34c93bb8f27804c4cb9c66d3dd3eb3b004ac5938d1faf81925209c11a03b12d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T20:28:51.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:51 smithi146 ceph-mon[103278]: pgmap v244: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:51.953 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:51 smithi146 podman[124135]: 2024-09-20 20:28:51.617414506 +0000 UTC m=+0.747529090 container remove 34c93bb8f27804c4cb9c66d3dd3eb3b004ac5938d1faf81925209c11a03b12d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5-activate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T20:28:51.953 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:51 smithi146 podman[124196]: 2024-09-20 20:28:51.826661753 +0000 UTC m=+0.106619964 container create 0678af9b109f7fc40d7b0f5b8ff89a6f5dbe14b4b134b067eb2a7ba67605a71e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-20T20:28:51.953 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:51 smithi146 podman[124196]: 2024-09-20 20:28:51.73706961 +0000 UTC m=+0.017027839 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:28:52.116 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:51 smithi078 ceph-mon[99959]: pgmap v244: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:52.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:51 smithi078 ceph-mon[91136]: pgmap v244: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:52.267 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:52 smithi146 podman[124196]: 2024-09-20 20:28:52.0540359 +0000 UTC m=+0.333994125 container init 0678af9b109f7fc40d7b0f5b8ff89a6f5dbe14b4b134b067eb2a7ba67605a71e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-20T20:28:52.267 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:52 smithi146 podman[124196]: 2024-09-20 20:28:52.057242427 +0000 UTC m=+0.337200637 container start 0678af9b109f7fc40d7b0f5b8ff89a6f5dbe14b4b134b067eb2a7ba67605a71e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:28:52.267 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:52 smithi146 ceph-osd[124215]: -- 172.21.15.146:0/3830443244 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce5c525a40 con 0x55ce5c503c00 2024-09-20T20:28:52.267 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:52 smithi146 bash[124196]: 0678af9b109f7fc40d7b0f5b8ff89a6f5dbe14b4b134b067eb2a7ba67605a71e 2024-09-20T20:28:52.267 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:52 smithi146 systemd[1]: Started Ceph osd.5 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:28:53.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:28:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:28:52.575+0000 7fcfc9d8b640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:28:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:28:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:28:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:28:53.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-mon[103278]: pgmap v245: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:53.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:28:53.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[99959]: pgmap v245: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-20T20:28:53.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:53 smithi078 ceph-mon[91136]: pgmap v245: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:54.250 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:53 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[124211]: 2024-09-20T20:28:53.947+0000 7f57a3529640 -1 Falling back to public interface 2024-09-20T20:28:56.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:55 smithi146 ceph-mon[103278]: pgmap v246: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 343 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:55 smithi078 ceph-mon[99959]: pgmap v246: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 343 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:55 smithi078 ceph-mon[91136]: pgmap v246: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 343 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:56 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:57.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:56 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:57.157 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:56 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 75/633 objects degraded (11.848%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T20:28:57.597 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:57 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[124211]: 2024-09-20T20:28:57.476+0000 7f57a3529640 -1 osd.5 145 log_to_monitors true 2024-09-20T20:28:58.175 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 20 20:28:57 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-5[124211]: 2024-09-20T20:28:57.928+0000 7f579b2d3640 -1 osd.5 145 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:28:58.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:57 smithi146 ceph-mon[103278]: pgmap v247: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:58.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:57 smithi146 ceph-mon[103278]: from='osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:58.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:57 smithi146 ceph-mon[103278]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:58.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:57 smithi078 ceph-mon[99959]: pgmap v247: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:58.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:57 smithi078 ceph-mon[99959]: from='osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:58.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:57 smithi078 ceph-mon[99959]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:58.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:57 smithi078 ceph-mon[91136]: pgmap v247: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:58.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:57 smithi078 ceph-mon[91136]: from='osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:58.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:57 smithi078 ceph-mon[91136]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: osdmap e148: 8 total, 7 up, 8 in 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: pgmap v249: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:59.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: osdmap e148: 8 total, 7 up, 8 in 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: pgmap v249: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: osdmap e148: 8 total, 7 up, 8 in 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: pgmap v249: 161 pgs: 37 active+undersized, 23 active+undersized+degraded, 101 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s; 75/633 objects degraded (11.848%); 0 B/s, 0 objects/s recovering 2024-09-20T20:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451] boot 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.5 on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:29:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:28:59 smithi146 ceph-mon[103278]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T20:29:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451] boot 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.5 on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[91136]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: osd.5 [v2:172.21.15.146:6808/2669689451,v1:172.21.15.146:6809/2669689451] boot 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T20:29:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-20T20:29:00.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:00.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: Health check failed: 1 failed cephadm daemon(s) ['daemon osd.5 on smithi146 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-09-20T20:29:00.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:28:59 smithi078 ceph-mon[99959]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T20:29:00.928 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:29:00] "GET /metrics HTTP/1.1" 200 38692 "" "Prometheus/2.43.0" 2024-09-20T20:29:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:00 smithi146 ceph-mon[103278]: pgmap v252: 161 pgs: 23 active+undersized, 17 active+undersized+degraded, 121 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:29:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:00 smithi078 ceph-mon[99959]: pgmap v252: 161 pgs: 23 active+undersized, 17 active+undersized+degraded, 121 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:29:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:00 smithi078 ceph-mon[91136]: pgmap v252: 161 pgs: 23 active+undersized, 17 active+undersized+degraded, 121 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:29:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:01 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:01 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:01 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 54/633 objects degraded (8.531%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:02 smithi146 ceph-mon[103278]: pgmap v253: 161 pgs: 23 active+undersized, 17 active+undersized+degraded, 121 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:29:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:02 smithi078 ceph-mon[91136]: pgmap v253: 161 pgs: 23 active+undersized, 17 active+undersized+degraded, 121 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:29:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:02 smithi078 ceph-mon[99959]: pgmap v253: 161 pgs: 23 active+undersized, 17 active+undersized+degraded, 121 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s; 54/633 objects degraded (8.531%) 2024-09-20T20:29:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:03 smithi146 ceph-mon[103278]: pgmap v254: 161 pgs: 8 active+undersized, 5 active+undersized+degraded, 148 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s; 13/633 objects degraded (2.054%) 2024-09-20T20:29:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[91136]: pgmap v254: 161 pgs: 8 active+undersized, 5 active+undersized+degraded, 148 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s; 13/633 objects degraded (2.054%) 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[99959]: pgmap v254: 161 pgs: 8 active+undersized, 5 active+undersized+degraded, 148 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s; 13/633 objects degraded (2.054%) 2024-09-20T20:29:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:04 smithi146 ceph-mon[103278]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.5 on smithi146 is in unknown state']) 2024-09-20T20:29:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:04 smithi078 ceph-mon[91136]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.5 on smithi146 is in unknown state']) 2024-09-20T20:29:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:04 smithi078 ceph-mon[99959]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon osd.5 on smithi146 is in unknown state']) 2024-09-20T20:29:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:05 smithi146 ceph-mon[103278]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-09-20T20:29:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:05 smithi078 ceph-mon[91136]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-09-20T20:29:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:05 smithi078 ceph-mon[99959]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-09-20T20:29:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:06 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/633 objects degraded (2.054%), 5 pgs degraded) 2024-09-20T20:29:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:06 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/633 objects degraded (2.054%), 5 pgs degraded) 2024-09-20T20:29:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:06 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/633 objects degraded (2.054%), 5 pgs degraded) 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: Upgrade: osd.6 is safe to restart 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: Upgrade: osd.6 is safe to restart 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-09-20T20:29:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:29:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:29:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: Upgrade: osd.6 is safe to restart 2024-09-20T20:29:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: Upgrade: Updating osd.6 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: Deploying daemon osd.6 on smithi146 2024-09-20T20:29:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: Upgrade: Updating osd.6 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: Deploying daemon osd.6 on smithi146 2024-09-20T20:29:10.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: Upgrade: Updating osd.6 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: Deploying daemon osd.6 on smithi146 2024-09-20T20:29:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:29:10] "GET /metrics HTTP/1.1" 200 38692 "" "Prometheus/2.43.0" 2024-09-20T20:29:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:12 smithi146 ceph-mon[103278]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-09-20T20:29:12.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:12 smithi146 systemd[1]: Stopping Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:29:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:12 smithi078 ceph-mon[91136]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-09-20T20:29:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:12 smithi078 ceph-mon[99959]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-09-20T20:29:12.321 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:29:12.767 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[38746]: 2024-09-20T20:29:12.408+0000 7fe14cfe3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:29:12.767 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[38746]: 2024-09-20T20:29:12.408+0000 7fe14cfe3640 -1 osd.6 150 *** Got signal Terminated *** 2024-09-20T20:29:12.767 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[38746]: 2024-09-20T20:29:12.408+0000 7fe14cfe3640 -1 osd.6 150 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:29:12.876 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:29:12.876 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 19m - - 2024-09-20T20:29:12.876 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 19m - - 2024-09-20T20:29:12.876 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (13m) 4s ago 15m 25.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:29:12.876 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (14m) 9s ago 14m 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:29:12.876 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7m) 4s ago 12m 61.9M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6m) 9s ago 19m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (8m) 4s ago 20m 520M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6m) 4s ago 20m 71.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5m) 9s ago 19m 49.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5m) 4s ago 19m 54.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (14m) 4s ago 14m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (14m) 9s ago 14m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4m) 4s ago 18m 52.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3m) 4s ago 18m 50.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2m) 4s ago 17m 46.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (116s) 4s ago 17m 48.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (84s) 9s ago 17m 48.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:29:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (21s) 9s ago 16m 37.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (16m) 9s ago 16m 60.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 a17a22c4218f 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (15m) 9s ago 15m 72.5M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7m) 9s ago 15m 53.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (13m) 4s ago 13m 83.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (13m) 9s ago 13m 83.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (13m) 4s ago 13m 83.6M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:29:12.879 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (13m) 9s ago 13m 85.9M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:29:13.204 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:29:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:13.206 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:29:13.206 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 5, 2024-09-20T20:29:13.206 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 11 2024-09-20T20:29:13.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:29:13.206 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[91136]: from='client.64481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[91136]: osd.6 marked itself down and dead 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[91136]: from='client.64487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[91136]: from='client.44565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[99959]: from='client.64481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[99959]: osd.6 marked itself down and dead 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[99959]: from='client.64487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:13 smithi078 ceph-mon[99959]: from='client.44565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:13 smithi146 ceph-mon[103278]: from='client.64481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:13 smithi146 ceph-mon[103278]: osd.6 marked itself down and dead 2024-09-20T20:29:13.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:13 smithi146 ceph-mon[103278]: from='client.64487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:13 smithi146 ceph-mon[103278]: from='client.44565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "11/23 daemons upgraded", 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:29:13.488 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:29:13.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:13 smithi146 podman[126255]: 2024-09-20 20:29:13.360264688 +0000 UTC m=+1.032771377 container died a17a22c4218f5c30a96604ebd79cdf8849d36b1ab48f28345b92724f45199456 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:29:13.797 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down 2024-09-20T20:29:13.797 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:13.797 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:29:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: osd.6 (root=default,host=smithi146) is down 2024-09-20T20:29:13.995 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:13 smithi146 podman[126255]: 2024-09-20 20:29:13.741948546 +0000 UTC m=+1.414455225 container cleanup a17a22c4218f5c30a96604ebd79cdf8849d36b1ab48f28345b92724f45199456 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-0b2cc4d, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T20:29:13.995 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:13 smithi146 bash[126255]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6 2024-09-20T20:29:14.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:13 smithi146 podman[126271]: 2024-09-20 20:29:13.993278608 +0000 UTC m=+0.625893822 container remove a17a22c4218f5c30a96604ebd79cdf8849d36b1ab48f28345b92724f45199456 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: osdmap e151: 8 total, 7 up, 8 in 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/387720807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: from='client.64505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:14.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1541194184' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: osdmap e151: 8 total, 7 up, 8 in 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/387720807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: from='client.64505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1541194184' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: osdmap e151: 8 total, 7 up, 8 in 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/387720807' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: from='client.64505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:14.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1541194184' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:14.615 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:14 smithi146 podman[126362]: 2024-09-20 20:29:14.235367534 +0000 UTC m=+0.015816780 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:29:14.615 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:14 smithi146 podman[126362]: 2024-09-20 20:29:14.35283818 +0000 UTC m=+0.133287457 container create 298f2bd9eebdc103a59319d6c376302571e680236ba4748c8f0eae1a2d421373 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-20T20:29:14.920 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:14 smithi146 podman[126362]: 2024-09-20 20:29:14.613804451 +0000 UTC m=+0.394253708 container init 298f2bd9eebdc103a59319d6c376302571e680236ba4748c8f0eae1a2d421373 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T20:29:14.920 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:14 smithi146 podman[126362]: 2024-09-20 20:29:14.617514622 +0000 UTC m=+0.397963862 container start 298f2bd9eebdc103a59319d6c376302571e680236ba4748c8f0eae1a2d421373 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T20:29:14.920 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:14 smithi146 podman[126362]: 2024-09-20 20:29:14.696920493 +0000 UTC m=+0.477369748 container attach 298f2bd9eebdc103a59319d6c376302571e680236ba4748c8f0eae1a2d421373 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T20:29:14.920 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:14 smithi146 podman[126362]: 2024-09-20 20:29:14.917965037 +0000 UTC m=+0.698414277 container died 298f2bd9eebdc103a59319d6c376302571e680236ba4748c8f0eae1a2d421373 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T20:29:15.210 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:15 smithi146 ceph-mon[103278]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T20:29:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:15 smithi078 ceph-mon[91136]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T20:29:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:15 smithi078 ceph-mon[99959]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T20:29:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:16 smithi146 ceph-mon[103278]: pgmap v262: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:29:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:16 smithi146 ceph-mon[103278]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T20:29:16.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 podman[126399]: 2024-09-20 20:29:16.030205113 +0000 UTC m=+1.207329295 container remove 298f2bd9eebdc103a59319d6c376302571e680236ba4748c8f0eae1a2d421373 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-20T20:29:16.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.6.service: Deactivated successfully. 2024-09-20T20:29:16.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 systemd[1]: Stopped Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:29:16.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.6.service: Consumed 8.875s CPU time. 2024-09-20T20:29:16.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 systemd[1]: Starting Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:29:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:16 smithi078 ceph-mon[91136]: pgmap v262: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:29:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:16 smithi078 ceph-mon[91136]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T20:29:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:16 smithi078 ceph-mon[99959]: pgmap v262: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T20:29:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:16 smithi078 ceph-mon[99959]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T20:29:16.883 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 podman[126519]: 2024-09-20 20:29:16.774824701 +0000 UTC m=+0.110597992 container create a0c80f6320b67151bf4ac9fce271bbd78919a35537a99022edf74eeac444cb5f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T20:29:16.884 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 podman[126519]: 2024-09-20 20:29:16.680301432 +0000 UTC m=+0.016074689 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:29:17.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 37/633 objects degraded (5.845%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:17.140 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 podman[126519]: 2024-09-20 20:29:16.985615781 +0000 UTC m=+0.321389035 container init a0c80f6320b67151bf4ac9fce271bbd78919a35537a99022edf74eeac444cb5f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T20:29:17.140 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:16 smithi146 podman[126519]: 2024-09-20 20:29:16.988932919 +0000 UTC m=+0.324706169 container start a0c80f6320b67151bf4ac9fce271bbd78919a35537a99022edf74eeac444cb5f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T20:29:17.140 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 podman[126519]: 2024-09-20 20:29:17.068916192 +0000 UTC m=+0.404689455 container attach a0c80f6320b67151bf4ac9fce271bbd78919a35537a99022edf74eeac444cb5f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS) 2024-09-20T20:29:17.140 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:17.141 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:17.141 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:17.141 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:17 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 37/633 objects degraded (5.845%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:17 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 37/633 objects degraded (5.845%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:17.897 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:17.897 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:17.897 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:17.897 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:17.897 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:17.898 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:18.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:18 smithi146 ceph-mon[103278]: pgmap v263: 161 pgs: 15 active+undersized, 10 peering, 3 stale+active+clean, 8 active+undersized+degraded, 125 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 37/633 objects degraded (5.845%) 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[126535]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 bash[126519]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:29:18.188 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:17 smithi146 podman[126519]: 2024-09-20 20:29:17.927924015 +0000 UTC m=+1.263697261 container died a0c80f6320b67151bf4ac9fce271bbd78919a35537a99022edf74eeac444cb5f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-20T20:29:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:18 smithi078 ceph-mon[99959]: pgmap v263: 161 pgs: 15 active+undersized, 10 peering, 3 stale+active+clean, 8 active+undersized+degraded, 125 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 37/633 objects degraded (5.845%) 2024-09-20T20:29:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:18 smithi078 ceph-mon[91136]: pgmap v263: 161 pgs: 15 active+undersized, 10 peering, 3 stale+active+clean, 8 active+undersized+degraded, 125 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 37/633 objects degraded (5.845%) 2024-09-20T20:29:18.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:18 smithi146 podman[126708]: 2024-09-20 20:29:18.60603985 +0000 UTC m=+0.672243902 container remove a0c80f6320b67151bf4ac9fce271bbd78919a35537a99022edf74eeac444cb5f (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-20T20:29:18.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:18 smithi146 podman[126857]: 2024-09-20 20:29:18.815285358 +0000 UTC m=+0.111777863 container create 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T20:29:18.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:18 smithi146 podman[126857]: 2024-09-20 20:29:18.720039747 +0000 UTC m=+0.016532218 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:29:19.006 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:29:18 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:29:18.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:29:19.006 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:29:18 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:29:18.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:29:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:19 smithi078 ceph-mon[99959]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T20:29:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:19 smithi078 ceph-mon[91136]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T20:29:19.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:19 smithi146 ceph-mon[103278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T20:29:19.365 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:19 smithi146 podman[126857]: 2024-09-20 20:29:19.058978133 +0000 UTC m=+0.355470597 container init 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:29:19.365 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:19 smithi146 podman[126857]: 2024-09-20 20:29:19.062454756 +0000 UTC m=+0.358947216 container start 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:29:19.365 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:19 smithi146 ceph-osd[126878]: -- 172.21.15.146:0/1791827109 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e868c19a40 con 0x55e8699e8400 2024-09-20T20:29:19.365 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:19 smithi146 bash[126857]: 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 2024-09-20T20:29:19.365 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:19 smithi146 systemd[1]: Started Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:29:20.478 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[126873]: 2024-09-20T20:29:20.211+0000 7fbb3f431640 -1 Falling back to public interface 2024-09-20T20:29:20.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: pgmap v264: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:20.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: pgmap v264: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: pgmap v264: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:20.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:20.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.552 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:20.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:29:20] "GET /metrics HTTP/1.1" 200 38838 "" "Prometheus/2.43.0" 2024-09-20T20:29:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.642 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.643 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: pgmap v265: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.522 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.523 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: pgmap v265: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: pgmap v265: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.542 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:22.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:22.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:22.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:22.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:22.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:29:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.546 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:23.547 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16} : dispatch 2024-09-20T20:29:24.016 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:23 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[126873]: 2024-09-20T20:29:23.788+0000 7fbb3f431640 -1 osd.6 150 log_to_monitors true 2024-09-20T20:29:24.378 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[126873]: 2024-09-20T20:29:24.269+0000 7fbb371db640 -1 osd.6 150 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:29:24.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:24.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: pgmap v266: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:24.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:24.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:29:24.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: Upgrade: osd.6 is safe to restart 2024-09-20T20:29:24.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: from='osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:24.379 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:24 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: pgmap v266: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:29:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: Upgrade: osd.6 is safe to restart 2024-09-20T20:29:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: from='osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: pgmap v266: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: Upgrade: osd.6 is safe to restart 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: from='osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:24 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:25.263 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:29:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=sqlstore.transactions t=2024-09-20T20:29:25.013323058Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T20:29:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:25 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:29:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:25 smithi146 ceph-mon[103278]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T20:29:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:25 smithi146 ceph-mon[103278]: from='osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:25 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:29:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[91136]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T20:29:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[91136]: from='osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:29:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[99959]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T20:29:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[99959]: from='osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:25 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:26 smithi146 ceph-mon[103278]: pgmap v268: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:26 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:26 smithi146 ceph-mon[103278]: osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365] boot 2024-09-20T20:29:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:26 smithi146 ceph-mon[103278]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T20:29:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[91136]: pgmap v268: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[91136]: osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365] boot 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[91136]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[99959]: pgmap v268: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:26.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[99959]: osd.6 [v2:172.21.15.146:6816/934725365,v1:172.21.15.146:6817/934725365] boot 2024-09-20T20:29:26.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[99959]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T20:29:26.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:29:27.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:27 smithi146 ceph-mon[103278]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T20:29:27.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:27 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:27 smithi146 ceph-mon[103278]: Upgrade: Updating osd.6 2024-09-20T20:29:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:29:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:27 smithi146 ceph-mon[103278]: Deploying daemon osd.6 on smithi146 2024-09-20T20:29:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[91136]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T20:29:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[91136]: Upgrade: Updating osd.6 2024-09-20T20:29:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:29:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[91136]: Deploying daemon osd.6 on smithi146 2024-09-20T20:29:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[99959]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T20:29:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[99959]: Upgrade: Updating osd.6 2024-09-20T20:29:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-20T20:29:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:27 smithi078 ceph-mon[99959]: Deploying daemon osd.6 on smithi146 2024-09-20T20:29:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:28 smithi146 ceph-mon[103278]: pgmap v271: 161 pgs: 1 peering, 32 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[91136]: pgmap v271: 161 pgs: 1 peering, 32 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[99959]: pgmap v271: 161 pgs: 1 peering, 32 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:29.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:29:28.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:29:29.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:29:28.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:29:29.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:29:28 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:29:28.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:29:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:30 smithi078 ceph-mon[91136]: pgmap v272: 161 pgs: 1 peering, 16 active+undersized, 10 active+undersized+degraded, 134 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:29:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:30 smithi078 ceph-mon[99959]: pgmap v272: 161 pgs: 1 peering, 16 active+undersized, 10 active+undersized+degraded, 134 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:29:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:30.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:30 smithi146 ceph-mon[103278]: pgmap v272: 161 pgs: 1 peering, 16 active+undersized, 10 active+undersized+degraded, 134 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:29:30.564 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:29:30] "GET /metrics HTTP/1.1" 200 38698 "" "Prometheus/2.43.0" 2024-09-20T20:29:31.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:31 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded) 2024-09-20T20:29:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:31 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded) 2024-09-20T20:29:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:31 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded) 2024-09-20T20:29:32.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-09-20T20:29:32.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 761 B/s rd, 0 op/s 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:32.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:33.011 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:32 smithi146 systemd[1]: Stopping Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:29:33.266 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[126873]: 2024-09-20T20:29:33.009+0000 7fbb3c3c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:29:33.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[126873]: 2024-09-20T20:29:33.009+0000 7fbb3c3c9640 -1 osd.6 155 *** Got signal Terminated *** 2024-09-20T20:29:33.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[126873]: 2024-09-20T20:29:33.009+0000 7fbb3c3c9640 -1 osd.6 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:29:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:33 smithi146 ceph-mon[103278]: osd.6 marked itself down and dead 2024-09-20T20:29:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:33 smithi078 ceph-mon[99959]: osd.6 marked itself down and dead 2024-09-20T20:29:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:33 smithi078 ceph-mon[91136]: osd.6 marked itself down and dead 2024-09-20T20:29:34.087 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:33 smithi146 podman[129280]: 2024-09-20 20:29:33.796271431 +0000 UTC m=+0.887457529 container died 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-20T20:29:34.421 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:34 smithi146 podman[129280]: 2024-09-20 20:29:34.301691528 +0000 UTC m=+1.392877661 container cleanup 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-20T20:29:34.421 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:34 smithi146 bash[129280]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6 2024-09-20T20:29:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:34 smithi146 ceph-mon[103278]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:34 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:29:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:34 smithi146 ceph-mon[103278]: osdmap e156: 8 total, 7 up, 8 in 2024-09-20T20:29:34.767 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:34 smithi146 podman[129298]: 2024-09-20 20:29:34.553063118 +0000 UTC m=+0.751407687 container remove 95e75196e3b23c9dc644a26927982a1b9d7bc1d8f8790c871f171de3dbf3e435 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T20:29:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:34 smithi078 ceph-mon[99959]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:34 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:29:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:34 smithi078 ceph-mon[99959]: osdmap e156: 8 total, 7 up, 8 in 2024-09-20T20:29:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:34 smithi078 ceph-mon[91136]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:34.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:34 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:29:34.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:34 smithi078 ceph-mon[91136]: osdmap e156: 8 total, 7 up, 8 in 2024-09-20T20:29:35.225 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:34 smithi146 podman[129385]: 2024-09-20 20:29:34.81747837 +0000 UTC m=+0.015833096 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:29:35.225 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:34 smithi146 podman[129385]: 2024-09-20 20:29:34.929044757 +0000 UTC m=+0.127399514 container create e82eb35bef58da58d2758929403b5c4ff0884fa0171c218217c0d79fccb1af15 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T20:29:35.225 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:35 smithi146 podman[129385]: 2024-09-20 20:29:35.122908532 +0000 UTC m=+0.321263252 container init e82eb35bef58da58d2758929403b5c4ff0884fa0171c218217c0d79fccb1af15 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T20:29:35.226 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:35 smithi146 podman[129385]: 2024-09-20 20:29:35.126492851 +0000 UTC m=+0.324847569 container start e82eb35bef58da58d2758929403b5c4ff0884fa0171c218217c0d79fccb1af15 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2) 2024-09-20T20:29:35.226 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:35 smithi146 podman[129385]: 2024-09-20 20:29:35.223095605 +0000 UTC m=+0.421450321 container attach e82eb35bef58da58d2758929403b5c4ff0884fa0171c218217c0d79fccb1af15 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T20:29:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:35 smithi146 ceph-mon[103278]: osdmap e157: 8 total, 7 up, 8 in 2024-09-20T20:29:35.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:35 smithi146 podman[129385]: 2024-09-20 20:29:35.314238224 +0000 UTC m=+0.512592947 container died e82eb35bef58da58d2758929403b5c4ff0884fa0171c218217c0d79fccb1af15 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:29:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:35 smithi078 ceph-mon[99959]: osdmap e157: 8 total, 7 up, 8 in 2024-09-20T20:29:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:35 smithi078 ceph-mon[91136]: osdmap e157: 8 total, 7 up, 8 in 2024-09-20T20:29:36.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 podman[129409]: 2024-09-20 20:29:36.093921342 +0000 UTC m=+0.773921565 container remove e82eb35bef58da58d2758929403b5c4ff0884fa0171c218217c0d79fccb1af15 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T20:29:36.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.6.service: Deactivated successfully. 2024-09-20T20:29:36.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 systemd[1]: Stopped Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:29:36.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.6.service: Consumed 1.667s CPU time. 2024-09-20T20:29:36.432 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 systemd[1]: Starting Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:29:36.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:36 smithi146 ceph-mon[103278]: pgmap v277: 161 pgs: 9 peering, 13 stale+active+clean, 139 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:36.741 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 podman[129520]: 2024-09-20 20:29:36.53703957 +0000 UTC m=+0.113176721 container create d7fdbc90240d73d115b2c262328fc741c16230e6d01f9d02c0d1811c4fdd049d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-20T20:29:36.741 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 podman[129520]: 2024-09-20 20:29:36.441362705 +0000 UTC m=+0.017499869 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:29:36.741 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 podman[129520]: 2024-09-20 20:29:36.739359879 +0000 UTC m=+0.315497040 container init d7fdbc90240d73d115b2c262328fc741c16230e6d01f9d02c0d1811c4fdd049d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T20:29:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:36 smithi078 ceph-mon[99959]: pgmap v277: 161 pgs: 9 peering, 13 stale+active+clean, 139 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:36 smithi078 ceph-mon[91136]: pgmap v277: 161 pgs: 9 peering, 13 stale+active+clean, 139 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:29:37.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 podman[129520]: 2024-09-20 20:29:36.743029284 +0000 UTC m=+0.319166439 container start d7fdbc90240d73d115b2c262328fc741c16230e6d01f9d02c0d1811c4fdd049d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:29:37.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 podman[129520]: 2024-09-20 20:29:36.83091764 +0000 UTC m=+0.407054791 container attach d7fdbc90240d73d115b2c262328fc741c16230e6d01f9d02c0d1811c4fdd049d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-20T20:29:37.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:37.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 bash[129520]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:37.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:37.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:36 smithi146 bash[129520]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:29:37.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 37/633 objects degraded (5.845%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:37.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:37.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:37.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:37.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:37.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:37.629 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T20:29:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:37 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 37/633 objects degraded (5.845%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:37 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 37/633 objects degraded (5.845%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:37.952 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate[129536]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 bash[129520]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T20:29:37.953 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:37 smithi146 podman[129520]: 2024-09-20 20:29:37.655638527 +0000 UTC m=+1.231775693 container died d7fdbc90240d73d115b2c262328fc741c16230e6d01f9d02c0d1811c4fdd049d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T20:29:38.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:38 smithi146 ceph-mon[103278]: pgmap v278: 161 pgs: 15 active+undersized, 9 peering, 4 stale+active+clean, 8 active+undersized+degraded, 125 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 37/633 objects degraded (5.845%) 2024-09-20T20:29:38.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:38.725 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 podman[129662]: 2024-09-20 20:29:38.376981241 +0000 UTC m=+0.711959629 container remove d7fdbc90240d73d115b2c262328fc741c16230e6d01f9d02c0d1811c4fdd049d (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6-activate, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T20:29:38.726 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 podman[129722]: 2024-09-20 20:29:38.487228367 +0000 UTC m=+0.016105118 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:29:38.726 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 podman[129722]: 2024-09-20 20:29:38.61108767 +0000 UTC m=+0.139964401 container create fead6e0b8b60650f815f22ed6ad963fc1d3ab110f2229b02a25737a33cae477e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-20T20:29:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:38 smithi078 ceph-mon[91136]: pgmap v278: 161 pgs: 15 active+undersized, 9 peering, 4 stale+active+clean, 8 active+undersized+degraded, 125 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 37/633 objects degraded (5.845%) 2024-09-20T20:29:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:38 smithi078 ceph-mon[99959]: pgmap v278: 161 pgs: 15 active+undersized, 9 peering, 4 stale+active+clean, 8 active+undersized+degraded, 125 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 37/633 objects degraded (5.845%) 2024-09-20T20:29:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:38.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:29:38 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:29:38.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:29:39.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 podman[129722]: 2024-09-20 20:29:38.846980544 +0000 UTC m=+0.375857280 container init fead6e0b8b60650f815f22ed6ad963fc1d3ab110f2229b02a25737a33cae477e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, ceph=True) 2024-09-20T20:29:39.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 podman[129722]: 2024-09-20 20:29:38.850236466 +0000 UTC m=+0.379113197 container start fead6e0b8b60650f815f22ed6ad963fc1d3ab110f2229b02a25737a33cae477e (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T20:29:39.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 ceph-osd[129741]: -- 172.21.15.146:0/3254511947 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c206723a40 con 0x55c2074f2400 2024-09-20T20:29:39.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 bash[129722]: fead6e0b8b60650f815f22ed6ad963fc1d3ab110f2229b02a25737a33cae477e 2024-09-20T20:29:39.017 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:38 smithi146 systemd[1]: Started Ceph osd.6 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:29:40.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: pgmap v279: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: pgmap v279: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: pgmap v279: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:40.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:40.635 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[129737]: 2024-09-20T20:29:40.484+0000 7f30280cf640 -1 Falling back to public interface 2024-09-20T20:29:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:29:40] "GET /metrics HTTP/1.1" 200 38698 "" "Prometheus/2.43.0" 2024-09-20T20:29:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:41.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:41.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:41.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:41.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: pgmap v280: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: pgmap v280: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: pgmap v280: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 77/633 objects degraded (12.164%) 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:42.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.181 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:29:44.267 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[129737]: 2024-09-20T20:29:44.036+0000 7f30280cf640 -1 osd.6 155 log_to_monitors true 2024-09-20T20:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: pgmap v281: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: pgmap v281: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: pgmap v281: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:44.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:44.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:44.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:44.739 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 20m - - 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 20m - - 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (14m) 16s ago 15m 25.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (15m) 12s ago 15m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7m) 16s ago 13m 61.9M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6m) 12s ago 19m 450M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (8m) 16s ago 21m 521M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6m) 16s ago 21m 72.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5m) 12s ago 19m 52.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5m) 16s ago 19m 55.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (15m) 16s ago 15m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (15m) 12s ago 15m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:29:44.740 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4m) 16s ago 19m 53.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3m) 16s ago 18m 51.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2m) 16s ago 18m 46.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2m) 16s ago 18m 47.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (116s) 12s ago 17m 50.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (52s) 12s ago 17m 43.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 starting - - - 4096M 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (16m) 12s ago 16m 73.0M 4096M 18.2.4-869-g0b2cc4d4 4404209dfb71 3c0d9a82257f 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (8m) 12s ago 15m 53.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (13m) 16s ago 13m 83.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (13m) 12s ago 13m 83.3M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (13m) 16s ago 13m 83.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:29:44.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (13m) 12s ago 13m 86.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:29:44.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 1, 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 6 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 5, 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 11 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:29:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='client.44595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='client.44604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='client.44595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='client.44595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='client.44604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='client.44604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='client.44610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:45.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:45.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:45.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:29:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:29:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "11/23 daemons upgraded", 2024-09-20T20:29:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:29:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:29:45.407 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:29:45.597 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-6[129737]: 2024-09-20T20:29:45.443+0000 7f301be71640 -1 osd.6 155 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:29:45.736 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); 1 osds down; Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded 2024-09-20T20:29:45.736 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: osd.6 (root=default,host=smithi146) is down 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-09-20T20:29:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.e is active+undersized+degraded, acting [4,2] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 6.3 is active+undersized+degraded, acting [7,2] 2024-09-20T20:29:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: pgmap v282: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 484 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: osdmap e158: 8 total, 7 up, 8 in 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3442115329' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='osd.6 ' entity='osd.6' 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3268491873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: pgmap v282: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 484 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: osdmap e158: 8 total, 7 up, 8 in 2024-09-20T20:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3442115329' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: pgmap v282: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 484 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: osdmap e158: 8 total, 7 up, 8 in 2024-09-20T20:29:46.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3442115329' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='osd.6 ' entity='osd.6' 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='client.44622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='osd.6 ' entity='osd.6' 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3268491873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3268491873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:46.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:46.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:46.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898] boot 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: osdmap e159: 8 total, 8 up, 8 in 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898] boot 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: osdmap e159: 8 total, 8 up, 8 in 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: osd.6 [v2:172.21.15.146:6816/4222897898,v1:172.21.15.146:6817/4222897898] boot 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: osdmap e159: 8 total, 8 up, 8 in 2024-09-20T20:29:47.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 77/633 objects degraded (12.164%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: pgmap v285: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: pgmap v285: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: pgmap v285: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/633 objects degraded (12.164%) 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:48.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:49.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: pgmap v287: 161 pgs: 17 active+undersized, 10 active+undersized+degraded, 134 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: pgmap v287: 161 pgs: 17 active+undersized, 10 active+undersized+degraded, 134 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:29:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: pgmap v287: 161 pgs: 17 active+undersized, 10 active+undersized+degraded, 134 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 39/633 objects degraded (6.161%) 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:50.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.008 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:29:50] "GET /metrics HTTP/1.1" 200 38842 "" "Prometheus/2.43.0" 2024-09-20T20:29:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:51.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:51.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:51.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:51.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:51.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.316 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded) 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:29:52.320 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded (PG_DEGRADED) 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded) 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:29:52.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:52.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 39/633 objects degraded (6.161%), 10 pgs degraded) 2024-09-20T20:29:52.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:29:52.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:29:52.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:29:53.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:53 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:29:53.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:53 smithi146 ceph-mon[103278]: Upgrade: osd.7 is safe to restart 2024-09-20T20:29:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:53 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:29:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:53 smithi078 ceph-mon[91136]: Upgrade: osd.7 is safe to restart 2024-09-20T20:29:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:53 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:29:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:53 smithi078 ceph-mon[99959]: Upgrade: osd.7 is safe to restart 2024-09-20T20:29:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:54 smithi146 ceph-mon[103278]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:54 smithi078 ceph-mon[91136]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:54 smithi078 ceph-mon[99959]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:56 smithi146 ceph-mon[103278]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[91136]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[99959]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: Upgrade: Updating osd.7 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: Deploying daemon osd.7 on smithi146 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: Upgrade: Updating osd.7 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: Deploying daemon osd.7 on smithi146 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: Upgrade: Updating osd.7 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: Deploying daemon osd.7 on smithi146 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:58.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[99959]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[91136]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:59.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:58 smithi146 ceph-mon[103278]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-20T20:29:59.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:29:59.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:29:59.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:00.501 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:00 smithi146 systemd[1]: Stopping Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:30:00.766 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[42208]: 2024-09-20T20:30:00.498+0000 7f50b10dc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:30:00.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[42208]: 2024-09-20T20:30:00.498+0000 7f50b10dc640 -1 osd.7 160 *** Got signal Terminated *** 2024-09-20T20:30:00.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[42208]: 2024-09-20T20:30:00.498+0000 7f50b10dc640 -1 osd.7 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:30:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[91136]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[91136]: overall HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[91136]: osd.7 marked itself down and dead 2024-09-20T20:30:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:30:00] "GET /metrics HTTP/1.1" 200 38836 "" "Prometheus/2.43.0" 2024-09-20T20:30:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[99959]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[99959]: overall HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:00 smithi078 ceph-mon[99959]: osd.7 marked itself down and dead 2024-09-20T20:30:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-mon[103278]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-mon[103278]: overall HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:00 smithi146 ceph-mon[103278]: osd.7 marked itself down and dead 2024-09-20T20:30:01.657 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:01 smithi146 podman[134122]: 2024-09-20 20:30:01.346171106 +0000 UTC m=+0.973618418 container died 3c0d9a82257fb157a470dcdcb07b9fd07d457c547935063818480cb9055cec05 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-0b2cc4d, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T20:30:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:01 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:30:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:01 smithi146 ceph-mon[103278]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T20:30:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:01 smithi146 ceph-mon[103278]: osdmap e161: 8 total, 7 up, 8 in 2024-09-20T20:30:02.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:01 smithi146 podman[134122]: 2024-09-20 20:30:01.656337608 +0000 UTC m=+1.283784906 container cleanup 3c0d9a82257fb157a470dcdcb07b9fd07d457c547935063818480cb9055cec05 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-0b2cc4d, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T20:30:02.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:01 smithi146 bash[134122]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7 2024-09-20T20:30:02.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:01 smithi146 podman[134136]: 2024-09-20 20:30:01.8744926 +0000 UTC m=+0.530427855 container remove 3c0d9a82257fb157a470dcdcb07b9fd07d457c547935063818480cb9055cec05 (image=quay.ceph.io/ceph-ci/ceph@sha256:60bdb877e719cf1d7b60ecd82aef2943629dccf9486362590057fcf65f14ecce, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-0b2cc4d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T20:30:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:01 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:30:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:01 smithi078 ceph-mon[91136]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T20:30:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:01 smithi078 ceph-mon[91136]: osdmap e161: 8 total, 7 up, 8 in 2024-09-20T20:30:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:01 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:30:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:01 smithi078 ceph-mon[99959]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T20:30:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:01 smithi078 ceph-mon[99959]: osdmap e161: 8 total, 7 up, 8 in 2024-09-20T20:30:02.529 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:02 smithi146 podman[134225]: 2024-09-20 20:30:02.179843361 +0000 UTC m=+0.016208593 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:30:02.529 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:02 smithi146 podman[134225]: 2024-09-20 20:30:02.283590401 +0000 UTC m=+0.119955638 container create 2d75a613a0b27df260ed2fded8936d8691590a0b7779584a55f4e5b468880cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-20T20:30:02.814 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:02 smithi146 podman[134225]: 2024-09-20 20:30:02.527789197 +0000 UTC m=+0.364154429 container init 2d75a613a0b27df260ed2fded8936d8691590a0b7779584a55f4e5b468880cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T20:30:02.814 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:02 smithi146 podman[134225]: 2024-09-20 20:30:02.531177496 +0000 UTC m=+0.367542731 container start 2d75a613a0b27df260ed2fded8936d8691590a0b7779584a55f4e5b468880cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T20:30:02.814 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:02 smithi146 podman[134225]: 2024-09-20 20:30:02.610928611 +0000 UTC m=+0.447293858 container attach 2d75a613a0b27df260ed2fded8936d8691590a0b7779584a55f4e5b468880cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-20T20:30:02.815 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:02 smithi146 podman[134225]: 2024-09-20 20:30:02.725085591 +0000 UTC m=+0.561450824 container died 2d75a613a0b27df260ed2fded8936d8691590a0b7779584a55f4e5b468880cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T20:30:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:02 smithi078 ceph-mon[99959]: pgmap v294: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:30:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:02 smithi078 ceph-mon[99959]: osdmap e162: 8 total, 7 up, 8 in 2024-09-20T20:30:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:02 smithi078 ceph-mon[91136]: pgmap v294: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:30:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:02 smithi078 ceph-mon[91136]: osdmap e162: 8 total, 7 up, 8 in 2024-09-20T20:30:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:02 smithi146 ceph-mon[103278]: pgmap v294: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:30:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:02 smithi146 ceph-mon[103278]: osdmap e162: 8 total, 7 up, 8 in 2024-09-20T20:30:03.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 podman[134247]: 2024-09-20 20:30:03.331615043 +0000 UTC m=+0.600561672 container remove 2d75a613a0b27df260ed2fded8936d8691590a0b7779584a55f4e5b468880cb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T20:30:03.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.7.service: Deactivated successfully. 2024-09-20T20:30:03.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 systemd[1]: Stopped Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:30:03.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.7.service: Consumed 10.572s CPU time. 2024-09-20T20:30:03.767 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 systemd[1]: Starting Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:30:04.053 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:03 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 35/633 objects degraded (5.529%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:04.053 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 podman[134348]: 2024-09-20 20:30:03.673056904 +0000 UTC m=+0.016294061 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:30:04.053 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:03 smithi146 podman[134348]: 2024-09-20 20:30:03.800413269 +0000 UTC m=+0.143650485 container create f218baf731c28137f8b26e9eeba15c849dadad676743ca407c486794d4118787 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T20:30:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:03 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 35/633 objects degraded (5.529%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:03 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 35/633 objects degraded (5.529%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 podman[134348]: 2024-09-20 20:30:04.051645973 +0000 UTC m=+0.394883135 container init f218baf731c28137f8b26e9eeba15c849dadad676743ca407c486794d4118787 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 podman[134348]: 2024-09-20 20:30:04.055077289 +0000 UTC m=+0.398314446 container start f218baf731c28137f8b26e9eeba15c849dadad676743ca407c486794d4118787 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=) 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 podman[134348]: 2024-09-20 20:30:04.143414024 +0000 UTC m=+0.486651199 container attach f218baf731c28137f8b26e9eeba15c849dadad676743ca407c486794d4118787 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:04.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:04.942 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-mon[103278]: pgmap v296: 161 pgs: 11 active+undersized, 19 stale+active+clean, 11 active+undersized+degraded, 120 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 35/633 objects degraded (5.529%) 2024-09-20T20:30:04.942 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:04.942 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:04.942 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:04.942 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:04.942 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:05.240 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[134363]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 bash[134348]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T20:30:05.241 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:04 smithi146 podman[134348]: 2024-09-20 20:30:04.968223136 +0000 UTC m=+1.311460291 container died f218baf731c28137f8b26e9eeba15c849dadad676743ca407c486794d4118787 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T20:30:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:04 smithi078 ceph-mon[91136]: pgmap v296: 161 pgs: 11 active+undersized, 19 stale+active+clean, 11 active+undersized+degraded, 120 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 35/633 objects degraded (5.529%) 2024-09-20T20:30:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:04 smithi078 ceph-mon[99959]: pgmap v296: 161 pgs: 11 active+undersized, 19 stale+active+clean, 11 active+undersized+degraded, 120 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 35/633 objects degraded (5.529%) 2024-09-20T20:30:05.895 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:05 smithi146 podman[134499]: 2024-09-20 20:30:05.564228994 +0000 UTC m=+0.589605608 container remove f218baf731c28137f8b26e9eeba15c849dadad676743ca407c486794d4118787 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T20:30:05.895 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:05 smithi146 podman[134558]: 2024-09-20 20:30:05.673108249 +0000 UTC m=+0.015915520 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:30:05.895 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:05 smithi146 podman[134558]: 2024-09-20 20:30:05.781795768 +0000 UTC m=+0.124603029 container create e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T20:30:06.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:06 smithi146 podman[134558]: 2024-09-20 20:30:06.017621351 +0000 UTC m=+0.360428621 container init e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:30:06.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:06 smithi146 podman[134558]: 2024-09-20 20:30:06.020718928 +0000 UTC m=+0.363526190 container start e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-20T20:30:06.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-osd[134577]: -- 172.21.15.146:0/3215834388 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef7f0e3a40 con 0x55ef7f0c1c00 2024-09-20T20:30:06.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:06 smithi146 bash[134558]: e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 2024-09-20T20:30:06.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:06 smithi146 systemd[1]: Started Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: pgmap v297: 161 pgs: 18 active+undersized, 16 stale+active+clean, 11 active+undersized+degraded, 116 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 35/633 objects degraded (5.529%) 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: pgmap v297: 161 pgs: 18 active+undersized, 16 stale+active+clean, 11 active+undersized+degraded, 116 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 35/633 objects degraded (5.529%) 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: pgmap v297: 161 pgs: 18 active+undersized, 16 stale+active+clean, 11 active+undersized+degraded, 116 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 35/633 objects degraded (5.529%) 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:07.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.016 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[134573]: 2024-09-20T20:30:07.671+0000 7ff7562c3640 -1 Falling back to public interface 2024-09-20T20:30:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:08.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: pgmap v298: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: pgmap v298: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: pgmap v298: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:09.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:09.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:09.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:10.298 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:10.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:30:10] "GET /metrics HTTP/1.1" 200 38836 "" "Prometheus/2.43.0" 2024-09-20T20:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: pgmap v299: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.273 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: pgmap v299: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: pgmap v299: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.698 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[134573]: 2024-09-20T20:30:11.268+0000 7ff7562c3640 -1 osd.7 160 log_to_monitors true 2024-09-20T20:30:11.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:11.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.997 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:11.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:30:12.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:12.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:12.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:12.766 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[134573]: 2024-09-20T20:30:12.294+0000 7ff74a065640 -1 osd.7 160 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: pgmap v300: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: osdmap e163: 8 total, 7 up, 8 in 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:30:13.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.230 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: pgmap v300: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: pgmap v300: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: osdmap e163: 8 total, 7 up, 8 in 2024-09-20T20:30:13.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: osdmap e163: 8 total, 7 up, 8 in 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:13.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782] boot 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: osdmap e164: 8 total, 8 up, 8 in 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:14.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782] boot 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: osdmap e164: 8 total, 8 up, 8 in 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.301 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:14.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:14.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.305 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:30:14.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: osd.7 [v2:172.21.15.146:6824/950922782,v1:172.21.15.146:6825/950922782] boot 2024-09-20T20:30:14.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: osdmap e164: 8 total, 8 up, 8 in 2024-09-20T20:30:14.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:30:14.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:14.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:14.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:14.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:14.312 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: pgmap v302: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: osdmap e165: 8 total, 8 up, 8 in 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: pgmap v302: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: pgmap v302: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 84/633 objects degraded (13.270%) 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: osdmap e165: 8 total, 8 up, 8 in 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: osdmap e165: 8 total, 8 up, 8 in 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.043 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:30:16.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.082 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.082 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.082 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.292 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.294 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:16.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 20m - - 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 20m - - 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (15m) 8s ago 16m 25.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (15m) 18s ago 15m 87.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (8m) 8s ago 13m 61.9M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7m) 18s ago 20m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:30:16.612 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (9m) 8s ago 21m 523M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7m) 8s ago 21m 75.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6m) 18s ago 20m 51.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6m) 8s ago 20m 56.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (15m) 8s ago 15m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (15m) 18s ago 15m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5m) 8s ago 19m 54.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4m) 8s ago 19m 53.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3m) 8s ago 18m 48.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3m) 8s ago 18m 50.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2m) 18s ago 18m 52.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (84s) 18s ago 17m 43.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (38s) 18s ago 17m 49.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 starting - - - 4096M 2024-09-20T20:30:16.613 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (8m) 18s ago 16m 56.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (14m) 8s ago 14m 83.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (14m) 18s ago 14m 83.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (14m) 8s ago 14m 84.2M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (14m) 18s ago 14m 86.5M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:30:16.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: pgmap v305: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:16.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:30:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4, 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 13 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:30:16.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr" 2024-09-20T20:30:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:30:17.280 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "12/23 daemons upgraded", 2024-09-20T20:30:17.280 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T20:30:17.280 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:30:17.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:30:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: pgmap v305: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: pgmap v305: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.289 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-09-20T20:30:17.636 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-09-20T20:30:17.637 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-09-20T20:30:17.638 INFO:teuthology.orchestra.run.smithi078.stdout: pg 6.3 is active+undersized+degraded, acting [6,2] 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='client.64565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='client.64577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1812145784' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2713414025' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:17.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='client.64565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='client.64577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1812145784' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='client.64565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='client.44643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='client.64577 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1812145784' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.290 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2713414025' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2713414025' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:19.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: pgmap v306: 161 pgs: 27 active+undersized, 18 active+undersized+degraded, 116 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:19.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: from='client.44661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:30:19.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:18 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 61/633 objects degraded (9.637%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:19.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: pgmap v306: 161 pgs: 27 active+undersized, 18 active+undersized+degraded, 116 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: from='client.44661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 61/633 objects degraded (9.637%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: pgmap v306: 161 pgs: 27 active+undersized, 18 active+undersized+degraded, 116 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: from='client.44661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16} : dispatch 2024-09-20T20:30:19.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:18 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 61/633 objects degraded (9.637%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:19 smithi146 ceph-mon[103278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:30:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:19 smithi146 ceph-mon[103278]: Upgrade: osd.7 is safe to restart 2024-09-20T20:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:19 smithi078 ceph-mon[91136]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:19 smithi078 ceph-mon[91136]: Upgrade: osd.7 is safe to restart 2024-09-20T20:30:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:19 smithi078 ceph-mon[99959]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T20:30:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:19 smithi078 ceph-mon[99959]: Upgrade: osd.7 is safe to restart 2024-09-20T20:30:20.907 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:30:20] "GET /metrics HTTP/1.1" 200 38920 "" "Prometheus/2.43.0" 2024-09-20T20:30:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:20 smithi146 ceph-mon[103278]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T20:30:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:20 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/633 objects degraded (9.637%), 18 pgs degraded) 2024-09-20T20:30:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-mon[91136]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T20:30:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/633 objects degraded (9.637%), 18 pgs degraded) 2024-09-20T20:30:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-mon[99959]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T20:30:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:20 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/633 objects degraded (9.637%), 18 pgs degraded) 2024-09-20T20:30:22.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:22 smithi146 ceph-mon[103278]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:30:22.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:30:22.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:22.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[91136]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[99959]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:30:22.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-20T20:30:22.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:22.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:22.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:23.591 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:23 smithi078 ceph-mon[99959]: Upgrade: Updating osd.7 2024-09-20T20:30:23.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:23 smithi078 ceph-mon[99959]: Deploying daemon osd.7 on smithi146 2024-09-20T20:30:23.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:23 smithi078 ceph-mon[91136]: Upgrade: Updating osd.7 2024-09-20T20:30:23.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:23 smithi078 ceph-mon[91136]: Deploying daemon osd.7 on smithi146 2024-09-20T20:30:23.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:23 smithi146 ceph-mon[103278]: Upgrade: Updating osd.7 2024-09-20T20:30:23.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:23 smithi146 ceph-mon[103278]: Deploying daemon osd.7 on smithi146 2024-09-20T20:30:24.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:24 smithi146 ceph-mon[103278]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:30:24.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:24.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:24 smithi078 ceph-mon[91136]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:30:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:24 smithi078 ceph-mon[99959]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T20:30:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:25.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:26.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:26 smithi146 ceph-mon[103278]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:30:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:26 smithi078 ceph-mon[91136]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:30:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:26 smithi078 ceph-mon[99959]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T20:30:28.135 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:27 smithi146 systemd[1]: Stopping Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:30:28.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[134573]: 2024-09-20T20:30:28.132+0000 7ff75325b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T20:30:28.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[134573]: 2024-09-20T20:30:28.132+0000 7ff75325b640 -1 osd.7 165 *** Got signal Terminated *** 2024-09-20T20:30:28.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[134573]: 2024-09-20T20:30:28.132+0000 7ff75325b640 -1 osd.7 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: osd.7 marked itself down and dead 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: osd.7 marked itself down and dead 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: osd.7 marked itself down and dead 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:29.810 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:29 smithi146 podman[138821]: 2024-09-20 20:30:29.53184737 +0000 UTC m=+1.503001818 container died e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T20:30:29.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:29 smithi146 ceph-mon[103278]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:30:29.811 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:29 smithi146 ceph-mon[103278]: osdmap e166: 8 total, 7 up, 8 in 2024-09-20T20:30:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:29 smithi078 ceph-mon[99959]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:30:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:29 smithi078 ceph-mon[99959]: osdmap e166: 8 total, 7 up, 8 in 2024-09-20T20:30:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:29 smithi078 ceph-mon[91136]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T20:30:30.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:29 smithi078 ceph-mon[91136]: osdmap e166: 8 total, 7 up, 8 in 2024-09-20T20:30:30.202 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:29 smithi146 podman[138821]: 2024-09-20 20:30:29.907414728 +0000 UTC m=+1.878569165 container cleanup e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-20T20:30:30.202 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:29 smithi146 bash[138821]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7 2024-09-20T20:30:30.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:30 smithi146 podman[138836]: 2024-09-20 20:30:30.200462083 +0000 UTC m=+0.665851456 container remove e2d14d3dbfe937cba63548f56e5fad2c0f076de5d0051e18d73a7e0806280cb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.vendor=CentOS) 2024-09-20T20:30:30.838 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:30 smithi146 ceph-mon[103278]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:30:30.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:30.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:30 smithi146 ceph-mon[103278]: osdmap e167: 8 total, 7 up, 8 in 2024-09-20T20:30:30.839 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:30 smithi146 podman[138923]: 2024-09-20 20:30:30.435740826 +0000 UTC m=+0.015918278 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:30:30.839 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:30 smithi146 podman[138923]: 2024-09-20 20:30:30.651339662 +0000 UTC m=+0.231517099 container create 91f721105a63a073ee8ffaa2257ce2ec11a18e332ff8864837345772045ed9e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T20:30:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-mon[91136]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:30:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-mon[91136]: osdmap e167: 8 total, 7 up, 8 in 2024-09-20T20:30:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-mon[99959]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T20:30:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-mon[99959]: osdmap e167: 8 total, 7 up, 8 in 2024-09-20T20:30:31.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:30:30] "GET /metrics HTTP/1.1" 200 38910 "" "Prometheus/2.43.0" 2024-09-20T20:30:31.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:30 smithi146 podman[138923]: 2024-09-20 20:30:30.837126715 +0000 UTC m=+0.417304163 container init 91f721105a63a073ee8ffaa2257ce2ec11a18e332ff8864837345772045ed9e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-20T20:30:31.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:30 smithi146 podman[138923]: 2024-09-20 20:30:30.840687582 +0000 UTC m=+0.420865028 container start 91f721105a63a073ee8ffaa2257ce2ec11a18e332ff8864837345772045ed9e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-20T20:30:31.267 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:30 smithi146 podman[138923]: 2024-09-20 20:30:30.928610024 +0000 UTC m=+0.508787470 container attach 91f721105a63a073ee8ffaa2257ce2ec11a18e332ff8864837345772045ed9e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-20T20:30:31.268 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:31 smithi146 podman[138923]: 2024-09-20 20:30:31.024623995 +0000 UTC m=+0.604801445 container died 91f721105a63a073ee8ffaa2257ce2ec11a18e332ff8864837345772045ed9e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-20T20:30:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:31 smithi146 ceph-mon[103278]: Health check failed: Degraded data redundancy: 44/633 objects degraded (6.951%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:32.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:31 smithi146 podman[138946]: 2024-09-20 20:30:31.582005189 +0000 UTC m=+0.550224852 container remove 91f721105a63a073ee8ffaa2257ce2ec11a18e332ff8864837345772045ed9e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-20T20:30:32.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:31 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.7.service: Deactivated successfully. 2024-09-20T20:30:32.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:31 smithi146 systemd[1]: Stopped Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:30:32.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:31 smithi146 systemd[1]: ceph-f2df6e16-778b-11ef-baf5-efdc52797490@osd.7.service: Consumed 1.825s CPU time. 2024-09-20T20:30:32.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:31 smithi146 systemd[1]: Starting Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490... 2024-09-20T20:30:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:31 smithi078 ceph-mon[91136]: Health check failed: Degraded data redundancy: 44/633 objects degraded (6.951%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:31 smithi078 ceph-mon[99959]: Health check failed: Degraded data redundancy: 44/633 objects degraded (6.951%), 12 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:32.337 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 podman[139044]: 2024-09-20 20:30:31.934457653 +0000 UTC m=+0.015760716 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:30:32.337 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 podman[139044]: 2024-09-20 20:30:32.033195047 +0000 UTC m=+0.114498091 container create 67fc326465974ebc998db99a4f3ade31c63c6553533fe6f3ebd226d153e8c8fe (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T20:30:32.337 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 podman[139044]: 2024-09-20 20:30:32.260778593 +0000 UTC m=+0.342081647 container init 67fc326465974ebc998db99a4f3ade31c63c6553533fe6f3ebd226d153e8c8fe (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1) 2024-09-20T20:30:32.337 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 podman[139044]: 2024-09-20 20:30:32.264018693 +0000 UTC m=+0.345321743 container start 67fc326465974ebc998db99a4f3ade31c63c6553533fe6f3ebd226d153e8c8fe (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:30:32.338 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 podman[139044]: 2024-09-20 20:30:32.335604057 +0000 UTC m=+0.416907122 container attach 67fc326465974ebc998db99a4f3ade31c63c6553533fe6f3ebd226d153e8c8fe (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:30:32.651 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:32.651 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 bash[139044]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:32.651 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:32.651 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 bash[139044]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T20:30:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:32 smithi146 ceph-mon[103278]: pgmap v315: 161 pgs: 12 active+undersized, 10 peering, 12 stale+active+clean, 12 active+undersized+degraded, 115 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 44/633 objects degraded (6.951%) 2024-09-20T20:30:33.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:33.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 bash[139044]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:33.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:33.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:32 smithi146 bash[139044]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:32 smithi078 ceph-mon[91136]: pgmap v315: 161 pgs: 12 active+undersized, 10 peering, 12 stale+active+clean, 12 active+undersized+degraded, 115 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 44/633 objects degraded (6.951%) 2024-09-20T20:30:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:32 smithi078 ceph-mon[99959]: pgmap v315: 161 pgs: 12 active+undersized, 10 peering, 12 stale+active+clean, 12 active+undersized+degraded, 115 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 44/633 objects degraded (6.951%) 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 bash[139044]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 bash[139044]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 bash[139044]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 bash[139044]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate[139059]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 bash[139044]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T20:30:33.485 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 podman[139044]: 2024-09-20 20:30:33.18736825 +0000 UTC m=+1.268671305 container died 67fc326465974ebc998db99a4f3ade31c63c6553533fe6f3ebd226d153e8c8fe (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T20:30:34.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 podman[139190]: 2024-09-20 20:30:33.741293041 +0000 UTC m=+0.547567364 container remove 67fc326465974ebc998db99a4f3ade31c63c6553533fe6f3ebd226d153e8c8fe (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7-activate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T20:30:34.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 podman[139252]: 2024-09-20 20:30:33.851837435 +0000 UTC m=+0.016151909 image pull 82f5f45939a4d8f62db4f02ae9374981c40785acf9bb0102013462d795122b30 quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d 2024-09-20T20:30:34.017 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:33 smithi146 podman[139252]: 2024-09-20 20:30:33.967232885 +0000 UTC m=+0.131547393 container create 515bbfa3c73c53d5d976b2f097916844080dabff00672f6e67957ba59be6dc85 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-20T20:30:34.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:34 smithi146 podman[139252]: 2024-09-20 20:30:34.169629948 +0000 UTC m=+0.333944428 container init 515bbfa3c73c53d5d976b2f097916844080dabff00672f6e67957ba59be6dc85 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T20:30:34.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:34 smithi146 podman[139252]: 2024-09-20 20:30:34.172859491 +0000 UTC m=+0.337173948 container start 515bbfa3c73c53d5d976b2f097916844080dabff00672f6e67957ba59be6dc85 (image=quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d, name=ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-adk-testing-2024-09-20-0910-e8baca1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T20:30:34.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-osd[139271]: -- 172.21.15.146:0/92097686 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561db6687a40 con 0x561db6665c00 2024-09-20T20:30:34.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:34 smithi146 bash[139252]: 515bbfa3c73c53d5d976b2f097916844080dabff00672f6e67957ba59be6dc85 2024-09-20T20:30:34.517 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:34 smithi146 systemd[1]: Started Ceph osd.7 for f2df6e16-778b-11ef-baf5-efdc52797490. 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: pgmap v316: 161 pgs: 23 active+undersized, 10 peering, 9 stale+active+clean, 23 active+undersized+degraded, 96 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 79/633 objects degraded (12.480%) 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: pgmap v316: 161 pgs: 23 active+undersized, 10 peering, 9 stale+active+clean, 23 active+undersized+degraded, 96 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 79/633 objects degraded (12.480%) 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: pgmap v316: 161 pgs: 23 active+undersized, 10 peering, 9 stale+active+clean, 23 active+undersized+degraded, 96 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 79/633 objects degraded (12.480%) 2024-09-20T20:30:35.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.657 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[139266]: 2024-09-20T20:30:35.575+0000 7f1a51af5640 -1 Falling back to public interface 2024-09-20T20:30:35.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:35.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:35.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:35.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:35.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: pgmap v317: 161 pgs: 30 active+undersized, 10 peering, 6 stale+active+clean, 23 active+undersized+degraded, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 79/633 objects degraded (12.480%) 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:36.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: pgmap v317: 161 pgs: 30 active+undersized, 10 peering, 6 stale+active+clean, 23 active+undersized+degraded, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 79/633 objects degraded (12.480%) 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: pgmap v317: 161 pgs: 30 active+undersized, 10 peering, 6 stale+active+clean, 23 active+undersized+degraded, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 79/633 objects degraded (12.480%) 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:38.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: pgmap v318: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: pgmap v318: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: pgmap v318: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 84/633 objects degraded (13.270%), 28 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.453 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[139266]: 2024-09-20T20:30:39.111+0000 7f1a51af5640 -1 osd.7 165 log_to_monitors true 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:30:39.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:39.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:39.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:39.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:40.266 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-osd-7[139266]: 2024-09-20T20:30:40.134+0000 7f1a4989f640 -1 osd.7 165 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T20:30:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: pgmap v319: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: osdmap e168: 8 total, 7 up, 8 in 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:30:40] "GET /metrics HTTP/1.1" 200 38910 "" "Prometheus/2.43.0" 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: pgmap v319: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: osdmap e168: 8 total, 7 up, 8 in 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: pgmap v319: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: osdmap e168: 8 total, 7 up, 8 in 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:41.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735] boot 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: osdmap e169: 8 total, 8 up, 8 in 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735] boot 2024-09-20T20:30:41.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: osdmap e169: 8 total, 8 up, 8 in 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:41.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: osd.7 [v2:172.21.15.146:6824/2512158735,v1:172.21.15.146:6825/2512158735] boot 2024-09-20T20:30:42.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: osdmap e169: 8 total, 8 up, 8 in 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: pgmap v321: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: osdmap e170: 8 total, 8 up, 8 in 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.771 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.773 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: pgmap v321: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: pgmap v321: 161 pgs: 41 active+undersized, 28 active+undersized+degraded, 92 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/633 objects degraded (13.270%) 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.789 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: osdmap e170: 8 total, 8 up, 8 in 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.795 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: osdmap e170: 8 total, 8 up, 8 in 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.796 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:42.801 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:43.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:43.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:43.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:43.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:43.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:44.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: pgmap v324: 161 pgs: 31 active+undersized, 19 active+undersized+degraded, 111 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: Health check update: Degraded data redundancy: 61/633 objects degraded (9.637%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: pgmap v324: 161 pgs: 31 active+undersized, 19 active+undersized+degraded, 111 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: Health check update: Degraded data redundancy: 61/633 objects degraded (9.637%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: pgmap v324: 161 pgs: 31 active+undersized, 19 active+undersized+degraded, 111 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: Health check update: Degraded data redundancy: 61/633 objects degraded (9.637%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T20:30:45.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.042 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.043 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:30:45.047 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Upgrade: Setting container_image for all osd 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-20T20:30:45.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-20T20:30:45.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T20:30:45.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T20:30:45.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-20T20:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Upgrade: Setting container_image for all osd 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T20:30:46.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Upgrade: Setting container_image for all osd 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.0"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.1"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.2"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.3"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.4"} : dispatch 2024-09-20T20:30:46.040 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.5"} : dispatch 2024-09-20T20:30:46.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T20:30:46.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.6"} : dispatch 2024-09-20T20:30:46.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T20:30:46.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd.7"} : dispatch 2024-09-20T20:30:46.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T20:30:46.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T20:30:46.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd require-osd-release", "release": "squid"} : dispatch 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: pgmap v325: 161 pgs: 27 active+undersized, 19 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: osdmap e171: 8 total, 8 up, 8 in 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: Upgrade: Setting container_image for all mds 2024-09-20T20:30:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: pgmap v325: 161 pgs: 27 active+undersized, 19 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: osdmap e171: 8 total, 8 up, 8 in 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: Upgrade: Setting container_image for all mds 2024-09-20T20:30:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:46.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: pgmap v325: 161 pgs: 27 active+undersized, 19 active+undersized+degraded, 115 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/633 objects degraded (9.637%) 2024-09-20T20:30:46.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T20:30:46.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T20:30:46.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: osdmap e171: 8 total, 8 up, 8 in 2024-09-20T20:30:46.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:46.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: Upgrade: Setting container_image for all mds 2024-09-20T20:30:46.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:47.928 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 20s ago 21m - - 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 21m - - 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (15m) 20s ago 16m 25.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (16m) 3s ago 16m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (8m) 20s ago 14m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7m) 3s ago 20m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (9m) 20s ago 22m 524M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7m) 20s ago 22m 78.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6m) 3s ago 21m 54.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:30:48.472 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7m) 20s ago 20m 60.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (16m) 20s ago 16m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (16m) 3s ago 16m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5m) 20s ago 20m 55.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4m) 20s ago 19m 54.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4m) 20s ago 19m 49.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3m) 20s ago 19m 51.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3m) 3s ago 18m 54.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (116s) 3s ago 18m 46.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (69s) 3s ago 17m 50.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (42s) 3s ago 17m 45.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 e2d14d3dbfe9 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (9m) 3s ago 16m 56.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (14m) 20s ago 14m 84.0M - 18.2.4-869-g0b2cc4d4 4404209dfb71 c908e9dd5451 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (14m) 3s ago 14m 83.8M - 18.2.4-869-g0b2cc4d4 4404209dfb71 0071a1442f92 2024-09-20T20:30:48.473 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (14m) 20s ago 14m 84.4M - 18.2.4-869-g0b2cc4d4 4404209dfb71 10942c903d02 2024-09-20T20:30:48.474 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (14m) 3s ago 14m 86.7M - 18.2.4-869-g0b2cc4d4 4404209dfb71 88f20608c8da 2024-09-20T20:30:48.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: Upgrade: Updating rgw.foo.smithi078.fewvbr (1/4) 2024-09-20T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: Deploying daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:48 smithi146 ceph-mon[103278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/633 objects degraded (9.637%), 19 pgs degraded) 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: Upgrade: Updating rgw.foo.smithi078.fewvbr (1/4) 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: Deploying daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[91136]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/633 objects degraded (9.637%), 19 pgs degraded) 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: Upgrade: Updating rgw.foo.smithi078.fewvbr (1/4) 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:48.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi078.fewvbr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:48.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:48.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: Deploying daemon rgw.foo.smithi078.fewvbr on smithi078 2024-09-20T20:30:48.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:48 smithi078 ceph-mon[99959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 61/633 objects degraded (9.637%), 19 pgs degraded) 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:30:48.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 18.2.4-869-g0b2cc4d4 (0b2cc4d4c5d6e81859ac42f8244f80e09ad79f52) reef (stable)": 4, 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 13 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:30:48.794 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:30:49.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:30:49.093 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:30:49.093 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:30:49.093 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:30:49.094 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:30:49.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:30:49.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:30:49.094 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:30:49.094 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:30:49.094 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "13/23 daemons upgraded", 2024-09-20T20:30:49.095 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:30:49.095 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:30:49.095 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:30:49.412 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_WARN Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:49.412 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:49.413 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='client.64604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='client.64610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='client.44688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2678568505' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3594377891' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:49.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='client.64604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='client.64610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='client.44688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2678568505' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:49.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3594377891' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:49.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='client.64604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='client.64610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='client.44688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:49.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:49.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2678568505' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:30:49.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3594377891' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:30:50.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:50 smithi146 ceph-mon[103278]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:30:50.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:50 smithi146 ceph-mon[103278]: from='client.44700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:50.677 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:50.936 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:30:50] "GET /metrics HTTP/1.1" 200 38914 "" "Prometheus/2.43.0" 2024-09-20T20:30:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-mon[91136]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:30:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-mon[91136]: from='client.44700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:50.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-mon[99959]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T20:30:50.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-mon[99959]: from='client.44700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:30:50.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:30:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:52.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:52 smithi146 ceph-mon[103278]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[91136]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[99959]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi078.qphqws", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:53 smithi146 ceph-mon[103278]: Upgrade: Updating rgw.smpl.smithi078.qphqws (2/4) 2024-09-20T20:30:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:53 smithi146 ceph-mon[103278]: Deploying daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:30:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:53 smithi078 ceph-mon[91136]: Upgrade: Updating rgw.smpl.smithi078.qphqws (2/4) 2024-09-20T20:30:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:53 smithi078 ceph-mon[91136]: Deploying daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:30:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:53 smithi078 ceph-mon[99959]: Upgrade: Updating rgw.smpl.smithi078.qphqws (2/4) 2024-09-20T20:30:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:53 smithi078 ceph-mon[99959]: Deploying daemon rgw.smpl.smithi078.qphqws on smithi078 2024-09-20T20:30:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:54.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:54 smithi078 ceph-mon[99959]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T20:30:54.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:54 smithi078 ceph-mon[91136]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T20:30:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:54 smithi146 ceph-mon[103278]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T20:30:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:56 smithi146 ceph-mon[103278]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T20:30:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:56 smithi078 ceph-mon[91136]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T20:30:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:56 smithi078 ceph-mon[99959]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 33 op/s 2024-09-20T20:30:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:58 smithi146 ceph-mon[103278]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 91 B/s wr, 90 op/s 2024-09-20T20:30:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[99959]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 91 B/s wr, 90 op/s 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:30:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[91136]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 91 B/s wr, 90 op/s 2024-09-20T20:30:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:30:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:30:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.dulerg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:30:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:59 smithi146 ceph-mon[103278]: Upgrade: Updating rgw.foo.smithi146.dulerg (3/4) 2024-09-20T20:31:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:30:59 smithi146 ceph-mon[103278]: Deploying daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:31:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:59 smithi078 ceph-mon[91136]: Upgrade: Updating rgw.foo.smithi146.dulerg (3/4) 2024-09-20T20:31:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:30:59 smithi078 ceph-mon[91136]: Deploying daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:31:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:59 smithi078 ceph-mon[99959]: Upgrade: Updating rgw.foo.smithi146.dulerg (3/4) 2024-09-20T20:31:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:30:59 smithi078 ceph-mon[99959]: Deploying daemon rgw.foo.smithi146.dulerg on smithi146 2024-09-20T20:31:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:00 smithi146 ceph-mon[103278]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 85 B/s wr, 111 op/s 2024-09-20T20:31:01.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:00 smithi078 ceph-mon[91136]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 85 B/s wr, 111 op/s 2024-09-20T20:31:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:00 smithi078 ceph-mon[99959]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 85 B/s wr, 111 op/s 2024-09-20T20:31:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:31:00] "GET /metrics HTTP/1.1" 200 38876 "" "Prometheus/2.43.0" 2024-09-20T20:31:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:02 smithi078 ceph-mon[91136]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:02 smithi078 ceph-mon[99959]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:03.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:02 smithi146 ceph-mon[103278]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:03.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:03.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:04.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:04 smithi146 ceph-mon[103278]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-09-20T20:31:04.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:04 smithi146 ceph-mon[103278]: Upgrade: Updating rgw.smpl.smithi146.gfhcht (4/4) 2024-09-20T20:31:04.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:04.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:31:04.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:04.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:04 smithi146 ceph-mon[103278]: Deploying daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:31:04.670 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[91136]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[91136]: Upgrade: Updating rgw.smpl.smithi146.gfhcht (4/4) 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[91136]: Deploying daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[99959]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 127 KiB/s rd, 170 B/s wr, 195 op/s 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[99959]: Upgrade: Updating rgw.smpl.smithi146.gfhcht (4/4) 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.gfhcht", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:04.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:04 smithi078 ceph-mon[99959]: Deploying daemon rgw.smpl.smithi146.gfhcht on smithi146 2024-09-20T20:31:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:06 smithi146 ceph-mon[103278]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:06 smithi078 ceph-mon[99959]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:06 smithi078 ceph-mon[91136]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:08.575 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 255 B/s wr, 224 op/s 2024-09-20T20:31:08.575 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:31:08.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 255 B/s wr, 224 op/s 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:31:08.577 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:31:08.290+0000 7fcfbbcef640 -1 no module 'dashboard' 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:31:08.290+0000 7fcfbbcef640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:31:08.290+0000 7fcfbbcef640 -1 cephadm.serve: 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:31:08.290+0000 7fcfbbcef640 -1 Traceback (most recent call last): 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/module.py", line 866, in serve 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: serve.serve() 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 94, in serve 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: self.mgr.remote('dashboard', 'set_rgw_credentials') 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: File "/usr/share/ceph/mgr/mgr_module.py", line 2376, in remote 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: return self._ceph_dispatch_remote(module_name, method_name, 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ImportError: Module not found 2024-09-20T20:31:08.578 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:08 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: 2024-09-20T20:31:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 255 B/s wr, 224 op/s 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-20T20:31:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-20T20:31:09.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:31:09.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: Checking dashboard <-> RGW credentials 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: Checking dashboard <-> RGW credentials 2024-09-20T20:31:09.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:31:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:09.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: Checking dashboard <-> RGW credentials 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: Unhandled exception from module 'cephadm' while running on mgr.y: Module not found 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:09 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:31:09.968+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:31:10] "GET /metrics HTTP/1.1" 200 38876 "" "Prometheus/2.43.0" 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[99959]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[99959]: Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[91136]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[91136]: Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:31:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:10 smithi146 ceph-mon[103278]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:10.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:10.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:10 smithi146 ceph-mon[103278]: Health check failed: Module 'cephadm' has failed: Module not found (MGR_MODULE_ERROR) 2024-09-20T20:31:10.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:10.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:12 smithi146 ceph-mon[103278]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 255 B/s wr, 197 op/s 2024-09-20T20:31:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[91136]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 255 B/s wr, 197 op/s 2024-09-20T20:31:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[99959]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 255 B/s wr, 197 op/s 2024-09-20T20:31:12.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:12.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:14 smithi146 ceph-mon[103278]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:14.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:14 smithi078 ceph-mon[91136]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:14.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:14.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:14.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:14 smithi078 ceph-mon[99959]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T20:31:14.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:14.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:15.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:16 smithi146 ceph-mon[103278]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-09-20T20:31:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:16 smithi078 ceph-mon[91136]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-09-20T20:31:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:16 smithi078 ceph-mon[99959]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-09-20T20:31:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:18 smithi146 ceph-mon[103278]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-09-20T20:31:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:18 smithi078 ceph-mon[91136]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-09-20T20:31:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:18 smithi078 ceph-mon[99959]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 170 B/s wr, 141 op/s 2024-09-20T20:31:19.708 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 21m - - 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 21m - - 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (16m) 10s ago 17m 25.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (16m) 5s ago 16m 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (9m) 10s ago 14m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (8m) 5s ago 21m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (10m) 10s ago 22m 526M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (8m) 10s ago 22m 80.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7m) 5s ago 21m 58.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7m) 10s ago 21m 62.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:31:20.231 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (16m) 10s ago 16m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (16m) 5s ago 16m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6m) 10s ago 20m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5m) 10s ago 20m 57.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4m) 10s ago 20m 52.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4m) 10s ago 19m 55.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3m) 5s ago 19m 58.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2m) 5s ago 18m 49.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (101s) 5s ago 18m 53.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (46s) 5s ago 18m 47.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (9m) 5s ago 17m 54.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (29s) 10s ago 15m 75.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (18s) 5s ago 15m 76.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (24s) 10s ago 15m 76.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:31:20.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (12s) 5s ago 15m 77.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:31:20.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:31:20.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:31:20.537 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:31:20.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:31:20.539 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:31:20.809 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:31:20.810 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:31:20.811 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:31:20.811 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:31:20] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:31:20.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[99959]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:31:20.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:20.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[99959]: from='client.44802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:20.811 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2153642569' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:20.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[91136]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:31:20.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:20.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[91136]: from='client.44802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:20.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2153642569' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:20 smithi146 ceph-mon[103278]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:20 smithi146 ceph-mon[103278]: from='client.44802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2153642569' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:21.116 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:31:21.116 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:21.116 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:21.116 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:21.116 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:21 smithi146 ceph-mon[103278]: from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:21 smithi146 ceph-mon[103278]: from='client.44814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:21 smithi146 ceph-mon[103278]: from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/110682280' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[91136]: from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[91136]: from='client.44814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[91136]: from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/110682280' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[99959]: from='client.44808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[99959]: from='client.44814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[99959]: from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/110682280' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:22 smithi146 ceph-mon[103278]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:31:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:22 smithi078 ceph-mon[91136]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:31:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:22 smithi078 ceph-mon[99959]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T20:31:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:24 smithi146 ceph-mon[103278]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T20:31:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:24 smithi078 ceph-mon[91136]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T20:31:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:24 smithi078 ceph-mon[99959]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T20:31:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:26 smithi146 ceph-mon[103278]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:26 smithi078 ceph-mon[91136]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:26 smithi078 ceph-mon[99959]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:28 smithi146 ceph-mon[103278]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:28 smithi078 ceph-mon[91136]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:29.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:28 smithi078 ceph-mon[99959]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:29.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:31:30] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:31:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[99959]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[91136]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:30 smithi146 ceph-mon[103278]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:30.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:30.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:30.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:32.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:32 smithi146 ceph-mon[103278]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:32 smithi078 ceph-mon[91136]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:32 smithi078 ceph-mon[99959]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:34 smithi146 ceph-mon[103278]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:34 smithi078 ceph-mon[91136]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:34 smithi078 ceph-mon[99959]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:36 smithi078 ceph-mon[99959]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:36 smithi078 ceph-mon[91136]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:36 smithi146 ceph-mon[103278]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:38 smithi146 ceph-mon[103278]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:38 smithi078 ceph-mon[91136]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:38 smithi078 ceph-mon[99959]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:31:40] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:31:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:40 smithi078 ceph-mon[91136]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:40 smithi078 ceph-mon[99959]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:40 smithi146 ceph-mon[103278]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:42 smithi146 ceph-mon[103278]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:42 smithi078 ceph-mon[91136]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:42 smithi078 ceph-mon[99959]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:31:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:31:43.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:31:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:31:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:31:43.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:31:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:45.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:44 smithi146 ceph-mon[103278]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:44 smithi078 ceph-mon[91136]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:44 smithi078 ceph-mon[99959]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:46 smithi146 ceph-mon[103278]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:46 smithi078 ceph-mon[91136]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:46 smithi078 ceph-mon[99959]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:48 smithi146 ceph-mon[103278]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:48 smithi078 ceph-mon[91136]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:48 smithi078 ceph-mon[99959]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:50.746 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:31:50] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T20:31:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[91136]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[99959]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:50 smithi146 ceph-mon[103278]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:31:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:51.404 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 22m - - 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 22m - - 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (16m) 1s ago 17m 25.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (17m) 17s ago 17m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (9m) 1s ago 15m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (8m) 17s ago 21m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (10m) 1s ago 23m 526M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (9m) 1s ago 23m 82.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7m) 17s ago 22m 59.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (8m) 1s ago 21m 63.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (17m) 1s ago 17m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (17m) 17s ago 17m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6m) 1s ago 21m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5m) 1s ago 20m 58.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5m) 1s ago 20m 53.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4m) 1s ago 20m 56.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4m) 17s ago 19m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3m) 17s ago 19m 49.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2m) 17s ago 19m 53.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (78s) 17s ago 18m 47.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (10m) 17s ago 18m 54.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (61s) 1s ago 15m 75.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (50s) 17s ago 15m 76.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (55s) 1s ago 15m 76.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (44s) 17s ago 15m 77.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:31:52.278 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:31:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:31:52.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:31:52.558 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:31:52.558 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:31:52.558 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:31:52.558 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:31:52.888 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:31:52.888 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:31:52.888 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:31:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:52 smithi146 ceph-mon[103278]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:52 smithi146 ceph-mon[103278]: from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:52 smithi146 ceph-mon[103278]: from='client.64766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1805076275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[91136]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[91136]: from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[91136]: from='client.64766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1805076275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[99959]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[99959]: from='client.44838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[99959]: from='client.64766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1805076275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:31:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:31:53.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:31:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:31:53.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:31:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:31:53.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:31:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:53 smithi146 ceph-mon[103278]: from='client.44850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:53 smithi146 ceph-mon[103278]: from='client.44862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3799865360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-mon[99959]: from='client.44850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-mon[99959]: from='client.44862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3799865360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-mon[91136]: from='client.44850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-mon[91136]: from='client.44862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:31:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3799865360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:31:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:54 smithi146 ceph-mon[103278]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:54 smithi078 ceph-mon[99959]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:54 smithi078 ceph-mon[91136]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:31:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:31:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:56 smithi146 ceph-mon[103278]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:56 smithi078 ceph-mon[91136]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:56 smithi078 ceph-mon[99959]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:58 smithi146 ceph-mon[103278]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:31:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:58 smithi078 ceph-mon[99959]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:31:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:58 smithi078 ceph-mon[91136]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:31:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:00.919 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:32:00] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T20:32:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:00 smithi146 ceph-mon[103278]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:00 smithi078 ceph-mon[99959]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:00 smithi078 ceph-mon[91136]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:01 smithi146 ceph-mon[103278]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:01 smithi078 ceph-mon[99959]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:01 smithi078 ceph-mon[91136]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:03.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:03.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:03.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:04 smithi146 ceph-mon[103278]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:04 smithi078 ceph-mon[99959]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:04 smithi078 ceph-mon[91136]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:06 smithi146 ceph-mon[103278]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:06 smithi078 ceph-mon[91136]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:06 smithi078 ceph-mon[99959]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:08 smithi146 ceph-mon[103278]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:08 smithi078 ceph-mon[91136]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:08 smithi078 ceph-mon[99959]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:10 smithi146 ceph-mon[103278]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:10 smithi078 ceph-mon[99959]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:10 smithi078 ceph-mon[91136]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:32:10] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T20:32:11.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:11.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:12.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:12 smithi078 ceph-mon[91136]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:12.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:12 smithi078 ceph-mon[99959]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:12.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:12 smithi146 ceph-mon[103278]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:13.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:13.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:13.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:13.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:13.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:15.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:14 smithi146 ceph-mon[103278]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:14 smithi078 ceph-mon[99959]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:14 smithi078 ceph-mon[91136]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:16 smithi146 ceph-mon[103278]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[91136]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[99959]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:18 smithi146 ceph-mon[103278]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:18 smithi078 ceph-mon[91136]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:18 smithi078 ceph-mon[99959]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:20 smithi146 ceph-mon[103278]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:20 smithi078 ceph-mon[91136]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:20 smithi078 ceph-mon[99959]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:32:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:32:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:22 smithi146 ceph-mon[103278]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:22 smithi078 ceph-mon[91136]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:22 smithi078 ceph-mon[99959]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:23.183 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:32:23.707 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:32:23.707 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 23m - - 2024-09-20T20:32:23.707 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 22m - - 2024-09-20T20:32:23.707 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (17m) 13s ago 18m 25.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:32:23.707 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (18m) 8s ago 17m 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:32:23.708 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (10m) 13s ago 15m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:32:23.708 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (9m) 8s ago 22m 451M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:32:23.708 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (11m) 13s ago 23m 527M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:32:23.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (9m) 13s ago 24m 83.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:32:23.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (8m) 8s ago 22m 60.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:32:23.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (8m) 13s ago 22m 64.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:32:23.709 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (17m) 13s ago 17m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:32:23.709 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (17m) 8s ago 17m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:32:23.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7m) 13s ago 21m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:32:23.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6m) 13s ago 21m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:32:23.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5m) 13s ago 21m 53.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:32:23.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5m) 13s ago 20m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:32:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4m) 8s ago 20m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:32:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3m) 8s ago 19m 50.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:32:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2m) 8s ago 19m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:32:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (109s) 8s ago 19m 47.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:32:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (10m) 8s ago 18m 55.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:32:23.710 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (92s) 13s ago 16m 75.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:32:23.711 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (82s) 8s ago 16m 76.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:32:23.711 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (87s) 13s ago 16m 76.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:32:23.711 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (76s) 8s ago 16m 78.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:32:24.014 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:24.015 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:32:24.016 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:32:24.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:32:24.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:32:24.016 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:23.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:24.016 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:23.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:24.016 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:23.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:24.284 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:32:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:32:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:32:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:32:24.284 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:32:24.285 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:32:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[99959]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[99959]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[99959]: from='client.64802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[99959]: from='client.44886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1467918611' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[91136]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[91136]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[91136]: from='client.64802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[91136]: from='client.44886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1467918611' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:24 smithi146 ceph-mon[103278]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:24 smithi146 ceph-mon[103278]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:24 smithi146 ceph-mon[103278]: from='client.64802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:24 smithi146 ceph-mon[103278]: from='client.44886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1467918611' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:24.594 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:32:24.594 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:24.594 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:24.594 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:24.594 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:25 smithi146 ceph-mon[103278]: from='client.64820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:25.460 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3034492176' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:25 smithi078 ceph-mon[91136]: from='client.64820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3034492176' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:25 smithi078 ceph-mon[99959]: from='client.64820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3034492176' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:26 smithi146 ceph-mon[103278]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:26 smithi078 ceph-mon[91136]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:26 smithi078 ceph-mon[99959]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:28 smithi146 ceph-mon[103278]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:28 smithi078 ceph-mon[91136]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:28 smithi078 ceph-mon[99959]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:30 smithi146 ceph-mon[103278]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:32:30] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T20:32:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:30 smithi078 ceph-mon[99959]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:30 smithi078 ceph-mon[91136]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:32 smithi146 ceph-mon[103278]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:33.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:32 smithi078 ceph-mon[91136]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:33.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:32 smithi078 ceph-mon[99959]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:33.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:33.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:33.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:33.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:33.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:33.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:34.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:34 smithi146 ceph-mon[103278]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:34.856 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:34 smithi078 ceph-mon[99959]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:34.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:34 smithi078 ceph-mon[91136]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:36.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:36 smithi146 ceph-mon[103278]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:36.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[91136]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[99959]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:38 smithi146 ceph-mon[103278]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:38 smithi078 ceph-mon[99959]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:38 smithi078 ceph-mon[91136]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:40 smithi146 ceph-mon[103278]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:40 smithi078 ceph-mon[91136]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:40 smithi078 ceph-mon[99959]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:32:40] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T20:32:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:42 smithi146 ceph-mon[103278]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:42 smithi078 ceph-mon[91136]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:42 smithi078 ceph-mon[99959]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:43.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:43.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:43.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:44 smithi146 ceph-mon[103278]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:44 smithi078 ceph-mon[91136]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:44 smithi078 ceph-mon[99959]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:46 smithi146 ceph-mon[103278]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:46 smithi078 ceph-mon[91136]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:46 smithi078 ceph-mon[99959]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:48 smithi146 ceph-mon[103278]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:48 smithi078 ceph-mon[99959]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:48 smithi078 ceph-mon[91136]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:50 smithi146 ceph-mon[103278]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:50 smithi078 ceph-mon[99959]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:50 smithi078 ceph-mon[91136]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:32:50.865 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:32:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T20:32:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:53.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:52 smithi146 ceph-mon[103278]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:53.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:52 smithi078 ceph-mon[91136]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:53.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:52 smithi078 ceph-mon[99959]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:53.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:53.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:32:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:32:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:32:53.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:32:54.889 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:32:55.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:54 smithi146 ceph-mon[103278]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:55.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:54 smithi078 ceph-mon[99959]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:55.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:54 smithi078 ceph-mon[91136]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:32:55.424 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 23m - - 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 23m - - 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (17m) 4s ago 18m 26.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (18m) 0s ago 18m 87.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (11m) 4s ago 16m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (9m) 0s ago 22m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (11m) 4s ago 24m 528M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (10m) 4s ago 24m 84.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (8m) 0s ago 23m 61.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (9m) 4s ago 23m 65.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (18m) 4s ago 18m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:32:55.425 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (18m) 0s ago 18m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7m) 4s ago 22m 59.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6m) 4s ago 22m 58.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6m) 4s ago 21m 53.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5m) 4s ago 21m 56.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5m) 0s ago 20m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4m) 0s ago 20m 50.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3m) 0s ago 20m 54.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2m) 0s ago 19m 48.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (11m) 0s ago 19m 54.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2m) 4s ago 16m 76.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (113s) 0s ago 16m 77.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (119s) 4s ago 16m 77.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:32:55.426 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (107s) 0s ago 16m 78.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:32:55.737 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:32:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:32:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:32:55.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:32:55.738 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:32:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:32:55.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:32:55.739 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:32:56.012 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:32:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:32:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:32:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:32:56.013 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[99959]: from='client.64832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2992298819' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[91136]: from='client.64832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2992298819' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:56.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:55 smithi146 ceph-mon[103278]: from='client.64832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:56.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:32:56.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2992298819' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:32:56.320 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:32:56.320 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:32:56.320 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:56.320 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:32:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:56 smithi146 ceph-mon[103278]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:56 smithi146 ceph-mon[103278]: from='client.64838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:56 smithi146 ceph-mon[103278]: from='client.44922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2578639147' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[91136]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[91136]: from='client.64838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[91136]: from='client.44922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2578639147' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[99959]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[99959]: from='client.64838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[99959]: from='client.44922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:57.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2578639147' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:32:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:57 smithi146 ceph-mon[103278]: from='client.64856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:57 smithi078 ceph-mon[91136]: from='client.64856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:57 smithi078 ceph-mon[99959]: from='client.64856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:32:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:58 smithi146 ceph-mon[103278]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:58 smithi078 ceph-mon[91136]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:58 smithi078 ceph-mon[99959]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:00.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:33:00] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:33:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:00 smithi146 ceph-mon[103278]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:00 smithi078 ceph-mon[91136]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:00 smithi078 ceph-mon[99959]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:02 smithi146 ceph-mon[103278]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:02 smithi078 ceph-mon[91136]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:02 smithi078 ceph-mon[99959]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:03.561Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:03.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:03.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:04 smithi146 ceph-mon[103278]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:04 smithi078 ceph-mon[91136]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:04 smithi078 ceph-mon[99959]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:06 smithi146 ceph-mon[103278]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:06 smithi078 ceph-mon[99959]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:06 smithi078 ceph-mon[91136]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:08 smithi146 ceph-mon[103278]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:08 smithi078 ceph-mon[91136]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:08 smithi078 ceph-mon[99959]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:10.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:33:10] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:33:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:10 smithi146 ceph-mon[103278]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:10 smithi078 ceph-mon[91136]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:10 smithi078 ceph-mon[99959]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:11 smithi146 ceph-mon[103278]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[91136]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:11 smithi078 ceph-mon[99959]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:13.893 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:13.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:13.893 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:13.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:13.894 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:13.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:14.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:14 smithi078 ceph-mon[99959]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:14.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:14.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:14 smithi078 ceph-mon[91136]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:14.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:14 smithi146 ceph-mon[103278]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:16 smithi146 ceph-mon[103278]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[91136]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[99959]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:18 smithi146 ceph-mon[103278]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:18 smithi078 ceph-mon[91136]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:18 smithi078 ceph-mon[99959]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:20 smithi146 ceph-mon[103278]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:20 smithi078 ceph-mon[91136]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:33:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:33:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:20 smithi078 ceph-mon[99959]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:22 smithi146 ceph-mon[103278]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:22 smithi078 ceph-mon[91136]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:22 smithi078 ceph-mon[99959]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:23.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:23.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:23.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:24 smithi146 ceph-mon[103278]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:24 smithi078 ceph-mon[91136]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:24 smithi078 ceph-mon[99959]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:26.618 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:33:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:26 smithi146 ceph-mon[103278]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:26 smithi078 ceph-mon[91136]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:26 smithi078 ceph-mon[99959]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:27.144 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:33:27.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 24m - - 2024-09-20T20:33:27.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 23m - - 2024-09-20T20:33:27.144 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (18m) 16s ago 19m 27.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:33:27.144 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (19m) 11s ago 19m 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (11m) 16s ago 17m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (10m) 11s ago 23m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (12m) 16s ago 24m 528M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (10m) 16s ago 25m 85.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (9m) 11s ago 23m 62.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (9m) 16s ago 23m 66.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (18m) 16s ago 18m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (18m) 11s ago 18m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (8m) 16s ago 22m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7m) 16s ago 22m 58.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6m) 16s ago 22m 53.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6m) 16s ago 21m 56.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:33:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5m) 11s ago 21m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4m) 11s ago 20m 51.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3m) 11s ago 20m 54.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2m) 11s ago 20m 48.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (11m) 11s ago 19m 54.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2m) 16s ago 17m 76.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2m) 11s ago 17m 77.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2m) 16s ago 17m 77.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:33:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2m) 11s ago 17m 78.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:33:27.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:33:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:33:27.720 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:33:27.721 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:33:27.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:27 smithi078 ceph-mon[91136]: from='client.64868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:27.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:27 smithi078 ceph-mon[91136]: from='client.64874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:27.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:27 smithi078 ceph-mon[99959]: from='client.64868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:27.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:27 smithi078 ceph-mon[99959]: from='client.64874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:27 smithi146 ceph-mon[103278]: from='client.64868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:27 smithi146 ceph-mon[103278]: from='client.64874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.028 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:33:28.029 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:28.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:28.029 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:28.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[99959]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[99959]: from='client.64880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1304190572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[99959]: from='client.44970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2837103997' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:28.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[91136]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:28.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[91136]: from='client.64880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1304190572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:28.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[91136]: from='client.44970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2837103997' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:28.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:28 smithi146 ceph-mon[103278]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:28 smithi146 ceph-mon[103278]: from='client.64880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1304190572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:33:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:28 smithi146 ceph-mon[103278]: from='client.44970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:33:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2837103997' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:33:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:30 smithi146 ceph-mon[103278]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:30 smithi078 ceph-mon[99959]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:30 smithi078 ceph-mon[91136]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:33:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T20:33:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:32 smithi146 ceph-mon[103278]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[91136]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[99959]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:32.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:33.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:33.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:33.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:34.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:34 smithi146 ceph-mon[103278]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:34 smithi078 ceph-mon[91136]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:34 smithi078 ceph-mon[99959]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:36.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:36 smithi146 ceph-mon[103278]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:36.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.661 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.662 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[91136]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[99959]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:36.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:38 smithi146 ceph-mon[103278]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:38 smithi078 ceph-mon[99959]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:38 smithi078 ceph-mon[91136]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:40 smithi078 ceph-mon[91136]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:33:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T20:33:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:40 smithi078 ceph-mon[99959]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:40 smithi146 ceph-mon[103278]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:42 smithi146 ceph-mon[103278]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:42 smithi078 ceph-mon[91136]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:42 smithi078 ceph-mon[99959]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:43.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:43.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:43.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:44 smithi146 ceph-mon[103278]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:44 smithi078 ceph-mon[99959]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:44 smithi078 ceph-mon[91136]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:46 smithi146 ceph-mon[103278]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:46 smithi078 ceph-mon[99959]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:46 smithi078 ceph-mon[91136]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:48.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:48 smithi078 ceph-mon[91136]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:48.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:48 smithi078 ceph-mon[99959]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:48 smithi146 ceph-mon[103278]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:50 smithi146 ceph-mon[103278]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:50 smithi078 ceph-mon[91136]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:33:50] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:33:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:50 smithi078 ceph-mon[99959]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:33:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:52 smithi146 ceph-mon[103278]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[91136]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[99959]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:53.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:53.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:33:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:33:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:33:53.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:33:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:54 smithi146 ceph-mon[103278]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:54 smithi078 ceph-mon[91136]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:54 smithi078 ceph-mon[99959]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:33:56.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[99959]: pgmap v421: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[91136]: pgmap v421: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:56 smithi146 ceph-mon[103278]: pgmap v421: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:56.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:56.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:33:58.325 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:33:58.857 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:58 smithi078 ceph-mon[91136]: pgmap v422: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:58.857 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:58.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:58 smithi078 ceph-mon[99959]: pgmap v422: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:58.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:58.858 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 24m - - 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 24m - - 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (18m) 7s ago 19m 27.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (19m) 3s ago 19m 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (12m) 7s ago 17m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (10m) 3s ago 23m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (12m) 7s ago 25m 529M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (11m) 7s ago 25m 84.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (9m) 3s ago 24m 59.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 7s ago 24m 63.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (19m) 7s ago 19m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (19m) 3s ago 19m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:33:58.859 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (8m) 7s ago 23m 60.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7m) 7s ago 23m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7m) 7s ago 22m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6m) 7s ago 22m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6m) 3s ago 21m 60.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5m) 3s ago 21m 51.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4m) 3s ago 21m 55.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3m) 3s ago 20m 48.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (12m) 3s ago 20m 53.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3m) 7s ago 17m 76.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2m) 3s ago 17m 77.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3m) 7s ago 17m 77.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:33:58.860 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2m) 3s ago 17m 79.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:33:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:58 smithi146 ceph-mon[103278]: pgmap v422: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:33:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:33:59.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:33:59.165 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:33:59.431 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:33:59.431 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:33:59.431 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:33:59.431 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:33:59.432 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:33:59.742 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:33:59.743 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:59.743 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:33:59.743 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:33:59.743 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:59 smithi146 ceph-mon[103278]: from='client.64904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:59 smithi146 ceph-mon[103278]: from='client.44988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:59 smithi146 ceph-mon[103278]: from='client.64916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:33:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/158106115' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[91136]: from='client.64904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[91136]: from='client.44988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[91136]: from='client.64916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/158106115' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[99959]: from='client.64904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[99959]: from='client.44988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[99959]: from='client.64916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:33:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/158106115' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:00 smithi146 ceph-mon[103278]: pgmap v423: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:00 smithi146 ceph-mon[103278]: from='client.45006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3881298739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:34:00] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T20:34:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[91136]: pgmap v423: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[91136]: from='client.45006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3881298739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[99959]: pgmap v423: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[99959]: from='client.45006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3881298739' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:02 smithi146 ceph-mon[103278]: pgmap v424: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:02 smithi078 ceph-mon[91136]: pgmap v424: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:02 smithi078 ceph-mon[99959]: pgmap v424: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:03.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:03.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:03.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:04 smithi146 ceph-mon[103278]: pgmap v425: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:04 smithi078 ceph-mon[91136]: pgmap v425: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:04 smithi078 ceph-mon[99959]: pgmap v425: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:06 smithi146 ceph-mon[103278]: pgmap v426: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:06 smithi078 ceph-mon[99959]: pgmap v426: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:06 smithi078 ceph-mon[91136]: pgmap v426: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:09.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:08 smithi146 ceph-mon[103278]: pgmap v427: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:08 smithi078 ceph-mon[99959]: pgmap v427: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:08 smithi078 ceph-mon[91136]: pgmap v427: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:10 smithi146 ceph-mon[103278]: pgmap v428: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:10 smithi078 ceph-mon[99959]: pgmap v428: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:10 smithi078 ceph-mon[91136]: pgmap v428: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:34:10] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T20:34:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:12 smithi146 ceph-mon[103278]: pgmap v429: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[91136]: pgmap v429: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[99959]: pgmap v429: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:13.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:13.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:13.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:14 smithi146 ceph-mon[103278]: pgmap v430: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:14 smithi078 ceph-mon[99959]: pgmap v430: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:14 smithi078 ceph-mon[91136]: pgmap v430: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:16 smithi146 ceph-mon[103278]: pgmap v431: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[91136]: pgmap v431: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[99959]: pgmap v431: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:17.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:18 smithi078 ceph-mon[91136]: pgmap v432: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:19.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:18 smithi078 ceph-mon[99959]: pgmap v432: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:18 smithi146 ceph-mon[103278]: pgmap v432: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:20 smithi078 ceph-mon[91136]: pgmap v433: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:20 smithi078 ceph-mon[99959]: pgmap v433: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:34:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:34:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:20 smithi146 ceph-mon[103278]: pgmap v433: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:22 smithi146 ceph-mon[103278]: pgmap v434: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:22 smithi078 ceph-mon[91136]: pgmap v434: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:22 smithi078 ceph-mon[99959]: pgmap v434: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:23.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:23.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:23.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:24 smithi146 ceph-mon[103278]: pgmap v435: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:24 smithi078 ceph-mon[91136]: pgmap v435: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:24 smithi078 ceph-mon[99959]: pgmap v435: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T20:34:25.849213333Z level=info msg="Completed cleanup jobs" duration=13.033988ms 2024-09-20T20:34:27.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:26 smithi146 ceph-mon[103278]: pgmap v436: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:26 smithi078 ceph-mon[91136]: pgmap v436: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:26 smithi078 ceph-mon[99959]: pgmap v436: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:28 smithi146 ceph-mon[103278]: pgmap v437: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:28 smithi078 ceph-mon[99959]: pgmap v437: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:28 smithi078 ceph-mon[91136]: pgmap v437: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:30.029 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 25m - - 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 24m - - 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (19m) 19s ago 20m 27.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (20m) 14s ago 20m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (12m) 19s ago 18m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (11m) 14s ago 24m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (13m) 19s ago 25m 529M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (11m) 19s ago 26m 84.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (10m) 14s ago 24m 59.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (10m) 19s ago 24m 64.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:34:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (19m) 19s ago 19m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (19m) 14s ago 19m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (9m) 19s ago 23m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (8m) 19s ago 23m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7m) 19s ago 23m 54.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7m) 19s ago 22m 57.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6m) 14s ago 22m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5m) 14s ago 22m 51.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4m) 14s ago 21m 55.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3m) 14s ago 21m 49.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (12m) 14s ago 20m 53.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3m) 19s ago 18m 77.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3m) 14s ago 18m 77.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3m) 19s ago 18m 78.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:34:30.556 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3m) 14s ago 18m 79.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:34:30.815 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:34:30] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:34:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:34:30.866 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:34:31.133 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:34:31.134 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:34:31.134 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:34:31.134 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:34:31.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:30 smithi078 ceph-mon[91136]: pgmap v438: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:31.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:31.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:30 smithi078 ceph-mon[99959]: pgmap v438: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:31.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:30 smithi146 ceph-mon[103278]: pgmap v438: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:31.444 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:34:31.444 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:34:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:34:31.444 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:34:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='client.64940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='client.64946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='client.64952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/715300367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2414156884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='client.64940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='client.64946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='client.64952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/715300367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2414156884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='client.64940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='client.64946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='client.64952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/715300367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2414156884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:32.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:32 smithi146 ceph-mon[103278]: pgmap v439: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:32 smithi146 ceph-mon[103278]: from='client.64964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:32 smithi078 ceph-mon[91136]: pgmap v439: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:32 smithi078 ceph-mon[91136]: from='client.64964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:32 smithi078 ceph-mon[99959]: pgmap v439: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:32 smithi078 ceph-mon[99959]: from='client.64964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:34:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:33.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:33.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:33.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:34 smithi146 ceph-mon[103278]: pgmap v440: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:34 smithi078 ceph-mon[91136]: pgmap v440: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:34 smithi078 ceph-mon[99959]: pgmap v440: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[91136]: pgmap v441: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[99959]: pgmap v441: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.175 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:36 smithi146 ceph-mon[103278]: pgmap v441: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:38 smithi146 ceph-mon[103278]: pgmap v442: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:38 smithi078 ceph-mon[91136]: pgmap v442: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:38 smithi078 ceph-mon[99959]: pgmap v442: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:34:40] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:34:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:40 smithi146 ceph-mon[103278]: pgmap v443: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:40 smithi078 ceph-mon[91136]: pgmap v443: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:40 smithi078 ceph-mon[99959]: pgmap v443: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:42 smithi146 ceph-mon[103278]: pgmap v444: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:42 smithi078 ceph-mon[91136]: pgmap v444: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:42 smithi078 ceph-mon[99959]: pgmap v444: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:43.945 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:43.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:43.945 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:43.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:43.945 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:43.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:44 smithi146 ceph-mon[103278]: pgmap v445: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:44 smithi078 ceph-mon[91136]: pgmap v445: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:44 smithi078 ceph-mon[99959]: pgmap v445: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:45 smithi146 ceph-mon[103278]: pgmap v446: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:45 smithi078 ceph-mon[91136]: pgmap v446: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:45 smithi078 ceph-mon[99959]: pgmap v446: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:48 smithi146 ceph-mon[103278]: pgmap v447: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:48 smithi078 ceph-mon[91136]: pgmap v447: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:48 smithi078 ceph-mon[99959]: pgmap v447: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:50 smithi146 ceph-mon[103278]: pgmap v448: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:50 smithi078 ceph-mon[91136]: pgmap v448: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:50 smithi078 ceph-mon[99959]: pgmap v448: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:34:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:34:50] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:34:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:52 smithi146 ceph-mon[103278]: pgmap v449: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[99959]: pgmap v449: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[91136]: pgmap v449: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:53.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:53.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:53.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:34:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:34:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:34:53.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:34:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:54 smithi146 ceph-mon[103278]: pgmap v450: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:54 smithi078 ceph-mon[91136]: pgmap v450: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:54 smithi078 ceph-mon[99959]: pgmap v450: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:34:56.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:56 smithi146 ceph-mon[103278]: pgmap v451: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:56.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:56.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:56.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[91136]: pgmap v451: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[99959]: pgmap v451: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:34:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:58 smithi078 ceph-mon[91136]: pgmap v452: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:58 smithi078 ceph-mon[99959]: pgmap v452: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:34:58.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:58 smithi146 ceph-mon[103278]: pgmap v452: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:34:58.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:00 smithi146 ceph-mon[103278]: pgmap v453: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:00 smithi078 ceph-mon[91136]: pgmap v453: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:35:00] "GET /metrics HTTP/1.1" 200 38981 "" "Prometheus/2.43.0" 2024-09-20T20:35:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:00 smithi078 ceph-mon[99959]: pgmap v453: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:01.741 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 25m - - 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 25m - - 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (19m) 10s ago 20m 27.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (20m) 6s ago 20m 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (13m) 10s ago 18m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:35:02.267 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (11m) 6s ago 24m 452M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (13m) 10s ago 26m 529M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (12m) 10s ago 26m 86.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (10m) 6s ago 25m 61.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (11m) 10s ago 25m 65.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (20m) 10s ago 20m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (20m) 6s ago 20m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (9m) 10s ago 24m 60.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (8m) 10s ago 24m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8m) 10s ago 23m 54.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7m) 10s ago 23m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7m) 6s ago 22m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6m) 6s ago 22m 52.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:35:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5m) 6s ago 22m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:35:02.269 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4m) 6s ago 21m 49.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:35:02.269 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (13m) 6s ago 21m 55.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:35:02.269 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4m) 10s ago 18m 77.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:35:02.269 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4m) 6s ago 19m 78.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:35:02.269 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4m) 10s ago 18m 78.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:35:02.269 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3m) 6s ago 18m 80.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:35:02.578 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:35:02.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:02.579 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:35:02.579 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:35:02.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:35:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:35:02.848 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:35:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:35:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:35:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:35:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:35:02.849 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:35:02.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:02 smithi078 ceph-mon[91136]: pgmap v454: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:02.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:02 smithi078 ceph-mon[91136]: from='client.45054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:02.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3481790678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:02.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:02 smithi078 ceph-mon[99959]: pgmap v454: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:02.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:02 smithi078 ceph-mon[99959]: from='client.45054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:02.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3481790678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:02 smithi146 ceph-mon[103278]: pgmap v454: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:02 smithi146 ceph-mon[103278]: from='client.45054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3481790678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:03.156 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:35:03.156 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:03.156 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:03 smithi146 ceph-mon[103278]: from='client.45060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:03 smithi146 ceph-mon[103278]: from='client.45066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:03 smithi146 ceph-mon[103278]: from='client.65000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/617769299' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[99959]: from='client.45060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[99959]: from='client.45066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[99959]: from='client.65000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/617769299' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[91136]: from='client.45060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[91136]: from='client.45066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[91136]: from='client.65000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/617769299' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:03.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:04 smithi146 ceph-mon[103278]: pgmap v455: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:04 smithi078 ceph-mon[91136]: pgmap v455: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:04 smithi078 ceph-mon[99959]: pgmap v455: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:06 smithi146 ceph-mon[103278]: pgmap v456: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:06 smithi078 ceph-mon[91136]: pgmap v456: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:06 smithi078 ceph-mon[99959]: pgmap v456: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:08 smithi146 ceph-mon[103278]: pgmap v457: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:08 smithi078 ceph-mon[91136]: pgmap v457: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:08 smithi078 ceph-mon[99959]: pgmap v457: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:10 smithi146 ceph-mon[103278]: pgmap v458: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:11.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:10 smithi078 ceph-mon[91136]: pgmap v458: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:11.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:11.018 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:35:10] "GET /metrics HTTP/1.1" 200 38981 "" "Prometheus/2.43.0" 2024-09-20T20:35:11.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:10 smithi078 ceph-mon[99959]: pgmap v458: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:11.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:12 smithi146 ceph-mon[103278]: pgmap v459: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[91136]: pgmap v459: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[99959]: pgmap v459: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:13.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:13.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:13.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:13.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:13.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:14 smithi078 ceph-mon[99959]: pgmap v460: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:14 smithi078 ceph-mon[91136]: pgmap v460: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:14 smithi146 ceph-mon[103278]: pgmap v460: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[99959]: pgmap v461: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[91136]: pgmap v461: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:16 smithi146 ceph-mon[103278]: pgmap v461: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:17.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:17.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:19.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:18 smithi078 ceph-mon[99959]: pgmap v462: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:19.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:18 smithi078 ceph-mon[91136]: pgmap v462: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:19.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:18 smithi146 ceph-mon[103278]: pgmap v462: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:20.781 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:35:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T20:35:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:20 smithi078 ceph-mon[91136]: pgmap v463: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:20 smithi078 ceph-mon[99959]: pgmap v463: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:20 smithi146 ceph-mon[103278]: pgmap v463: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:22 smithi078 ceph-mon[99959]: pgmap v464: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:22 smithi078 ceph-mon[91136]: pgmap v464: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:22 smithi146 ceph-mon[103278]: pgmap v464: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:23.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:23.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:23.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:24 smithi146 ceph-mon[103278]: pgmap v465: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:24 smithi078 ceph-mon[91136]: pgmap v465: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:24 smithi078 ceph-mon[99959]: pgmap v465: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:26 smithi146 ceph-mon[103278]: pgmap v466: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:26 smithi078 ceph-mon[91136]: pgmap v466: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:26 smithi078 ceph-mon[99959]: pgmap v466: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:28 smithi146 ceph-mon[103278]: pgmap v467: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:28 smithi078 ceph-mon[91136]: pgmap v467: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:28 smithi078 ceph-mon[99959]: pgmap v467: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:35:30] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:35:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:30 smithi078 ceph-mon[99959]: pgmap v468: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:30 smithi078 ceph-mon[91136]: pgmap v468: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:30 smithi146 ceph-mon[103278]: pgmap v468: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:32 smithi146 ceph-mon[103278]: pgmap v469: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[99959]: pgmap v469: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[91136]: pgmap v469: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:33.455 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:35:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:33.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:33.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:33.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:33.981 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:35:33.981 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 26m - - 2024-09-20T20:35:33.981 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 25m - - 2024-09-20T20:35:33.981 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (20m) 2s ago 21m 27.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:35:33.981 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (21m) 17s ago 21m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (13m) 2s ago 19m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (12m) 17s ago 25m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (14m) 2s ago 26m 530M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (12m) 2s ago 27m 86.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (11m) 17s ago 25m 59.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (11m) 2s ago 25m 63.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (20m) 2s ago 20m 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (20m) 17s ago 20m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (10m) 2s ago 25m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (9m) 2s ago 24m 59.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (8m) 2s ago 24m 54.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (8m) 2s ago 23m 58.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:35:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7m) 17s ago 23m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6m) 17s ago 23m 52.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5m) 17s ago 22m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5m) 17s ago 22m 49.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (13m) 17s ago 21m 55.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4m) 2s ago 19m 77.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4m) 17s ago 19m 78.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4m) 2s ago 19m 79.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:35:33.983 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4m) 17s ago 19m 80.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:35:34.292 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:35:34.292 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:35:34.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:35:34.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:35:34.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:35:34.294 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:35:34.558 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:35:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:35:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:35:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:35:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:35:34.558 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:35:34.559 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:35:34.865 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:35:34.865 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:35:34.865 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:34.865 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:35:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:34 smithi146 ceph-mon[103278]: pgmap v470: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:34 smithi146 ceph-mon[103278]: from='client.65012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:34 smithi146 ceph-mon[103278]: from='client.45096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2583045715' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2535053231' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[91136]: pgmap v470: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[91136]: from='client.65012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[91136]: from='client.45096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2583045715' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2535053231' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[99959]: pgmap v470: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[99959]: from='client.65012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[99959]: from='client.45096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2583045715' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:35:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2535053231' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:35:36.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:35 smithi146 ceph-mon[103278]: from='client.45102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:36.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:35 smithi146 ceph-mon[103278]: from='client.65036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:35 smithi078 ceph-mon[99959]: from='client.45102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:35 smithi078 ceph-mon[99959]: from='client.65036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:35 smithi078 ceph-mon[91136]: from='client.45102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:35 smithi078 ceph-mon[91136]: from='client.65036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:35:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:36 smithi146 ceph-mon[103278]: pgmap v471: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[99959]: pgmap v471: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[91136]: pgmap v471: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:38 smithi146 ceph-mon[103278]: pgmap v472: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:39.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:38 smithi078 ceph-mon[91136]: pgmap v472: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:39.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:38 smithi078 ceph-mon[99959]: pgmap v472: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:40.844 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:35:40] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:35:41.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:40 smithi078 ceph-mon[99959]: pgmap v473: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:41.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:41.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:40 smithi078 ceph-mon[91136]: pgmap v473: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:41.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:40 smithi146 ceph-mon[103278]: pgmap v473: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:42 smithi146 ceph-mon[103278]: pgmap v474: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:42 smithi078 ceph-mon[91136]: pgmap v474: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:42 smithi078 ceph-mon[99959]: pgmap v474: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:43.915 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:43.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:43.915 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:43.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:43.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:43.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:44 smithi146 ceph-mon[103278]: pgmap v475: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:44 smithi078 ceph-mon[91136]: pgmap v475: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:44 smithi078 ceph-mon[99959]: pgmap v475: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:46 smithi146 ceph-mon[103278]: pgmap v476: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:46 smithi078 ceph-mon[91136]: pgmap v476: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:46 smithi078 ceph-mon[99959]: pgmap v476: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:48 smithi146 ceph-mon[103278]: pgmap v477: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:48 smithi078 ceph-mon[91136]: pgmap v477: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:48 smithi078 ceph-mon[99959]: pgmap v477: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:35:50] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:35:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:50 smithi146 ceph-mon[103278]: pgmap v478: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:50 smithi078 ceph-mon[91136]: pgmap v478: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:50 smithi078 ceph-mon[99959]: pgmap v478: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:35:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:52 smithi146 ceph-mon[103278]: pgmap v479: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[91136]: pgmap v479: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[99959]: pgmap v479: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:53.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:53.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:35:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:35:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:35:53.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:35:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:54 smithi146 ceph-mon[103278]: pgmap v480: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:54 smithi078 ceph-mon[99959]: pgmap v480: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:54 smithi078 ceph-mon[91136]: pgmap v480: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:35:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:56 smithi146 ceph-mon[103278]: pgmap v481: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[99959]: pgmap v481: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[91136]: pgmap v481: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:35:58.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:58 smithi146 ceph-mon[103278]: pgmap v482: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:58.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:58 smithi078 ceph-mon[91136]: pgmap v482: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:58 smithi078 ceph-mon[99959]: pgmap v482: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:00.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:36:00] "GET /metrics HTTP/1.1" 200 38984 "" "Prometheus/2.43.0" 2024-09-20T20:36:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:00 smithi146 ceph-mon[103278]: pgmap v483: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:00 smithi078 ceph-mon[91136]: pgmap v483: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:00 smithi078 ceph-mon[99959]: pgmap v483: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:03.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:02 smithi078 ceph-mon[91136]: pgmap v484: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:03.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:02 smithi078 ceph-mon[99959]: pgmap v484: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:02 smithi146 ceph-mon[103278]: pgmap v484: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:03.981 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:03.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:03.981 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:03.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:03.981 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:03.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:05.160 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:36:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:04 smithi146 ceph-mon[103278]: pgmap v485: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:04 smithi078 ceph-mon[91136]: pgmap v485: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:04 smithi078 ceph-mon[99959]: pgmap v485: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:05.685 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 26m - - 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 26m - - 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (20m) 13s ago 21m 27.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (21m) 9s ago 21m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (14m) 13s ago 19m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (13m) 9s ago 25m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (14m) 13s ago 27m 530M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (13m) 13s ago 27m 86.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (11m) 9s ago 26m 60.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (12m) 13s ago 26m 64.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (21m) 13s ago 21m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:36:05.686 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (21m) 9s ago 21m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (10m) 13s ago 25m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (9m) 13s ago 25m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (9m) 13s ago 24m 54.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (8m) 13s ago 24m 58.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (8m) 9s ago 24m 61.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7m) 9s ago 23m 52.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6m) 9s ago 23m 56.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5m) 9s ago 22m 49.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (14m) 9s ago 22m 60.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5m) 13s ago 20m 78.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5m) 9s ago 20m 78.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5m) 13s ago 19m 79.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:36:05.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4m) 9s ago 19m 80.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:36:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:36:05.997 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:36:06.264 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:36:06.265 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:36:06.574 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:36:06.575 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:36:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:36:06.575 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:36:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:36:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:06 smithi146 ceph-mon[103278]: pgmap v486: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:06 smithi146 ceph-mon[103278]: from='client.65048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:06 smithi146 ceph-mon[103278]: from='client.65054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:06 smithi146 ceph-mon[103278]: from='client.65060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3412733423' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1652164841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[91136]: pgmap v486: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[91136]: from='client.65048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[91136]: from='client.65054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[91136]: from='client.65060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3412733423' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1652164841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[99959]: pgmap v486: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[99959]: from='client.65048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[99959]: from='client.65054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[99959]: from='client.65060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3412733423' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1652164841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:07 smithi146 ceph-mon[103278]: from='client.65072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:07 smithi078 ceph-mon[91136]: from='client.65072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:07 smithi078 ceph-mon[99959]: from='client.65072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:08 smithi146 ceph-mon[103278]: pgmap v487: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:08 smithi078 ceph-mon[91136]: pgmap v487: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:08 smithi078 ceph-mon[99959]: pgmap v487: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:10.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:36:10] "GET /metrics HTTP/1.1" 200 38984 "" "Prometheus/2.43.0" 2024-09-20T20:36:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:10 smithi146 ceph-mon[103278]: pgmap v488: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:10 smithi078 ceph-mon[91136]: pgmap v488: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:10 smithi078 ceph-mon[99959]: pgmap v488: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:12 smithi146 ceph-mon[103278]: pgmap v489: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[99959]: pgmap v489: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[91136]: pgmap v489: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:13.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:13.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:13.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:13 smithi146 ceph-mon[103278]: pgmap v490: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-mon[91136]: pgmap v490: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-mon[99959]: pgmap v490: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:16 smithi146 ceph-mon[103278]: pgmap v491: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:16 smithi078 ceph-mon[99959]: pgmap v491: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:16 smithi078 ceph-mon[91136]: pgmap v491: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:18.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:18 smithi146 ceph-mon[103278]: pgmap v492: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:18 smithi078 ceph-mon[99959]: pgmap v492: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:18 smithi078 ceph-mon[91136]: pgmap v492: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:20.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:20 smithi078 ceph-mon[91136]: pgmap v493: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:20.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:20.908 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:36:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:36:20.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:20 smithi078 ceph-mon[99959]: pgmap v493: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:20.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:20.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:20 smithi146 ceph-mon[103278]: pgmap v493: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:20.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:22 smithi078 ceph-mon[91136]: pgmap v494: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:22 smithi078 ceph-mon[99959]: pgmap v494: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:22 smithi146 ceph-mon[103278]: pgmap v494: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:23.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:23.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:23.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:24 smithi146 ceph-mon[103278]: pgmap v495: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:24 smithi078 ceph-mon[91136]: pgmap v495: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:24 smithi078 ceph-mon[99959]: pgmap v495: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:26 smithi146 ceph-mon[103278]: pgmap v496: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:26 smithi078 ceph-mon[91136]: pgmap v496: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:26 smithi078 ceph-mon[99959]: pgmap v496: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:28 smithi146 ceph-mon[103278]: pgmap v497: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:28 smithi078 ceph-mon[99959]: pgmap v497: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:28 smithi078 ceph-mon[91136]: pgmap v497: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:30.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:30 smithi146 ceph-mon[103278]: pgmap v498: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:30.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:31.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:30 smithi078 ceph-mon[91136]: pgmap v498: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:31.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:31.028 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:36:30] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:36:31.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:30 smithi078 ceph-mon[99959]: pgmap v498: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:31.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:32 smithi146 ceph-mon[103278]: pgmap v499: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[91136]: pgmap v499: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[99959]: pgmap v499: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:33.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:33.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:33.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:34 smithi146 ceph-mon[103278]: pgmap v500: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:34 smithi078 ceph-mon[91136]: pgmap v500: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:34 smithi078 ceph-mon[99959]: pgmap v500: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:36.876 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:36:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:36 smithi146 ceph-mon[103278]: pgmap v501: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:36 smithi078 ceph-mon[99959]: pgmap v501: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:36 smithi078 ceph-mon[91136]: pgmap v501: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:37.405 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 27m - - 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 27m - - 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (21m) 5s ago 22m 27.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (22m) 0s ago 22m 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (14m) 5s ago 20m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (13m) 0s ago 26m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (15m) 5s ago 28m 530M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (13m) 5s ago 28m 87.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (12m) 0s ago 26m 60.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (12m) 5s ago 26m 65.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:36:37.406 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (21m) 5s ago 21m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (21m) 0s ago 21m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (11m) 5s ago 26m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (10m) 5s ago 25m 59.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (9m) 5s ago 25m 55.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (9m) 5s ago 24m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (8m) 0s ago 24m 61.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7m) 0s ago 24m 52.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6m) 0s ago 23m 56.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6m) 0s ago 23m 50.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (14m) 0s ago 22m 60.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5m) 5s ago 20m 78.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5m) 0s ago 20m 79.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5m) 5s ago 20m 79.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:36:37.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5m) 0s ago 20m 80.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:36:37.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:36:37.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:36:37.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:36:37.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:37.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:36:37.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:36:37.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:36:37.715 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:36:37.985 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:36:37.986 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:36:37.987 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:36:37.987 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:36:37.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:37.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:37.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:37.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[99959]: from='client.45162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:37.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2463362948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:37.987 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:37.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:37.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:37.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[91136]: from='client.45162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:37.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2463362948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:37 smithi146 ceph-mon[103278]: from='client.45162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2463362948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:36:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:36:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:36:38.297 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:36:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:36:38.297 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:36:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:38 smithi146 ceph-mon[103278]: pgmap v502: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:38 smithi146 ceph-mon[103278]: from='client.45168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:38 smithi146 ceph-mon[103278]: from='client.65096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2428075860' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[91136]: pgmap v502: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[91136]: from='client.45168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[91136]: from='client.65096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2428075860' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[99959]: pgmap v502: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[99959]: from='client.45168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[99959]: from='client.65096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2428075860' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:36:39.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:39 smithi078 ceph-mon[99959]: from='client.45186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:39.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:39 smithi078 ceph-mon[91136]: from='client.45186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:39 smithi146 ceph-mon[103278]: from='client.45186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:36:40.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:40.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:40.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:40.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-mon[99959]: pgmap v503: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:40.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:36:40] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:36:40.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-mon[91136]: pgmap v503: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:40.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:40 smithi146 ceph-mon[103278]: pgmap v503: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:42 smithi146 ceph-mon[103278]: pgmap v504: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:43.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:42 smithi078 ceph-mon[91136]: pgmap v504: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:43.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:42 smithi078 ceph-mon[99959]: pgmap v504: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:43.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:43.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:43.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:43.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:43.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:43.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:44 smithi146 ceph-mon[103278]: pgmap v505: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:44 smithi078 ceph-mon[91136]: pgmap v505: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:44 smithi078 ceph-mon[99959]: pgmap v505: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:47.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:46 smithi146 ceph-mon[103278]: pgmap v506: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:46 smithi078 ceph-mon[91136]: pgmap v506: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:46 smithi078 ceph-mon[99959]: pgmap v506: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:48 smithi146 ceph-mon[103278]: pgmap v507: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:48 smithi078 ceph-mon[91136]: pgmap v507: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:48 smithi078 ceph-mon[99959]: pgmap v507: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:50.141Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:50.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:50.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:50 smithi146 ceph-mon[103278]: pgmap v508: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:36:50] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:36:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-mon[99959]: pgmap v508: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-mon[91136]: pgmap v508: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:36:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:52 smithi146 ceph-mon[103278]: pgmap v509: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[91136]: pgmap v509: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[99959]: pgmap v509: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:53.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:53.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:53.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:53.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:36:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:36:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:36:53.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:36:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:54 smithi078 ceph-mon[99959]: pgmap v510: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:54 smithi078 ceph-mon[91136]: pgmap v510: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:54 smithi146 ceph-mon[103278]: pgmap v510: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:36:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:56 smithi078 ceph-mon[99959]: pgmap v511: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:56 smithi078 ceph-mon[91136]: pgmap v511: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:56 smithi146 ceph-mon[103278]: pgmap v511: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:36:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:58 smithi146 ceph-mon[103278]: pgmap v512: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:36:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:58 smithi078 ceph-mon[99959]: pgmap v512: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:36:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:36:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:58 smithi078 ceph-mon[91136]: pgmap v512: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:36:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:36:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:00.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:00.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:00.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:00.863 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:37:00] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:37:00.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:00 smithi146 ceph-mon[103278]: pgmap v513: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:00.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-mon[91136]: pgmap v513: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-mon[99959]: pgmap v513: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:03.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:02 smithi078 ceph-mon[91136]: pgmap v514: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:03.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:02 smithi078 ceph-mon[99959]: pgmap v514: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:02 smithi146 ceph-mon[103278]: pgmap v514: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:03.776 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:03.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:03.776 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:03.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:03.777 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:03.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:04 smithi146 ceph-mon[103278]: pgmap v515: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:04 smithi078 ceph-mon[91136]: pgmap v515: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:04 smithi078 ceph-mon[99959]: pgmap v515: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:06 smithi146 ceph-mon[103278]: pgmap v516: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:06 smithi078 ceph-mon[91136]: pgmap v516: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:06 smithi078 ceph-mon[99959]: pgmap v516: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:08.595 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:37:09.120 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:37:09.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 27m - - 2024-09-20T20:37:09.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 27m - - 2024-09-20T20:37:09.120 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (21m) 16s ago 22m 27.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:37:09.120 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (22m) 12s ago 22m 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:37:09.120 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (15m) 16s ago 20m 62.0M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (14m) 12s ago 26m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (16m) 16s ago 28m 531M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (14m) 16s ago 28m 88.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (12m) 12s ago 27m 61.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (13m) 16s ago 27m 65.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (22m) 16s ago 22m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (22m) 12s ago 22m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (12m) 16s ago 26m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (11m) 16s ago 26m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (10m) 16s ago 25m 55.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (9m) 16s ago 25m 58.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (9m) 12s ago 25m 61.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (8m) 12s ago 24m 53.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:37:09.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7m) 12s ago 24m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:37:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6m) 12s ago 23m 50.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:37:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (15m) 12s ago 23m 61.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:37:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6m) 16s ago 21m 78.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:37:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6m) 12s ago 21m 79.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:37:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6m) 16s ago 21m 79.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:37:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6m) 12s ago 20m 81.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:37:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:08 smithi146 ceph-mon[103278]: pgmap v517: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:08 smithi078 ceph-mon[99959]: pgmap v517: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:08 smithi078 ceph-mon[91136]: pgmap v517: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:37:09.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:37:09.437 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:37:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:37:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:37:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:37:10.041 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:37:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:10.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:10.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:10.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:09 smithi078 ceph-mon[99959]: from='client.65120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:10.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:09 smithi078 ceph-mon[99959]: from='client.45204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:10.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1676037396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:10.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:09 smithi078 ceph-mon[91136]: from='client.65120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:10.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:09 smithi078 ceph-mon[91136]: from='client.45204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:10.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1676037396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:09 smithi146 ceph-mon[103278]: from='client.65120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:09 smithi146 ceph-mon[103278]: from='client.45204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1676037396' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:10.474 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:10.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:10.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:10.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:10.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:10.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:37:10] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:37:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:10 smithi146 ceph-mon[103278]: from='client.45210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:10 smithi146 ceph-mon[103278]: pgmap v518: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:10 smithi146 ceph-mon[103278]: from='client.65144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3935759778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[91136]: from='client.45210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[91136]: pgmap v518: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[91136]: from='client.65144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3935759778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[99959]: from='client.45210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[99959]: pgmap v518: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[99959]: from='client.65144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:11.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3935759778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:12 smithi146 ceph-mon[103278]: pgmap v519: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[99959]: pgmap v519: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[91136]: pgmap v519: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:13.889 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:13.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:13.889 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:13.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:13.889 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:13.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:14 smithi146 ceph-mon[103278]: pgmap v520: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:14 smithi078 ceph-mon[91136]: pgmap v520: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:14 smithi078 ceph-mon[99959]: pgmap v520: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:16 smithi146 ceph-mon[103278]: pgmap v521: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:16 smithi078 ceph-mon[91136]: pgmap v521: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:16 smithi078 ceph-mon[99959]: pgmap v521: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:17 smithi146 ceph-mon[103278]: pgmap v522: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[91136]: pgmap v522: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:17 smithi078 ceph-mon[99959]: pgmap v522: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-mon[91136]: pgmap v523: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-mon[99959]: pgmap v523: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:20.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:20.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:20.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:20.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:20 smithi146 ceph-mon[103278]: pgmap v523: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:20.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:37:20] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:37:22.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:22 smithi078 ceph-mon[91136]: pgmap v524: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:22.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:22 smithi078 ceph-mon[99959]: pgmap v524: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:22 smithi146 ceph-mon[103278]: pgmap v524: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:23.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:23.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:23.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:23.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:24 smithi146 ceph-mon[103278]: pgmap v525: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:24 smithi078 ceph-mon[91136]: pgmap v525: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:24 smithi078 ceph-mon[99959]: pgmap v525: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:26.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:26 smithi146 ceph-mon[103278]: pgmap v526: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:26 smithi078 ceph-mon[91136]: pgmap v526: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:26 smithi078 ceph-mon[99959]: pgmap v526: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:28 smithi146 ceph-mon[103278]: pgmap v527: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:28 smithi078 ceph-mon[91136]: pgmap v527: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:28 smithi078 ceph-mon[99959]: pgmap v527: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:30 smithi146 ceph-mon[103278]: pgmap v528: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-mon[91136]: pgmap v528: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-mon[99959]: pgmap v528: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:30.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:30.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:30.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:37:30] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:37:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:32 smithi146 ceph-mon[103278]: pgmap v529: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:32 smithi078 ceph-mon[91136]: pgmap v529: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:32 smithi078 ceph-mon[99959]: pgmap v529: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:33.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:33.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:33.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:34.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:34 smithi146 ceph-mon[103278]: pgmap v530: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:34 smithi078 ceph-mon[99959]: pgmap v530: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:34 smithi078 ceph-mon[91136]: pgmap v530: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:36 smithi146 ceph-mon[103278]: pgmap v531: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:36 smithi078 ceph-mon[91136]: pgmap v531: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:36 smithi078 ceph-mon[99959]: pgmap v531: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:38.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:38 smithi146 ceph-mon[103278]: pgmap v532: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[91136]: pgmap v532: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:38 smithi078 ceph-mon[99959]: pgmap v532: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:40.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-mon[99959]: pgmap v533: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:40.307 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:40.307 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-mon[91136]: pgmap v533: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:40.307 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:40.307 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:40.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:40.307 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:40.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:40.307 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:40.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:37:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:40 smithi146 ceph-mon[103278]: pgmap v533: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:40.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:37:40] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 28m - - 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 28m - - 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (22m) 8s ago 23m 32.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (23m) 3s ago 23m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (15m) 8s ago 21m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (14m) 3s ago 27m 453M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (16m) 8s ago 29m 531M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (14m) 8s ago 29m 88.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (13m) 3s ago 27m 61.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (13m) 8s ago 27m 65.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:37:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (23m) 8s ago 23m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (22m) 3s ago 22m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (12m) 8s ago 27m 61.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (11m) 8s ago 26m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (10m) 8s ago 26m 55.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10m) 8s ago 25m 58.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (9m) 3s ago 25m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (8m) 3s ago 25m 53.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (8m) 3s ago 24m 56.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7m) 3s ago 24m 50.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (16m) 3s ago 23m 61.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6m) 8s ago 21m 79.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6m) 3s ago 21m 79.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:37:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6m) 8s ago 21m 80.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:37:40.915 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6m) 3s ago 21m 81.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:37:41.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:37:41.236 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:37:41.507 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:37:41.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:37:41.508 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:37:41.508 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:37:41.508 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:37:41.508 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:37:41.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:41 smithi078 ceph-mon[99959]: from='client.65156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:41 smithi078 ceph-mon[99959]: from='client.45240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:41 smithi078 ceph-mon[99959]: from='client.45246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:41 smithi078 ceph-mon[91136]: from='client.65156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:41 smithi078 ceph-mon[91136]: from='client.45240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:41 smithi078 ceph-mon[91136]: from='client.45246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:41 smithi146 ceph-mon[103278]: from='client.65156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:41 smithi146 ceph-mon[103278]: from='client.45240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:41 smithi146 ceph-mon[103278]: from='client.45246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:41.821 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:37:41.822 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:41.822 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:37:41.822 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:41.822 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:37:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:42 smithi146 ceph-mon[103278]: pgmap v534: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1792902757' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:42 smithi146 ceph-mon[103278]: from='client.45258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/283656613' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[91136]: pgmap v534: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1792902757' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[91136]: from='client.45258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/283656613' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[99959]: pgmap v534: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1792902757' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[99959]: from='client.45258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:37:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/283656613' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:37:43.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:43.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:43.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:43.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:43.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:44 smithi146 ceph-mon[103278]: pgmap v535: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:44.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:44 smithi078 ceph-mon[99959]: pgmap v535: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:44.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:44 smithi078 ceph-mon[91136]: pgmap v535: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:46 smithi146 ceph-mon[103278]: pgmap v536: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:46 smithi078 ceph-mon[91136]: pgmap v536: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:46 smithi078 ceph-mon[99959]: pgmap v536: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:48 smithi146 ceph-mon[103278]: pgmap v537: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:48 smithi078 ceph-mon[99959]: pgmap v537: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:48 smithi078 ceph-mon[91136]: pgmap v537: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:50 smithi146 ceph-mon[103278]: pgmap v538: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-mon[91136]: pgmap v538: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-mon[99959]: pgmap v538: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:37:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:50.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:50.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:50.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:50.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:37:50] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:37:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:52 smithi146 ceph-mon[103278]: pgmap v539: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:52 smithi078 ceph-mon[91136]: pgmap v539: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:52 smithi078 ceph-mon[99959]: pgmap v539: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:53.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:53.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:37:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:37:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:37:53.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:37:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:54 smithi078 ceph-mon[99959]: pgmap v540: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:54 smithi078 ceph-mon[91136]: pgmap v540: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:55.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:54 smithi146 ceph-mon[103278]: pgmap v540: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:37:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:56 smithi078 ceph-mon[91136]: pgmap v541: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:56 smithi078 ceph-mon[99959]: pgmap v541: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:57.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:56 smithi146 ceph-mon[103278]: pgmap v541: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:58 smithi146 ceph-mon[103278]: pgmap v542: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[91136]: pgmap v542: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[99959]: pgmap v542: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:37:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:00 smithi146 ceph-mon[103278]: pgmap v543: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-mon[99959]: pgmap v543: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-mon[91136]: pgmap v543: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:00.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:00.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:00.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:38:00] "GET /metrics HTTP/1.1" 200 38978 "" "Prometheus/2.43.0" 2024-09-20T20:38:02.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:02 smithi146 ceph-mon[103278]: pgmap v544: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:02 smithi078 ceph-mon[91136]: pgmap v544: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:02 smithi078 ceph-mon[99959]: pgmap v544: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:03.857 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:03.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:03.857 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:03.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:03.857 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:03.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:04 smithi146 ceph-mon[103278]: pgmap v545: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:04 smithi078 ceph-mon[91136]: pgmap v545: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:04 smithi078 ceph-mon[99959]: pgmap v545: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:06.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:06 smithi078 ceph-mon[91136]: pgmap v546: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:06.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:06 smithi078 ceph-mon[99959]: pgmap v546: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:06 smithi146 ceph-mon[103278]: pgmap v546: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:08 smithi146 ceph-mon[103278]: pgmap v547: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:08 smithi078 ceph-mon[91136]: pgmap v547: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:08 smithi078 ceph-mon[99959]: pgmap v547: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:10 smithi146 ceph-mon[103278]: pgmap v548: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-mon[91136]: pgmap v548: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-mon[99959]: pgmap v548: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:10.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:10.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:10.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:38:10] "GET /metrics HTTP/1.1" 200 38978 "" "Prometheus/2.43.0" 2024-09-20T20:38:12.121 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:38:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:12 smithi146 ceph-mon[103278]: pgmap v549: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:12 smithi078 ceph-mon[91136]: pgmap v549: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:12 smithi078 ceph-mon[99959]: pgmap v549: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:12.667 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 28m - - 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 28m - - 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (22m) 19s ago 24m 32.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (23m) 15s ago 23m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (16m) 19s ago 21m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (15m) 15s ago 28m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (17m) 19s ago 29m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (15m) 19s ago 29m 89.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (14m) 15s ago 28m 61.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (14m) 19s ago 28m 66.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (23m) 19s ago 23m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (23m) 15s ago 23m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (13m) 19s ago 27m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:38:12.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (12m) 19s ago 27m 60.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (11m) 19s ago 26m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (10m) 19s ago 26m 58.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (10m) 15s ago 26m 62.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (9m) 15s ago 25m 53.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (8m) 15s ago 25m 56.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7m) 15s ago 24m 50.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (16m) 15s ago 24m 61.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7m) 19s ago 22m 79.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7m) 15s ago 22m 80.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7m) 19s ago 22m 80.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:38:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7m) 15s ago 22m 81.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:38:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:38:13.008 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:38:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:38:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:38:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:38:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:38:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:38:13.282 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:38:13.591 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:38:13.591 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:13.591 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:13.591 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:13.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:13.584Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:13.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:13.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:13.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:13.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='client.65192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='client.45276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='client.45282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3235145495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1117736923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='client.65192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='client.45276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='client.45282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3235145495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1117736923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='client.65192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='client.45276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='client.45282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3235145495' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:14.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1117736923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:14 smithi146 ceph-mon[103278]: pgmap v550: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:14 smithi146 ceph-mon[103278]: from='client.45294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:14 smithi078 ceph-mon[91136]: pgmap v550: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:14 smithi078 ceph-mon[91136]: from='client.45294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:14 smithi078 ceph-mon[99959]: pgmap v550: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:14 smithi078 ceph-mon[99959]: from='client.45294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:16 smithi146 ceph-mon[103278]: pgmap v551: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:16 smithi078 ceph-mon[99959]: pgmap v551: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:16 smithi078 ceph-mon[91136]: pgmap v551: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:18 smithi146 ceph-mon[103278]: pgmap v552: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[91136]: pgmap v552: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[99959]: pgmap v552: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:20.371 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:20.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:20.371 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:20.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:20.371 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:20.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:20 smithi146 ceph-mon[103278]: pgmap v553: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-mon[99959]: pgmap v553: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-mon[91136]: pgmap v553: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:38:20] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:38:22.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:22 smithi146 ceph-mon[103278]: pgmap v554: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:22 smithi078 ceph-mon[99959]: pgmap v554: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:22 smithi078 ceph-mon[91136]: pgmap v554: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:23.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:23.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:23.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:24.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:24 smithi078 ceph-mon[99959]: pgmap v555: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:24.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:24 smithi078 ceph-mon[91136]: pgmap v555: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:24 smithi146 ceph-mon[103278]: pgmap v555: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:26.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:26 smithi078 ceph-mon[99959]: pgmap v556: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:26.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:26 smithi078 ceph-mon[91136]: pgmap v556: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:26 smithi146 ceph-mon[103278]: pgmap v556: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:28 smithi146 ceph-mon[103278]: pgmap v557: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:28 smithi078 ceph-mon[91136]: pgmap v557: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:28 smithi078 ceph-mon[99959]: pgmap v557: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-mon[91136]: pgmap v558: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-mon[99959]: pgmap v558: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:30.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:30.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:30.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:30 smithi146 ceph-mon[103278]: pgmap v558: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:38:30] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:38:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:32 smithi146 ceph-mon[103278]: pgmap v559: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:32 smithi078 ceph-mon[91136]: pgmap v559: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:32 smithi078 ceph-mon[99959]: pgmap v559: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:34.017 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:33.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:34.017 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:33.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:34.017 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:33.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:34 smithi146 ceph-mon[103278]: pgmap v560: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[91136]: pgmap v560: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:34 smithi078 ceph-mon[99959]: pgmap v560: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:36 smithi146 ceph-mon[103278]: pgmap v561: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:36 smithi078 ceph-mon[91136]: pgmap v561: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:36 smithi078 ceph-mon[99959]: pgmap v561: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:38 smithi146 ceph-mon[103278]: pgmap v562: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[91136]: pgmap v562: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[99959]: pgmap v562: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:40.501 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:40.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:40.501 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:40.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:40.501 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:40.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:40 smithi146 ceph-mon[103278]: pgmap v563: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:38:40] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:38:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-mon[91136]: pgmap v563: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-mon[99959]: pgmap v563: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:42 smithi146 ceph-mon[103278]: pgmap v564: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:42 smithi078 ceph-mon[91136]: pgmap v564: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:42 smithi078 ceph-mon[99959]: pgmap v564: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:43.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:43.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:43.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:43.889 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:38:44.416 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:38:44.416 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 29m - - 2024-09-20T20:38:44.416 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 29m - - 2024-09-20T20:38:44.416 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (23m) 11s ago 24m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (24m) 6s ago 24m 87.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (16m) 11s ago 22m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (15m) 6s ago 28m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (17m) 11s ago 30m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (15m) 11s ago 30m 90.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (14m) 6s ago 28m 63.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (14m) 11s ago 28m 67.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (24m) 11s ago 24m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (24m) 6s ago 23m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (13m) 11s ago 28m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (12m) 11s ago 27m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (11m) 11s ago 27m 55.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (11m) 11s ago 27m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:38:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (10m) 6s ago 26m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (9m) 6s ago 26m 53.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (9m) 6s ago 25m 57.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (8m) 6s ago 25m 50.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (17m) 6s ago 24m 61.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7m) 11s ago 22m 79.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7m) 6s ago 22m 80.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7m) 11s ago 22m 80.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:38:44.418 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7m) 6s ago 22m 82.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:38:44.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:44 smithi146 ceph-mon[103278]: pgmap v565: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:44.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:44 smithi146 ceph-mon[103278]: from='client.65228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:38:44.730 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:38:44.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:44 smithi078 ceph-mon[99959]: pgmap v565: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:44.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:44 smithi078 ceph-mon[99959]: from='client.65228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:44.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:44 smithi078 ceph-mon[91136]: pgmap v565: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:44.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:44 smithi078 ceph-mon[91136]: from='client.65228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:38:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:38:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:38:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:38:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:38:45.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:38:45.320 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:38:45.320 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:38:45.321 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:38:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[99959]: from='client.65234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[99959]: from='client.65240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2871332972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:46.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2483205996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[91136]: from='client.65234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[91136]: from='client.65240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2871332972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2483205996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:45 smithi146 ceph-mon[103278]: from='client.65234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:45 smithi146 ceph-mon[103278]: from='client.65240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2871332972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:38:46.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2483205996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:38:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:46 smithi078 ceph-mon[91136]: from='client.45330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:46 smithi078 ceph-mon[91136]: pgmap v566: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:46 smithi078 ceph-mon[99959]: from='client.45330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:46 smithi078 ceph-mon[99959]: pgmap v566: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:46 smithi146 ceph-mon[103278]: from='client.45330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:38:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:46 smithi146 ceph-mon[103278]: pgmap v566: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:48 smithi078 ceph-mon[91136]: pgmap v567: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:48 smithi078 ceph-mon[99959]: pgmap v567: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:48 smithi146 ceph-mon[103278]: pgmap v567: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:50.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:50.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:50.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:50.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:50.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:50.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-mon[99959]: pgmap v568: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-mon[91136]: pgmap v568: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:38:50] "GET /metrics HTTP/1.1" 200 38978 "" "Prometheus/2.43.0" 2024-09-20T20:38:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:50 smithi146 ceph-mon[103278]: pgmap v568: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:38:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:52 smithi078 ceph-mon[99959]: pgmap v569: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:52 smithi078 ceph-mon[91136]: pgmap v569: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:52 smithi146 ceph-mon[103278]: pgmap v569: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:53.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:53.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:38:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:38:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:38:53.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:38:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:54 smithi146 ceph-mon[103278]: pgmap v570: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[99959]: pgmap v570: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:54 smithi078 ceph-mon[91136]: pgmap v570: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:38:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:56 smithi146 ceph-mon[103278]: pgmap v571: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:56 smithi078 ceph-mon[99959]: pgmap v571: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:56 smithi078 ceph-mon[91136]: pgmap v571: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:58 smithi146 ceph-mon[103278]: pgmap v572: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[91136]: pgmap v572: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[99959]: pgmap v572: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:38:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:00.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:00.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:00.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:00 smithi146 ceph-mon[103278]: pgmap v573: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:39:00] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:39:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-mon[91136]: pgmap v573: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-mon[99959]: pgmap v573: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:02.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:02 smithi146 ceph-mon[103278]: pgmap v574: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:02 smithi078 ceph-mon[99959]: pgmap v574: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:02 smithi078 ceph-mon[91136]: pgmap v574: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:03.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:03.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:03.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:04 smithi146 ceph-mon[103278]: pgmap v575: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:04 smithi078 ceph-mon[99959]: pgmap v575: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:04 smithi078 ceph-mon[91136]: pgmap v575: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:06 smithi078 ceph-mon[91136]: pgmap v576: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:06 smithi078 ceph-mon[99959]: pgmap v576: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:06 smithi146 ceph-mon[103278]: pgmap v576: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:08 smithi146 ceph-mon[103278]: pgmap v577: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:08 smithi078 ceph-mon[91136]: pgmap v577: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:08 smithi078 ceph-mon[99959]: pgmap v577: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:10.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:10.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:10.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:10.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-mon[91136]: pgmap v578: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:10.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:10.791 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:39:10] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:39:10.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-mon[99959]: pgmap v578: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:10.791 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:10 smithi146 ceph-mon[103278]: pgmap v578: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:12 smithi146 ceph-mon[103278]: pgmap v579: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:12 smithi078 ceph-mon[91136]: pgmap v579: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:12 smithi078 ceph-mon[99959]: pgmap v579: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:13.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:13.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:13.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:14 smithi146 ceph-mon[103278]: pgmap v580: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:14 smithi078 ceph-mon[99959]: pgmap v580: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:14 smithi078 ceph-mon[91136]: pgmap v580: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:15.617 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:39:16.142 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 29m - - 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 29m - - 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (24m) 2s ago 25m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (24m) 18s ago 24m 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (17m) 2s ago 22m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (16m) 18s ago 29m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (18m) 2s ago 30m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (16m) 2s ago 30m 91.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (15m) 18s ago 29m 64.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (15m) 2s ago 29m 68.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (24m) 2s ago 24m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (24m) 18s ago 24m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:39:16.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (14m) 2s ago 28m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (13m) 2s ago 28m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (12m) 2s ago 27m 56.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (11m) 2s ago 27m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (11m) 18s ago 27m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (10m) 18s ago 26m 53.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (9m) 18s ago 26m 57.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (8m) 18s ago 26m 50.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (17m) 18s ago 25m 61.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (8m) 2s ago 23m 79.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (8m) 18s ago 23m 80.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (8m) 2s ago 23m 80.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:39:16.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (8m) 18s ago 23m 82.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:39:16.453 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:39:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:39:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:39:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:39:16.455 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:39:16.729 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:39:16.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:39:16.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:39:16.730 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:39:16.730 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:39:16.730 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:39:16.730 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:39:16.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[99959]: pgmap v581: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:16.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[99959]: from='client.45342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:16.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[99959]: from='client.65270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:16.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2692131578' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:16.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[91136]: pgmap v581: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:16.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[91136]: from='client.45342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:16.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[91136]: from='client.65270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:16.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2692131578' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:16 smithi146 ceph-mon[103278]: pgmap v581: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:16 smithi146 ceph-mon[103278]: from='client.45342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:16 smithi146 ceph-mon[103278]: from='client.65270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2692131578' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:17.041 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:39:17.041 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:17.041 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:17.041 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:17.041 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:17 smithi146 ceph-mon[103278]: from='client.45354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:17 smithi146 ceph-mon[103278]: from='client.45366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2839729664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:17 smithi078 ceph-mon[99959]: from='client.45354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:17 smithi078 ceph-mon[99959]: from='client.45366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2839729664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:17 smithi078 ceph-mon[91136]: from='client.45354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:17 smithi078 ceph-mon[91136]: from='client.45366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2839729664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:18 smithi146 ceph-mon[103278]: pgmap v582: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[91136]: pgmap v582: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[99959]: pgmap v582: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:19.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:20.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:20.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:20.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:20 smithi146 ceph-mon[103278]: pgmap v583: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-mon[99959]: pgmap v583: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-mon[91136]: pgmap v583: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:39:20] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:39:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:22 smithi146 ceph-mon[103278]: pgmap v584: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:22 smithi078 ceph-mon[91136]: pgmap v584: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:22 smithi078 ceph-mon[99959]: pgmap v584: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:23.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:23.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:23.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:24 smithi146 ceph-mon[103278]: pgmap v585: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:24 smithi078 ceph-mon[99959]: pgmap v585: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:24 smithi078 ceph-mon[91136]: pgmap v585: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:26 smithi078 ceph-mon[91136]: pgmap v586: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:26 smithi078 ceph-mon[99959]: pgmap v586: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:26 smithi146 ceph-mon[103278]: pgmap v586: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:28 smithi078 ceph-mon[91136]: pgmap v587: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:28 smithi078 ceph-mon[99959]: pgmap v587: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:28 smithi146 ceph-mon[103278]: pgmap v587: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:30.153Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:30.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:30.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-mon[91136]: pgmap v588: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:39:30] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:39:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-mon[99959]: pgmap v588: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:30 smithi146 ceph-mon[103278]: pgmap v588: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:32 smithi078 ceph-mon[91136]: pgmap v589: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:32 smithi078 ceph-mon[99959]: pgmap v589: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:32 smithi146 ceph-mon[103278]: pgmap v589: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:33.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:33.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:33.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:34 smithi146 ceph-mon[103278]: pgmap v590: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[91136]: pgmap v590: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[99959]: pgmap v590: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:36 smithi146 ceph-mon[103278]: pgmap v591: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:36 smithi078 ceph-mon[99959]: pgmap v591: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:36 smithi078 ceph-mon[91136]: pgmap v591: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[99959]: pgmap v592: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[91136]: pgmap v592: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:38 smithi146 ceph-mon[103278]: pgmap v592: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:40.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:40.154Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:40.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:40.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:40.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:40.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:40.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:39:40] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:39:41.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-mon[91136]: pgmap v593: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:41.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:41.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-mon[99959]: pgmap v593: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:41.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:40 smithi146 ceph-mon[103278]: pgmap v593: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:42 smithi146 ceph-mon[103278]: pgmap v594: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:42 smithi078 ceph-mon[91136]: pgmap v594: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:42 smithi078 ceph-mon[99959]: pgmap v594: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:43.894 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:43.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:43.894 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:43.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:43.894 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:43.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:44.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:44 smithi146 ceph-mon[103278]: pgmap v595: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:44 smithi078 ceph-mon[99959]: pgmap v595: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:44 smithi078 ceph-mon[91136]: pgmap v595: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:47.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:46 smithi078 ceph-mon[91136]: pgmap v596: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:47.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:46 smithi078 ceph-mon[99959]: pgmap v596: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:46 smithi146 ceph-mon[103278]: pgmap v596: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:47.342 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 30m - - 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 30m - - 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (24m) 14s ago 25m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (25m) 9s ago 25m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (17m) 14s ago 23m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (16m) 9s ago 29m 454M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (18m) 14s ago 31m 532M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (16m) 14s ago 31m 92.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (15m) 9s ago 30m 64.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:39:47.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (15m) 14s ago 29m 69.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (25m) 14s ago 25m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (25m) 9s ago 25m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (14m) 14s ago 29m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (13m) 14s ago 28m 60.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (13m) 14s ago 28m 56.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (12m) 14s ago 28m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (11m) 9s ago 27m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (10m) 9s ago 27m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (10m) 9s ago 26m 57.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (9m) 9s ago 26m 51.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (18m) 9s ago 25m 61.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (8m) 14s ago 23m 80.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (8m) 9s ago 23m 81.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (8m) 14s ago 23m 81.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:39:47.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (8m) 9s ago 23m 82.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:39:48.195 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:39:48.195 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:39:48.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:39:48.196 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:39:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:39:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:39:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:39:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:39:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:39:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:39:48.476 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:39:48.791 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:39:48.791 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:48.791 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:39:48.792 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:48.792 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[91136]: pgmap v597: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[91136]: from='client.65300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[91136]: from='client.65306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[91136]: from='client.65312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3063875962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1552894937' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[99959]: pgmap v597: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[99959]: from='client.65300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[99959]: from='client.65306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[99959]: from='client.65312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3063875962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1552894937' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:48 smithi146 ceph-mon[103278]: pgmap v597: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:48 smithi146 ceph-mon[103278]: from='client.65300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:48 smithi146 ceph-mon[103278]: from='client.65306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:48 smithi146 ceph-mon[103278]: from='client.65312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3063875962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:39:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1552894937' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:39:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:49 smithi078 ceph-mon[91136]: from='client.65324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:49 smithi078 ceph-mon[99959]: from='client.65324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:49 smithi146 ceph-mon[103278]: from='client.65324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:39:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:50.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:50.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:50.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:50.908 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:39:50] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:39:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:50 smithi146 ceph-mon[103278]: pgmap v598: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-mon[99959]: pgmap v598: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-mon[91136]: pgmap v598: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:39:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:52 smithi146 ceph-mon[103278]: pgmap v599: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:52 smithi078 ceph-mon[91136]: pgmap v599: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:52 smithi078 ceph-mon[99959]: pgmap v599: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:53.593Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:53.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:39:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:39:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:39:53.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:39:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:54 smithi146 ceph-mon[103278]: pgmap v600: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[91136]: pgmap v600: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[99959]: pgmap v600: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:56 smithi146 ceph-mon[103278]: pgmap v601: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:56 smithi078 ceph-mon[91136]: pgmap v601: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:56 smithi078 ceph-mon[99959]: pgmap v601: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:58 smithi146 ceph-mon[103278]: pgmap v602: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[91136]: pgmap v602: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[99959]: pgmap v602: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:39:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:39:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:00.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:00.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:00.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:00.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: pgmap v603: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:00 smithi146 ceph-mon[103278]: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:40:00] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: pgmap v603: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[99959]: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: pgmap v603: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: Health detail: HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: [WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:00 smithi078 ceph-mon[91136]: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:02 smithi146 ceph-mon[103278]: pgmap v604: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:02 smithi078 ceph-mon[91136]: pgmap v604: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:02 smithi078 ceph-mon[99959]: pgmap v604: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:03.594Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:03.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:03.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:04.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:04 smithi146 ceph-mon[103278]: pgmap v605: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:04 smithi078 ceph-mon[91136]: pgmap v605: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:04 smithi078 ceph-mon[99959]: pgmap v605: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:06 smithi146 ceph-mon[103278]: pgmap v606: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:06 smithi078 ceph-mon[91136]: pgmap v606: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:06 smithi078 ceph-mon[99959]: pgmap v606: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:08 smithi078 ceph-mon[91136]: pgmap v607: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:08 smithi078 ceph-mon[99959]: pgmap v607: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:08.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:08 smithi146 ceph-mon[103278]: pgmap v607: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:10.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:10.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:10.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:10.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:10.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:10.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:10 smithi146 ceph-mon[103278]: pgmap v608: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-mon[91136]: pgmap v608: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-mon[99959]: pgmap v608: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:40:10] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:40:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:12 smithi146 ceph-mon[103278]: pgmap v609: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:12 smithi078 ceph-mon[91136]: pgmap v609: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:12 smithi078 ceph-mon[99959]: pgmap v609: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:13.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:13.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:13.892 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:13.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:13.892 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:13.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:13.893 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:13.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:14 smithi146 ceph-mon[103278]: pgmap v610: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[91136]: pgmap v610: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[99959]: pgmap v610: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:16 smithi146 ceph-mon[103278]: pgmap v611: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:16 smithi078 ceph-mon[91136]: pgmap v611: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:16 smithi078 ceph-mon[99959]: pgmap v611: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:18 smithi146 ceph-mon[103278]: pgmap v612: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[91136]: pgmap v612: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[99959]: pgmap v612: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:19.087 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:40:19.610 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 30m - - 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 30m - - 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (25m) 5s ago 26m 31.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (25m) 1s ago 25m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (18m) 5s ago 23m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (17m) 1s ago 30m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (19m) 5s ago 31m 533M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (17m) 5s ago 31m 93.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (16m) 1s ago 30m 67.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (16m) 5s ago 30m 69.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (25m) 5s ago 25m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (25m) 1s ago 25m 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:40:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (15m) 5s ago 29m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (14m) 5s ago 29m 60.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (13m) 5s ago 29m 56.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (13m) 5s ago 28m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (12m) 1s ago 28m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (11m) 1s ago 27m 54.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (10m) 1s ago 27m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (9m) 1s ago 27m 51.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (18m) 1s ago 26m 61.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (9m) 5s ago 24m 80.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (9m) 1s ago 24m 81.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (9m) 5s ago 24m 81.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:40:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (9m) 1s ago 24m 83.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:40:19.921 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:40:19.921 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:40:19.922 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:40:19.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:40:19.923 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:40:20.192 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:40:20.192 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:20.157Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:20.192 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:20.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:20.192 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:20.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:20.500 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:20.500 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:20.500 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:20.501 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:20.501 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:40:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: from='client.45414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: pgmap v613: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: from='client.65348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1014896316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4000599009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: from='client.45414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: pgmap v613: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: from='client.65348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1014896316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4000599009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: from='client.45414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: pgmap v613: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: from='client.65348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1014896316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4000599009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:21 smithi146 ceph-mon[103278]: from='client.45438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:21 smithi078 ceph-mon[91136]: from='client.45438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:21 smithi078 ceph-mon[99959]: from='client.45438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:22 smithi146 ceph-mon[103278]: pgmap v614: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:22 smithi078 ceph-mon[99959]: pgmap v614: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:22 smithi078 ceph-mon[91136]: pgmap v614: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:23.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:23.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:23.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:24.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:24 smithi146 ceph-mon[103278]: pgmap v615: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:24 smithi078 ceph-mon[91136]: pgmap v615: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:24 smithi078 ceph-mon[99959]: pgmap v615: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:26 smithi146 ceph-mon[103278]: pgmap v616: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:26 smithi078 ceph-mon[91136]: pgmap v616: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:26 smithi078 ceph-mon[99959]: pgmap v616: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:28.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:28 smithi078 ceph-mon[99959]: pgmap v617: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:28.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:28.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:28 smithi078 ceph-mon[91136]: pgmap v617: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:28.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:28 smithi146 ceph-mon[103278]: pgmap v617: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:30.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:30.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:30.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:30.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:30.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:30.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:40:30] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:40:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-mon[99959]: pgmap v618: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-mon[91136]: pgmap v618: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:30.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:30 smithi146 ceph-mon[103278]: pgmap v618: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:32 smithi146 ceph-mon[103278]: pgmap v619: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:32 smithi078 ceph-mon[91136]: pgmap v619: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:32 smithi078 ceph-mon[99959]: pgmap v619: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:33.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:33.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:33.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:34 smithi146 ceph-mon[103278]: pgmap v620: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[91136]: pgmap v620: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[99959]: pgmap v620: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:37.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:36 smithi146 ceph-mon[103278]: pgmap v621: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:36 smithi078 ceph-mon[91136]: pgmap v621: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:36 smithi078 ceph-mon[99959]: pgmap v621: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:38 smithi146 ceph-mon[103278]: pgmap v622: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[91136]: pgmap v622: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[99959]: pgmap v622: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:40.501 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:40.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:40.502 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:40.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:40.502 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:40.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:40:40] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:40:41.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-mon[91136]: pgmap v623: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:41.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:41.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-mon[99959]: pgmap v623: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:41.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:40 smithi146 ceph-mon[103278]: pgmap v623: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:42 smithi146 ceph-mon[103278]: pgmap v624: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:42 smithi078 ceph-mon[91136]: pgmap v624: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:42 smithi078 ceph-mon[99959]: pgmap v624: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:43.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:43.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:43.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:45.192 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:44 smithi146 ceph-mon[103278]: pgmap v625: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:44 smithi078 ceph-mon[91136]: pgmap v625: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:44 smithi078 ceph-mon[99959]: pgmap v625: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:47.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:46 smithi146 ceph-mon[103278]: pgmap v626: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:46 smithi078 ceph-mon[91136]: pgmap v626: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:46 smithi078 ceph-mon[99959]: pgmap v626: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:48 smithi146 ceph-mon[103278]: pgmap v627: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:49.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:48 smithi078 ceph-mon[99959]: pgmap v627: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:49.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:48 smithi078 ceph-mon[91136]: pgmap v627: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:50.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:50.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:50.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:50.796 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:40:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-mon[91136]: pgmap v628: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:40:50] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:40:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-mon[99959]: pgmap v628: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:50 smithi146 ceph-mon[103278]: pgmap v628: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:40:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:40:51.333 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 31m - - 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 31m - - 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (25m) 17s ago 26m 31.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (26m) 13s ago 26m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (18m) 17s ago 24m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (17m) 13s ago 30m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (19m) 17s ago 32m 533M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (18m) 17s ago 32m 93.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (16m) 13s ago 31m 64.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (17m) 17s ago 30m 69.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (26m) 17s ago 26m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (26m) 13s ago 26m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:40:51.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (15m) 17s ago 30m 61.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (14m) 17s ago 29m 60.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (14m) 17s ago 29m 56.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (13m) 17s ago 29m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (13m) 13s ago 28m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (11m) 13s ago 28m 54.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (11m) 13s ago 28m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (10m) 13s ago 27m 51.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (19m) 13s ago 27m 61.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (10m) 17s ago 24m 80.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (9m) 13s ago 24m 81.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (9m) 17s ago 24m 81.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:40:51.335 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (9m) 13s ago 24m 83.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:40:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:40:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:40:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:40:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:40:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:40:51.670 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:40:51.995 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:40:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:40:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:40:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:40:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:40:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:40:51.996 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:40:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:51 smithi078 ceph-mon[99959]: from='client.45450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4165457275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:51 smithi078 ceph-mon[91136]: from='client.45450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4165457275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:51 smithi146 ceph-mon[103278]: from='client.45450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4165457275' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:40:52.313 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:40:52.313 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:40:52.313 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:40:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:52 smithi146 ceph-mon[103278]: from='client.65375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:52 smithi146 ceph-mon[103278]: pgmap v629: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:52 smithi146 ceph-mon[103278]: from='client.45462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:52 smithi146 ceph-mon[103278]: from='client.45474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3608961352' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[99959]: from='client.65375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[99959]: pgmap v629: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[99959]: from='client.45462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[99959]: from='client.45474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3608961352' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[91136]: from='client.65375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[91136]: pgmap v629: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[91136]: from='client.45462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[91136]: from='client.45474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:40:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3608961352' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:40:53.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:53.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:53.962 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:53.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:40:53.962 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:40:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:40:53.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:40:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:54 smithi146 ceph-mon[103278]: pgmap v630: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:40:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[91136]: pgmap v630: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[99959]: pgmap v630: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:56 smithi146 ceph-mon[103278]: pgmap v631: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-20T20:40:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:56 smithi078 ceph-mon[91136]: pgmap v631: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-20T20:40:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:56 smithi078 ceph-mon[99959]: pgmap v631: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 43 op/s 2024-09-20T20:40:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:58 smithi146 ceph-mon[103278]: pgmap v632: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:40:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[91136]: pgmap v632: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[99959]: pgmap v632: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:40:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:00.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:00.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:00.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:00.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:41:00] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:41:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:00 smithi146 ceph-mon[103278]: pgmap v633: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:41:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-mon[99959]: pgmap v633: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:41:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-mon[91136]: pgmap v633: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:41:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:02 smithi146 ceph-mon[103278]: pgmap v634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:41:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:02 smithi078 ceph-mon[91136]: pgmap v634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:41:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:02 smithi078 ceph-mon[99959]: pgmap v634: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:41:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:03.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:03.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:03.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:04 smithi146 ceph-mon[103278]: pgmap v635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:41:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:04 smithi078 ceph-mon[91136]: pgmap v635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:41:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:04 smithi078 ceph-mon[99959]: pgmap v635: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:41:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:06 smithi146 ceph-mon[103278]: pgmap v636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T20:41:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:06 smithi078 ceph-mon[91136]: pgmap v636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T20:41:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:06 smithi078 ceph-mon[99959]: pgmap v636: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-09-20T20:41:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:08 smithi146 ceph-mon[103278]: pgmap v637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T20:41:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:08 smithi078 ceph-mon[91136]: pgmap v637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T20:41:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:08 smithi078 ceph-mon[99959]: pgmap v637: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-20T20:41:10.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:10.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:10.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:10.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:10.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:10.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:10.876 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:41:10] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:41:11.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-mon[99959]: pgmap v638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:41:11.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:11.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-mon[91136]: pgmap v638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:41:11.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:10 smithi146 ceph-mon[103278]: pgmap v638: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:41:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:12 smithi146 ceph-mon[103278]: pgmap v639: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T20:41:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:12 smithi078 ceph-mon[91136]: pgmap v639: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T20:41:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:12 smithi078 ceph-mon[99959]: pgmap v639: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T20:41:13.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:13.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:13.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:13.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:13.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:13.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:14 smithi146 ceph-mon[103278]: pgmap v640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:41:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[91136]: pgmap v640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[99959]: pgmap v640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:15.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:16 smithi146 ceph-mon[103278]: pgmap v641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T20:41:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:16 smithi078 ceph-mon[91136]: pgmap v641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T20:41:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:16 smithi078 ceph-mon[99959]: pgmap v641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T20:41:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:18 smithi146 ceph-mon[103278]: pgmap v642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:41:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[91136]: pgmap v642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:41:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[99959]: pgmap v642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 76 op/s 2024-09-20T20:41:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:20.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:20.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:20.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:20.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:41:20] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:41:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:20 smithi146 ceph-mon[103278]: pgmap v643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:41:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-mon[91136]: pgmap v643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:41:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-mon[99959]: pgmap v643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:41:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:22.615 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 32m - - 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 31m - - 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (26m) 9s ago 27m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (26m) 4s ago 26m 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (19m) 9s ago 24m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (18m) 4s ago 31m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (20m) 9s ago 32m 533M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (18m) 9s ago 33m 94.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (17m) 4s ago 31m 65.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (17m) 9s ago 31m 69.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:41:23.144 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (26m) 9s ago 26m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (26m) 4s ago 26m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (16m) 9s ago 30m 60.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (15m) 9s ago 30m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (14m) 9s ago 30m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (14m) 9s ago 29m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (13m) 4s ago 29m 65.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (12m) 4s ago 28m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (11m) 4s ago 28m 60.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (10m) 4s ago 28m 53.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (19m) 4s ago 27m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (10m) 9s ago 25m 81.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (10m) 4s ago 25m 82.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:41:23.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (10m) 9s ago 25m 82.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:41:23.146 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (10m) 4s ago 25m 83.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:41:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:22 smithi146 ceph-mon[103278]: pgmap v644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:41:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:22 smithi078 ceph-mon[99959]: pgmap v644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:41:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:22 smithi078 ceph-mon[91136]: pgmap v644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:41:23.454 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:41:23.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:41:23.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:41:23.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:41:23.456 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:41:23.725 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:41:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:41:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:41:23.726 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:41:23.726 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:23.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:23.726 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:23.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:23.726 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:23.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:24.037 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:41:24.038 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:24.038 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:24.038 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:24.038 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:23 smithi146 ceph-mon[103278]: from='client.65405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:23 smithi146 ceph-mon[103278]: from='client.45492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4004825009' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:23 smithi078 ceph-mon[91136]: from='client.65405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:23 smithi078 ceph-mon[91136]: from='client.45492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4004825009' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[99959]: from='client.65405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[99959]: from='client.45492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4004825009' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:25 smithi146 ceph-mon[103278]: from='client.65417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:25 smithi146 ceph-mon[103278]: pgmap v645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:41:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:25 smithi146 ceph-mon[103278]: from='client.65429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3343209703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:25 smithi078 ceph-mon[99959]: from='client.65417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:25 smithi078 ceph-mon[99959]: pgmap v645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:25 smithi078 ceph-mon[99959]: from='client.65429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3343209703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[91136]: from='client.65417 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[91136]: pgmap v645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[91136]: from='client.65429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3343209703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:27 smithi146 ceph-mon[103278]: pgmap v646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:27 smithi078 ceph-mon[91136]: pgmap v646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:27 smithi078 ceph-mon[99959]: pgmap v646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:29 smithi146 ceph-mon[103278]: pgmap v647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:29 smithi078 ceph-mon[91136]: pgmap v647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:29 smithi078 ceph-mon[99959]: pgmap v647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-mon[91136]: pgmap v648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-mon[99959]: pgmap v648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:30 smithi146 ceph-mon[103278]: pgmap v648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:30.494 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:30.162Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:30.495 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:30.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:30.495 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:30.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:41:30] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:41:32.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:32 smithi078 ceph-mon[91136]: pgmap v649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:32.492 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:32 smithi078 ceph-mon[99959]: pgmap v649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:32 smithi146 ceph-mon[103278]: pgmap v649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:33.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:33.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:33.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[99959]: pgmap v650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[91136]: pgmap v650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:34 smithi146 ceph-mon[103278]: pgmap v650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:36 smithi078 ceph-mon[91136]: pgmap v651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:36 smithi078 ceph-mon[99959]: pgmap v651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:36 smithi146 ceph-mon[103278]: pgmap v651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:38 smithi078 ceph-mon[91136]: pgmap v652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:38 smithi078 ceph-mon[99959]: pgmap v652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:38.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:38 smithi146 ceph-mon[103278]: pgmap v652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:40.162Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:40.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:40.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:40 smithi146 ceph-mon[103278]: pgmap v653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-mon[91136]: pgmap v653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:41:40] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:41:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-mon[99959]: pgmap v653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:42 smithi146 ceph-mon[103278]: pgmap v654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:42 smithi078 ceph-mon[99959]: pgmap v654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:42 smithi078 ceph-mon[91136]: pgmap v654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:43.602Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:43.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:43.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:44 smithi146 ceph-mon[103278]: pgmap v655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:44 smithi078 ceph-mon[91136]: pgmap v655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:44 smithi078 ceph-mon[99959]: pgmap v655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:47.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:46 smithi146 ceph-mon[103278]: pgmap v656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:46 smithi078 ceph-mon[99959]: pgmap v656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:46 smithi078 ceph-mon[91136]: pgmap v656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:48.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:48 smithi146 ceph-mon[103278]: pgmap v657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:48 smithi078 ceph-mon[99959]: pgmap v657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:48 smithi078 ceph-mon[91136]: pgmap v657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:50.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:50.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:50.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:50 smithi146 ceph-mon[103278]: pgmap v658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-mon[91136]: pgmap v658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:51.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:41:50] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:41:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-mon[99959]: pgmap v658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:41:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:52 smithi078 ceph-mon[91136]: pgmap v659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:52 smithi078 ceph-mon[99959]: pgmap v659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:52 smithi146 ceph-mon[103278]: pgmap v659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:53.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:53.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:41:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:41:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:41:53.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:41:54.349 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:41:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:41:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 32m - - 2024-09-20T20:41:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 32m - - 2024-09-20T20:41:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (26m) 0s ago 27m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:41:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (27m) 16s ago 27m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:41:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (20m) 0s ago 25m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (18m) 16s ago 31m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (20m) 0s ago 33m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (19m) 0s ago 33m 95.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (17m) 16s ago 32m 66.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (18m) 0s ago 32m 70.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (27m) 0s ago 27m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (27m) 16s ago 27m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (16m) 0s ago 31m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (15m) 0s ago 31m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (15m) 0s ago 30m 56.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (14m) 0s ago 30m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (14m) 16s ago 29m 65.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (13m) 16s ago 29m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (12m) 16s ago 29m 60.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:41:54.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (11m) 16s ago 28m 53.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:41:54.881 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (20m) 16s ago 28m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:41:54.881 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (11m) 0s ago 25m 81.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:41:54.881 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (10m) 16s ago 25m 82.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:41:54.881 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (10m) 0s ago 25m 82.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:41:54.881 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (10m) 16s ago 25m 84.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:41:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:54 smithi146 ceph-mon[103278]: pgmap v660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[99959]: pgmap v660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[91136]: pgmap v660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:41:55.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:41:55.218 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:41:55.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:41:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:41:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:41:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:41:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:41:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:41:55.491 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[91136]: from='client.45522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[91136]: from='client.65447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[91136]: from='client.45534 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2090556771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[99959]: from='client.45522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[99959]: from='client.65447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[99959]: from='client.45534 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2090556771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:55.804 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:41:55.805 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:41:55.805 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:41:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:55 smithi146 ceph-mon[103278]: from='client.45522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:55 smithi146 ceph-mon[103278]: from='client.65447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:55 smithi146 ceph-mon[103278]: from='client.45534 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2090556771' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:41:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:56 smithi146 ceph-mon[103278]: pgmap v661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:56 smithi146 ceph-mon[103278]: from='client.65465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1496524353' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:56 smithi078 ceph-mon[91136]: pgmap v661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:56 smithi078 ceph-mon[91136]: from='client.65465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1496524353' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:56 smithi078 ceph-mon[99959]: pgmap v661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:56 smithi078 ceph-mon[99959]: from='client.65465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:41:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1496524353' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:41:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:58 smithi146 ceph-mon[103278]: pgmap v662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:58 smithi078 ceph-mon[91136]: pgmap v662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:41:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:58 smithi078 ceph-mon[99959]: pgmap v662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:41:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:41:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:41:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.210 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.210 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.210 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:41:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:00.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:00.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:00.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-mon[99959]: pgmap v663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-mon[91136]: pgmap v663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:42:00] "GET /metrics HTTP/1.1" 200 38984 "" "Prometheus/2.43.0" 2024-09-20T20:42:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:00 smithi146 ceph-mon[103278]: pgmap v663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:02 smithi146 ceph-mon[103278]: pgmap v664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:02 smithi078 ceph-mon[99959]: pgmap v664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:02 smithi078 ceph-mon[91136]: pgmap v664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:03.604Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:03.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:03.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:04 smithi146 ceph-mon[103278]: pgmap v665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:04 smithi078 ceph-mon[99959]: pgmap v665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:04 smithi078 ceph-mon[91136]: pgmap v665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:07.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:06 smithi146 ceph-mon[103278]: pgmap v666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:06 smithi078 ceph-mon[99959]: pgmap v666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:06 smithi078 ceph-mon[91136]: pgmap v666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:09.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:08 smithi146 ceph-mon[103278]: pgmap v667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:08 smithi078 ceph-mon[99959]: pgmap v667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:08 smithi078 ceph-mon[91136]: pgmap v667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:10.419 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:10.164Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:10.419 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:10.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:10.419 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:10.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:42:10] "GET /metrics HTTP/1.1" 200 38984 "" "Prometheus/2.43.0" 2024-09-20T20:42:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:10 smithi146 ceph-mon[103278]: pgmap v668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-mon[99959]: pgmap v668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-mon[91136]: pgmap v668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:12.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:12 smithi078 ceph-mon[99959]: pgmap v669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:12.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:12 smithi078 ceph-mon[91136]: pgmap v669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:12 smithi146 ceph-mon[103278]: pgmap v669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:13.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:13.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:13.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:13.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:13.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:13.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:13.949 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:13.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:14 smithi146 ceph-mon[103278]: pgmap v670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[91136]: pgmap v670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[99959]: pgmap v670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:15.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:16 smithi146 ceph-mon[103278]: pgmap v671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:16 smithi078 ceph-mon[91136]: pgmap v671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:16 smithi078 ceph-mon[99959]: pgmap v671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:18 smithi146 ceph-mon[103278]: pgmap v672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:18 smithi078 ceph-mon[91136]: pgmap v672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:18 smithi078 ceph-mon[99959]: pgmap v672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:20.165Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:20.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:20.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:20.909 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:42:20] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:42:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:20 smithi146 ceph-mon[103278]: pgmap v673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-mon[91136]: pgmap v673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-mon[99959]: pgmap v673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:22 smithi146 ceph-mon[103278]: pgmap v674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:22 smithi078 ceph-mon[91136]: pgmap v674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:22 smithi078 ceph-mon[99959]: pgmap v674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:23.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:23.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:23.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:24 smithi146 ceph-mon[103278]: pgmap v675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:24 smithi078 ceph-mon[91136]: pgmap v675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:24 smithi078 ceph-mon[99959]: pgmap v675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:26.101 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 33m - - 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 32m - - 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (27m) 12s ago 28m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (28m) 7s ago 28m 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (20m) 12s ago 26m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (19m) 7s ago 32m 455M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (21m) 12s ago 33m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:42:26.630 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (19m) 12s ago 34m 96.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (18m) 7s ago 32m 66.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (18m) 12s ago 32m 71.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (27m) 12s ago 27m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (27m) 7s ago 27m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (17m) 12s ago 31m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (16m) 12s ago 31m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (15m) 12s ago 31m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (15m) 12s ago 30m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (14m) 7s ago 30m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (13m) 7s ago 29m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (12m) 7s ago 29m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (11m) 7s ago 29m 53.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (20m) 7s ago 28m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:42:26.631 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (11m) 12s ago 26m 81.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:42:26.632 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (11m) 7s ago 26m 82.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:42:26.632 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (11m) 12s ago 26m 82.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:42:26.632 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (11m) 7s ago 26m 84.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:42:26.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:42:26.945 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:42:27.211 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:42:27.211 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:42:27.212 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:42:27.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:26 smithi078 ceph-mon[91136]: pgmap v676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:27.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:26 smithi078 ceph-mon[99959]: pgmap v676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:26 smithi146 ceph-mon[103278]: pgmap v676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:27.524 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:42:27.524 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:27.524 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:27.524 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:27.524 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:27.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:27 smithi146 ceph-mon[103278]: from='client.65477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:27.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:27 smithi146 ceph-mon[103278]: from='client.45564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:27.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:27 smithi146 ceph-mon[103278]: from='client.45570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:27.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4261079542' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:27.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/933924535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[99959]: from='client.65477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[99959]: from='client.45564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[99959]: from='client.45570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4261079542' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/933924535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[91136]: from='client.65477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[91136]: from='client.45564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[91136]: from='client.45570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4261079542' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/933924535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:42:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:28 smithi146 ceph-mon[103278]: from='client.45582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:28 smithi146 ceph-mon[103278]: pgmap v677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:28 smithi078 ceph-mon[99959]: from='client.45582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:28 smithi078 ceph-mon[99959]: pgmap v677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:28 smithi078 ceph-mon[91136]: from='client.45582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:28 smithi078 ceph-mon[91136]: pgmap v677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:29.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:30.166Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:30.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:30.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:30.928 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:42:30] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:42:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:30 smithi146 ceph-mon[103278]: pgmap v678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-mon[99959]: pgmap v678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:31.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:31.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-mon[91136]: pgmap v678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:31.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:32 smithi078 ceph-mon[91136]: pgmap v679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:32 smithi078 ceph-mon[99959]: pgmap v679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:32 smithi146 ceph-mon[103278]: pgmap v679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:33.860 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:33.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:33.860 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:33.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:33.860 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:33.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:34 smithi146 ceph-mon[103278]: pgmap v680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[99959]: pgmap v680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[91136]: pgmap v680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:35.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:35.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:36 smithi146 ceph-mon[103278]: pgmap v681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:36 smithi078 ceph-mon[91136]: pgmap v681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:36 smithi078 ceph-mon[99959]: pgmap v681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:38 smithi146 ceph-mon[103278]: pgmap v682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:38 smithi078 ceph-mon[91136]: pgmap v682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:38 smithi078 ceph-mon[99959]: pgmap v682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:40.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[99959]: pgmap v683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[91136]: pgmap v683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:40.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:40 smithi146 ceph-mon[103278]: pgmap v683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:40.419 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:40.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:40.420 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:40.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:40.420 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:40.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:42:40] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:42:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:42 smithi146 ceph-mon[103278]: pgmap v684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:42 smithi078 ceph-mon[91136]: pgmap v684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:42 smithi078 ceph-mon[99959]: pgmap v684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:43.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:43.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:43.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:43.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:43.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:44 smithi146 ceph-mon[103278]: pgmap v685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:44 smithi078 ceph-mon[91136]: pgmap v685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:44 smithi078 ceph-mon[99959]: pgmap v685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:46 smithi146 ceph-mon[103278]: pgmap v686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:46 smithi078 ceph-mon[91136]: pgmap v686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:46 smithi078 ceph-mon[99959]: pgmap v686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:48 smithi146 ceph-mon[103278]: pgmap v687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:48 smithi078 ceph-mon[91136]: pgmap v687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:48 smithi078 ceph-mon[99959]: pgmap v687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-mon[91136]: pgmap v688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:50.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:50.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:50.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:50.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-mon[99959]: pgmap v688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:50.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:50.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:50 smithi146 ceph-mon[103278]: pgmap v688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:50.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:42:50.863 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:42:50] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:42:52.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:52 smithi078 ceph-mon[91136]: pgmap v689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:52.586 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:52 smithi078 ceph-mon[99959]: pgmap v689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:52 smithi146 ceph-mon[103278]: pgmap v689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:53.893 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:53.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:53.894 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:53.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:42:53.894 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:42:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:42:53.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:42:54.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:54 smithi078 ceph-mon[91136]: pgmap v690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:54.665 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:54 smithi078 ceph-mon[99959]: pgmap v690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:54 smithi146 ceph-mon[103278]: pgmap v690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:42:55.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:55.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:55.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:55.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:55.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:55.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:42:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:56 smithi146 ceph-mon[103278]: pgmap v691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:56 smithi078 ceph-mon[91136]: pgmap v691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:56 smithi078 ceph-mon[99959]: pgmap v691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:57.818 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:42:58.349 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:42:58.349 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 33m - - 2024-09-20T20:42:58.349 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 33m - - 2024-09-20T20:42:58.349 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (27m) 3s ago 28m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (28m) 19s ago 28m 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (21m) 3s ago 26m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (19m) 19s ago 32m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (21m) 3s ago 34m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (20m) 3s ago 34m 97.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (18m) 19s ago 33m 66.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (19m) 3s ago 33m 71.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (28m) 3s ago 28m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:42:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (28m) 19s ago 28m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (17m) 3s ago 32m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (16m) 3s ago 32m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (16m) 3s ago 31m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (15m) 3s ago 31m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (15m) 19s ago 30m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (14m) 19s ago 30m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (13m) 19s ago 30m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:42:58.351 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (12m) 19s ago 29m 53.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:42:58.352 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (21m) 19s ago 29m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:42:58.352 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (12m) 3s ago 26m 82.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:42:58.352 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (11m) 19s ago 26m 82.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:42:58.352 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (12m) 3s ago 26m 83.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:42:58.352 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (11m) 19s ago 26m 84.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:42:58.657 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:42:58.657 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:42:58.658 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:42:58.659 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:42:58.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:42:58.659 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:42:58.927 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:42:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:42:58.928 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:42:58.928 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:42:58.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[91136]: pgmap v692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:58.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[91136]: from='client.65513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:58.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:58.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1173695476' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:58.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[99959]: pgmap v692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:58.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[99959]: from='client.65513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:58.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:58.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1173695476' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:58 smithi146 ceph-mon[103278]: pgmap v692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:42:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:58 smithi146 ceph-mon[103278]: from='client.65513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:42:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:42:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1173695476' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:42:59.240 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:42:59.240 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:42:59.240 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:42:59.240 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='client.65519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='client.65525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='client.65537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:42:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3513532027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='client.65519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='client.65525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='client.65537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3513532027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='client.65519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='client.65525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='client.65537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:42:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3513532027' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:00.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:00.168Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:00.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:00.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:00.538 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:00.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:00 smithi146 ceph-mon[103278]: pgmap v693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:43:00] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T20:43:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-mon[99959]: pgmap v693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-mon[91136]: pgmap v693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:02 smithi146 ceph-mon[103278]: pgmap v694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:02 smithi078 ceph-mon[99959]: pgmap v694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:02 smithi078 ceph-mon[91136]: pgmap v694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:03.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:03.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:03.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:04 smithi146 ceph-mon[103278]: pgmap v695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:04 smithi078 ceph-mon[91136]: pgmap v695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:04 smithi078 ceph-mon[99959]: pgmap v695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:06 smithi146 ceph-mon[103278]: pgmap v696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:06 smithi078 ceph-mon[99959]: pgmap v696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:06 smithi078 ceph-mon[91136]: pgmap v696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:08 smithi146 ceph-mon[103278]: pgmap v697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:08 smithi078 ceph-mon[99959]: pgmap v697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:08 smithi078 ceph-mon[91136]: pgmap v697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:10.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:10.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:10.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:10.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:10 smithi146 ceph-mon[103278]: pgmap v698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:10.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:10.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-mon[91136]: pgmap v698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:10.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:10.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-mon[99959]: pgmap v698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:10.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:10.987 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:43:10] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T20:43:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:12 smithi146 ceph-mon[103278]: pgmap v699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:12 smithi078 ceph-mon[91136]: pgmap v699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:12 smithi078 ceph-mon[99959]: pgmap v699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:13.610Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:13.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:13.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:14 smithi146 ceph-mon[103278]: pgmap v700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:14 smithi078 ceph-mon[91136]: pgmap v700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:14 smithi078 ceph-mon[99959]: pgmap v700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:16.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:16 smithi146 ceph-mon[103278]: pgmap v701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:16 smithi078 ceph-mon[91136]: pgmap v701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:16 smithi078 ceph-mon[99959]: pgmap v701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:18 smithi146 ceph-mon[103278]: pgmap v702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:18 smithi078 ceph-mon[91136]: pgmap v702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:18 smithi078 ceph-mon[99959]: pgmap v702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:20 smithi146 ceph-mon[103278]: pgmap v703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[91136]: pgmap v703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[99959]: pgmap v703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:20.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:20.170Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:20.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:20.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:20.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:20.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:43:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:43:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:22 smithi078 ceph-mon[91136]: pgmap v704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:22 smithi078 ceph-mon[99959]: pgmap v704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:22 smithi146 ceph-mon[103278]: pgmap v704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:23.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:23.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:23.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:24 smithi078 ceph-mon[91136]: pgmap v705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:24 smithi078 ceph-mon[99959]: pgmap v705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:24 smithi146 ceph-mon[103278]: pgmap v705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:26 smithi078 ceph-mon[91136]: pgmap v706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:26 smithi078 ceph-mon[99959]: pgmap v706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:26 smithi146 ceph-mon[103278]: pgmap v706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:28 smithi078 ceph-mon[91136]: pgmap v707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:28 smithi078 ceph-mon[99959]: pgmap v707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:28 smithi146 ceph-mon[103278]: pgmap v707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:29.536 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:43:30.058 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:43:30.058 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 34m - - 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 33m - - 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (28m) 15s ago 29m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (29m) 10s ago 29m 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (21m) 15s ago 27m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (20m) 10s ago 33m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (22m) 15s ago 34m 534M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (20m) 15s ago 35m 98.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (19m) 10s ago 33m 68.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (19m) 15s ago 33m 71.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (28m) 15s ago 28m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (28m) 10s ago 28m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (18m) 15s ago 32m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:43:30.059 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (17m) 15s ago 32m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (16m) 15s ago 32m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (16m) 15s ago 31m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (15m) 10s ago 31m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (14m) 10s ago 31m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (13m) 10s ago 30m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (12m) 10s ago 30m 53.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (21m) 10s ago 29m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (12m) 15s ago 27m 82.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (12m) 10s ago 27m 83.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (12m) 15s ago 27m 83.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:43:30.060 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (12m) 10s ago 27m 85.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:43:30.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:43:30.376 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:43:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:30 smithi146 ceph-mon[103278]: pgmap v708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:30 smithi146 ceph-mon[103278]: from='client.65549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:30 smithi146 ceph-mon[103278]: from='client.65555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[91136]: pgmap v708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[91136]: from='client.65549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[91136]: from='client.65555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[99959]: pgmap v708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[99959]: from='client.65549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[99959]: from='client.65555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:30.170Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:30.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:30.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:30.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:30.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:43:30.646 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:43:30.647 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:43:30.647 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:43:30.647 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:43:30] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:43:30.966 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:43:30.966 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:43:30.966 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:43:30.966 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:30.966 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:43:31.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:31 smithi146 ceph-mon[103278]: from='client.45642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:31.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/143562746' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:31.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:31 smithi146 ceph-mon[103278]: from='client.45654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:31.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/46202950' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[91136]: from='client.45642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/143562746' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[91136]: from='client.45654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/46202950' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[99959]: from='client.45642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/143562746' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[99959]: from='client.45654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:43:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/46202950' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:43:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:32 smithi146 ceph-mon[103278]: pgmap v709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:32 smithi078 ceph-mon[91136]: pgmap v709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:32 smithi078 ceph-mon[99959]: pgmap v709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:33.989 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:33.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:33.989 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:33.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:33.989 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:33.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:34 smithi078 ceph-mon[91136]: pgmap v710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:34 smithi078 ceph-mon[99959]: pgmap v710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:34 smithi146 ceph-mon[103278]: pgmap v710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:36.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:36 smithi146 ceph-mon[103278]: pgmap v711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:36 smithi078 ceph-mon[99959]: pgmap v711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:36 smithi078 ceph-mon[91136]: pgmap v711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:38 smithi146 ceph-mon[103278]: pgmap v712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:38 smithi078 ceph-mon[91136]: pgmap v712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:38 smithi078 ceph-mon[99959]: pgmap v712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[91136]: pgmap v713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[99959]: pgmap v713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:40.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:40.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:40.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:40.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:40.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:40.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:40.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:40 smithi146 ceph-mon[103278]: pgmap v713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:43:40] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:43:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:42 smithi146 ceph-mon[103278]: pgmap v714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:42 smithi078 ceph-mon[91136]: pgmap v714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:42 smithi078 ceph-mon[99959]: pgmap v714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:43.613Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:43.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:43.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:43.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:44 smithi146 ceph-mon[103278]: pgmap v715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:44 smithi078 ceph-mon[91136]: pgmap v715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:44 smithi078 ceph-mon[99959]: pgmap v715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:46 smithi146 ceph-mon[103278]: pgmap v716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:46 smithi078 ceph-mon[91136]: pgmap v716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:46 smithi078 ceph-mon[99959]: pgmap v716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:48 smithi146 ceph-mon[103278]: pgmap v717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:48 smithi078 ceph-mon[91136]: pgmap v717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:48 smithi078 ceph-mon[99959]: pgmap v717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:50.431 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:50.172Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:50.431 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:50.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:50.431 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:50.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:50.734 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:50 smithi146 ceph-mon[103278]: pgmap v718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:50.734 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-mon[91136]: pgmap v718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:43:50] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:43:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-mon[99959]: pgmap v718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:43:52.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:52 smithi146 ceph-mon[103278]: pgmap v719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:52 smithi078 ceph-mon[91136]: pgmap v719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:52 smithi078 ceph-mon[99959]: pgmap v719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:53.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:53.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:43:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:43:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:43:53.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:43:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:54 smithi146 ceph-mon[103278]: pgmap v720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:54 smithi078 ceph-mon[91136]: pgmap v720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:54 smithi078 ceph-mon[99959]: pgmap v720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[91136]: pgmap v721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:56 smithi078 ceph-mon[99959]: pgmap v721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:43:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:56 smithi146 ceph-mon[103278]: pgmap v721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:58 smithi146 ceph-mon[103278]: pgmap v722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:58 smithi078 ceph-mon[91136]: pgmap v722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:43:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:58 smithi078 ceph-mon[99959]: pgmap v722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:43:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:00.173Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:00.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:00.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:00 smithi146 ceph-mon[103278]: pgmap v723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[99959]: pgmap v723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[91136]: pgmap v723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:44:00] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:44:01.265 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 34m - - 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 34m - - 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (28m) 6s ago 29m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (29m) 2s ago 29m 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (22m) 6s ago 27m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (20m) 2s ago 33m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (22m) 6s ago 35m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (21m) 6s ago 35m 99.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (19m) 2s ago 34m 68.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (20m) 6s ago 34m 73.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:44:01.798 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (29m) 6s ago 29m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (29m) 2s ago 29m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (18m) 6s ago 33m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (17m) 6s ago 33m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (17m) 6s ago 32m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (16m) 6s ago 32m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (16m) 2s ago 31m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (15m) 2s ago 31m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (14m) 2s ago 31m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (13m) 2s ago 30m 53.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (22m) 2s ago 30m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (13m) 6s ago 27m 82.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (13m) 2s ago 28m 83.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (13m) 6s ago 27m 83.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:44:01.799 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (12m) 2s ago 27m 85.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:44:02.105 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:44:02.105 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:44:02.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:44:02.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:02.105 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:44:02.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:44:02.106 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:44:02.376 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:44:02.376 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:44:02.377 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:44:02.688 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:44:02.688 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:02.689 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:02.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[91136]: from='client.45666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:02.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[91136]: pgmap v724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:02.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[91136]: from='client.65591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:02.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[91136]: from='client.45678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:02.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/110541722' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:02.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[99959]: from='client.45666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[99959]: pgmap v724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[99959]: from='client.65591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[99959]: from='client.45678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:02.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/110541722' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:02 smithi146 ceph-mon[103278]: from='client.45666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:02 smithi146 ceph-mon[103278]: pgmap v724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:02 smithi146 ceph-mon[103278]: from='client.65591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:02 smithi146 ceph-mon[103278]: from='client.45678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/110541722' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:03 smithi146 ceph-mon[103278]: from='client.45690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1653944171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-mon[99959]: from='client.45690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1653944171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-mon[91136]: from='client.45690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1653944171' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:03.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:03.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:03.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:04 smithi146 ceph-mon[103278]: pgmap v725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:04 smithi078 ceph-mon[99959]: pgmap v725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:04 smithi078 ceph-mon[91136]: pgmap v725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:06 smithi146 ceph-mon[103278]: pgmap v726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:06 smithi078 ceph-mon[99959]: pgmap v726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:06 smithi078 ceph-mon[91136]: pgmap v726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:08 smithi146 ceph-mon[103278]: pgmap v727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:08 smithi078 ceph-mon[91136]: pgmap v727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:08 smithi078 ceph-mon[99959]: pgmap v727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:10.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:10.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:10.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:10.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:10 smithi146 ceph-mon[103278]: pgmap v728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:10.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:10.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-mon[91136]: pgmap v728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:10.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:10.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-mon[99959]: pgmap v728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:10.794 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:10.794 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:44:10] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T20:44:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:12 smithi146 ceph-mon[103278]: pgmap v729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:12 smithi078 ceph-mon[99959]: pgmap v729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:12 smithi078 ceph-mon[91136]: pgmap v729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:13.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:13.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:13.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:14 smithi146 ceph-mon[103278]: pgmap v730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:14 smithi078 ceph-mon[91136]: pgmap v730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:14 smithi078 ceph-mon[99959]: pgmap v730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:16 smithi146 ceph-mon[103278]: pgmap v731: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[91136]: pgmap v731: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:16 smithi078 ceph-mon[99959]: pgmap v731: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:18 smithi078 ceph-mon[91136]: pgmap v732: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:18 smithi078 ceph-mon[99959]: pgmap v732: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:18 smithi146 ceph-mon[103278]: pgmap v732: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:20.175Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:20.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:20.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:20 smithi146 ceph-mon[103278]: pgmap v733: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:44:20] "GET /metrics HTTP/1.1" 200 38981 "" "Prometheus/2.43.0" 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[99959]: pgmap v733: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[91136]: pgmap v733: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:22 smithi146 ceph-mon[103278]: pgmap v734: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:22 smithi078 ceph-mon[99959]: pgmap v734: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:22 smithi078 ceph-mon[91136]: pgmap v734: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:23.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:23.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:23.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:24 smithi146 ceph-mon[103278]: pgmap v735: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:24 smithi078 ceph-mon[99959]: pgmap v735: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:24 smithi078 ceph-mon[91136]: pgmap v735: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T20:44:25.847363414Z level=info msg="Completed cleanup jobs" duration=11.953496ms 2024-09-20T20:44:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:26 smithi146 ceph-mon[103278]: pgmap v736: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:26 smithi078 ceph-mon[99959]: pgmap v736: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:26 smithi078 ceph-mon[91136]: pgmap v736: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:28 smithi146 ceph-mon[103278]: pgmap v737: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:28 smithi078 ceph-mon[99959]: pgmap v737: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:28 smithi078 ceph-mon[91136]: pgmap v737: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:30.175Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:30.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:30.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:30 smithi146 ceph-mon[103278]: pgmap v738: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:44:30] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:44:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-mon[99959]: pgmap v738: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-mon[91136]: pgmap v738: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:32 smithi146 ceph-mon[103278]: pgmap v739: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:32.983 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:32 smithi078 ceph-mon[91136]: pgmap v739: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:32.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:32 smithi078 ceph-mon[99959]: pgmap v739: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 35m - - 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 34m - - 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (29m) 18s ago 30m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (30m) 13s ago 30m 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (22m) 18s ago 28m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (21m) 13s ago 34m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (23m) 18s ago 35m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (21m) 18s ago 36m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (20m) 13s ago 34m 69.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (20m) 18s ago 34m 74.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:44:33.511 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (29m) 18s ago 29m 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (29m) 13s ago 29m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (19m) 18s ago 34m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (18m) 18s ago 33m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (17m) 18s ago 33m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (17m) 18s ago 32m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (16m) 13s ago 32m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (15m) 13s ago 32m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (14m) 13s ago 31m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (13m) 13s ago 31m 53.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (22m) 13s ago 30m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (13m) 18s ago 28m 83.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (13m) 13s ago 28m 83.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:44:33.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (13m) 18s ago 28m 84.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:44:33.513 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (13m) 13s ago 28m 85.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:44:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:33 smithi078 ceph-mon[99959]: from='client.65621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:33 smithi078 ceph-mon[91136]: from='client.65621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:33.616Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:33.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:33.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:33.825 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:44:33.826 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:44:33.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:44:33.826 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:44:33.826 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:44:33.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:44:33.826 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:44:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:33 smithi146 ceph-mon[103278]: from='client.65621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:44:34.093 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:44:34.094 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:44:34.094 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:44:34.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:44:34.403 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:44:34.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:34.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:44:34.404 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:34.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:44:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:34 smithi146 ceph-mon[103278]: from='client.45708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:34 smithi146 ceph-mon[103278]: pgmap v740: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:34 smithi146 ceph-mon[103278]: from='client.45714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1156465108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/113785117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[91136]: from='client.45708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[91136]: pgmap v740: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[91136]: from='client.45714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1156465108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/113785117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[99959]: from='client.45708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[99959]: pgmap v740: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[99959]: from='client.45714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1156465108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:44:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/113785117' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:44:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:35 smithi146 ceph-mon[103278]: from='client.45726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[91136]: from='client.45726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[99959]: from='client.45726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:44:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:36 smithi146 ceph-mon[103278]: pgmap v741: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:36 smithi078 ceph-mon[91136]: pgmap v741: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:36 smithi078 ceph-mon[99959]: pgmap v741: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:38 smithi146 ceph-mon[103278]: pgmap v742: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:38 smithi078 ceph-mon[91136]: pgmap v742: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:38 smithi078 ceph-mon[99959]: pgmap v742: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:40.176Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:40.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:40.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:40.813 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:44:40] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:44:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:40 smithi146 ceph-mon[103278]: pgmap v743: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[91136]: pgmap v743: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[99959]: pgmap v743: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:42 smithi146 ceph-mon[103278]: pgmap v744: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:42 smithi078 ceph-mon[91136]: pgmap v744: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:42 smithi078 ceph-mon[99959]: pgmap v744: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:43.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:43.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:43.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:44 smithi146 ceph-mon[103278]: pgmap v745: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:44 smithi078 ceph-mon[99959]: pgmap v745: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:44 smithi078 ceph-mon[91136]: pgmap v745: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:46 smithi146 ceph-mon[103278]: pgmap v746: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:46 smithi078 ceph-mon[99959]: pgmap v746: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:46 smithi078 ceph-mon[91136]: pgmap v746: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:48 smithi146 ceph-mon[103278]: pgmap v747: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:48 smithi078 ceph-mon[99959]: pgmap v747: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:48 smithi078 ceph-mon[91136]: pgmap v747: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:50.179 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:50.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:50.179 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:50.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:50.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-mon[91136]: pgmap v748: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:44:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:44:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-mon[99959]: pgmap v748: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:51.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:50 smithi146 ceph-mon[103278]: pgmap v748: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:51.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:44:53.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:52 smithi146 ceph-mon[103278]: pgmap v749: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:52 smithi078 ceph-mon[91136]: pgmap v749: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:52 smithi078 ceph-mon[99959]: pgmap v749: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:53.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:53.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:44:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:44:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:44:53.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:44:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:54 smithi146 ceph-mon[103278]: pgmap v750: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:54 smithi078 ceph-mon[91136]: pgmap v750: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:54 smithi078 ceph-mon[99959]: pgmap v750: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:44:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:56 smithi146 ceph-mon[103278]: pgmap v751: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[99959]: pgmap v751: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[91136]: pgmap v751: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:44:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:58 smithi146 ceph-mon[103278]: pgmap v752: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:58 smithi078 ceph-mon[91136]: pgmap v752: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:44:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:58 smithi078 ceph-mon[99959]: pgmap v752: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:44:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:00.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:00.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:00.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:00.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:45:00] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:45:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:00 smithi146 ceph-mon[103278]: pgmap v753: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[91136]: pgmap v753: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[99959]: pgmap v753: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:02 smithi146 ceph-mon[103278]: pgmap v754: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:02 smithi078 ceph-mon[91136]: pgmap v754: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:02 smithi078 ceph-mon[99959]: pgmap v754: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:03.618Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:03.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:03.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:04.704 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:45:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:04 smithi078 ceph-mon[91136]: pgmap v755: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:04 smithi078 ceph-mon[99959]: pgmap v755: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:05.235 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:45:05.235 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 35m - - 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 35m - - 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (29m) 9s ago 30m 31.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (30m) 5s ago 30m 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (23m) 9s ago 28m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (22m) 5s ago 34m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (23m) 9s ago 36m 535M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (22m) 9s ago 36m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (20m) 5s ago 35m 70.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:45:05.236 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (21m) 9s ago 35m 75.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (30m) 9s ago 30m 15.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (30m) 5s ago 30m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (19m) 9s ago 34m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (18m) 9s ago 34m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (18m) 9s ago 33m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (17m) 9s ago 33m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (17m) 5s ago 33m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (16m) 5s ago 32m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:45:05.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (15m) 5s ago 32m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:45:05.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (14m) 5s ago 31m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:45:05.238 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (23m) 5s ago 31m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:45:05.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (14m) 9s ago 29m 83.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:45:05.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (14m) 5s ago 29m 84.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:45:05.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (14m) 9s ago 28m 84.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:45:05.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (13m) 5s ago 28m 85.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:45:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:04 smithi146 ceph-mon[103278]: pgmap v755: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:05.547 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:45:05.547 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:45:05.548 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:45:05.814 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:45:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:45:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:45:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:45:05.814 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:45:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:45:06.124 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:45:06.124 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:45:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:45:06.124 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:45:06.124 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:45:06.125 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:05 smithi078 ceph-mon[99959]: from='client.65657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:06.125 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:05 smithi078 ceph-mon[99959]: from='client.45744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:06.125 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1741496491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:06.125 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:05 smithi078 ceph-mon[91136]: from='client.65657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:06.125 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:05 smithi078 ceph-mon[91136]: from='client.45744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:06.125 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1741496491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:05 smithi146 ceph-mon[103278]: from='client.65657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:05 smithi146 ceph-mon[103278]: from='client.45744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1741496491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:06 smithi146 ceph-mon[103278]: from='client.65669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:06 smithi146 ceph-mon[103278]: pgmap v756: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:06 smithi146 ceph-mon[103278]: from='client.65681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/328834525' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[91136]: from='client.65669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[91136]: pgmap v756: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[91136]: from='client.65681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/328834525' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[99959]: from='client.65669 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[99959]: pgmap v756: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[99959]: from='client.65681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/328834525' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:08 smithi146 ceph-mon[103278]: pgmap v757: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:08 smithi078 ceph-mon[91136]: pgmap v757: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:08 smithi078 ceph-mon[99959]: pgmap v757: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:10.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:10.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:10.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:10.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:10.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:10.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:45:10] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:45:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:10 smithi146 ceph-mon[103278]: pgmap v758: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-mon[99959]: pgmap v758: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-mon[91136]: pgmap v758: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:12 smithi146 ceph-mon[103278]: pgmap v759: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:12 smithi078 ceph-mon[91136]: pgmap v759: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:12 smithi078 ceph-mon[99959]: pgmap v759: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:13.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:13.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:13.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:13.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:13.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:13.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:14.217 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:14 smithi146 ceph-mon[103278]: pgmap v760: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:14 smithi078 ceph-mon[91136]: pgmap v760: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:14 smithi078 ceph-mon[99959]: pgmap v760: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:16 smithi146 ceph-mon[103278]: pgmap v761: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[99959]: pgmap v761: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[91136]: pgmap v761: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:18.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:18 smithi078 ceph-mon[91136]: pgmap v762: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:18.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:18 smithi078 ceph-mon[99959]: pgmap v762: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:18 smithi146 ceph-mon[103278]: pgmap v762: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:20.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:20.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:20.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:20 smithi146 ceph-mon[103278]: pgmap v763: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:45:20] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:45:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[99959]: pgmap v763: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[91136]: pgmap v763: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:22 smithi146 ceph-mon[103278]: pgmap v764: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:22 smithi078 ceph-mon[91136]: pgmap v764: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:22 smithi078 ceph-mon[99959]: pgmap v764: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:23.620Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:23.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:23.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:24 smithi146 ceph-mon[103278]: pgmap v765: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:24 smithi078 ceph-mon[91136]: pgmap v765: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:24 smithi078 ceph-mon[99959]: pgmap v765: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:26 smithi146 ceph-mon[103278]: pgmap v766: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:26 smithi078 ceph-mon[91136]: pgmap v766: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:26 smithi078 ceph-mon[99959]: pgmap v766: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:28 smithi146 ceph-mon[103278]: pgmap v767: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:28 smithi078 ceph-mon[91136]: pgmap v767: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:28 smithi078 ceph-mon[99959]: pgmap v767: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:30.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:30.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:30.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:30.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:30.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:30.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:30 smithi146 ceph-mon[103278]: pgmap v768: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-mon[99959]: pgmap v768: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-mon[91136]: pgmap v768: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:45:30] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:45:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:32 smithi146 ceph-mon[103278]: pgmap v769: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:32 smithi078 ceph-mon[91136]: pgmap v769: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:32 smithi078 ceph-mon[99959]: pgmap v769: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:33.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:33.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:33.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:34.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:34 smithi146 ceph-mon[103278]: pgmap v770: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:34 smithi078 ceph-mon[91136]: pgmap v770: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:34 smithi078 ceph-mon[99959]: pgmap v770: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:36.416 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:45:36.949 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:45:36.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 36m - - 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 36m - - 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (30m) 1s ago 31m 31.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (31m) 16s ago 31m 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (23m) 1s ago 29m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (22m) 16s ago 35m 456M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (24m) 1s ago 37m 536M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (22m) 1s ago 37m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (21m) 16s ago 35m 71.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (21m) 1s ago 35m 73.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (30m) 1s ago 30m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (30m) 16s ago 30m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:45:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (20m) 1s ago 35m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (19m) 1s ago 34m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (18m) 1s ago 34m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (18m) 1s ago 33m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (17m) 16s ago 33m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (16m) 16s ago 33m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (15m) 16s ago 32m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (15m) 16s ago 32m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (23m) 16s ago 31m 63.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (14m) 1s ago 29m 83.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (14m) 16s ago 29m 84.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (14m) 1s ago 29m 84.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:45:36.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (14m) 16s ago 29m 86.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:45:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:36 smithi146 ceph-mon[103278]: pgmap v771: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[99959]: pgmap v771: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[91136]: pgmap v771: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:37.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:45:37.259 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:45:37.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:45:37.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:37.259 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:45:37.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:45:37.260 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:45:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:45:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:45:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:45:37.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[99959]: from='client.45774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[99959]: from='client.45780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[99959]: from='client.45786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/160815832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[91136]: from='client.45774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[91136]: from='client.45780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[91136]: from='client.45786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/160815832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:37.839 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:45:37.839 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:45:37.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:45:37.839 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:45:37.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:45:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:37 smithi146 ceph-mon[103278]: from='client.45774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:37 smithi146 ceph-mon[103278]: from='client.45780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:37 smithi146 ceph-mon[103278]: from='client.45786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/160815832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:45:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:38 smithi146 ceph-mon[103278]: pgmap v772: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:38 smithi146 ceph-mon[103278]: from='client.45798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/53289998' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:38 smithi078 ceph-mon[91136]: pgmap v772: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:38 smithi078 ceph-mon[91136]: from='client.45798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/53289998' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:38 smithi078 ceph-mon[99959]: pgmap v772: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:38 smithi078 ceph-mon[99959]: from='client.45798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:45:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/53289998' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:45:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:40.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:40.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:40.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:40.950 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:45:40] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:45:40.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[99959]: pgmap v773: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[91136]: pgmap v773: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:40.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:40 smithi146 ceph-mon[103278]: pgmap v773: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:42 smithi146 ceph-mon[103278]: pgmap v774: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:42 smithi078 ceph-mon[91136]: pgmap v774: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:42 smithi078 ceph-mon[99959]: pgmap v774: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:43.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 9 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:43.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:43.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:44 smithi146 ceph-mon[103278]: pgmap v775: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:44 smithi078 ceph-mon[91136]: pgmap v775: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:44 smithi078 ceph-mon[99959]: pgmap v775: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:46 smithi146 ceph-mon[103278]: pgmap v776: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:46 smithi078 ceph-mon[91136]: pgmap v776: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:46 smithi078 ceph-mon[99959]: pgmap v776: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:48 smithi146 ceph-mon[103278]: pgmap v777: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:48 smithi078 ceph-mon[91136]: pgmap v777: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:48 smithi078 ceph-mon[99959]: pgmap v777: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:50.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:50.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:50.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:50 smithi146 ceph-mon[103278]: pgmap v778: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:51.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-mon[91136]: pgmap v778: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:45:50] "GET /metrics HTTP/1.1" 200 38984 "" "Prometheus/2.43.0" 2024-09-20T20:45:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-mon[99959]: pgmap v778: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:45:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:52 smithi146 ceph-mon[103278]: pgmap v779: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:52 smithi078 ceph-mon[91136]: pgmap v779: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:52 smithi078 ceph-mon[99959]: pgmap v779: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:53.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:53.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:45:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:45:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:45:53.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:45:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:54 smithi146 ceph-mon[103278]: pgmap v780: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:54 smithi078 ceph-mon[91136]: pgmap v780: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:54 smithi078 ceph-mon[99959]: pgmap v780: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:45:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:56 smithi146 ceph-mon[103278]: pgmap v781: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[99959]: pgmap v781: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[91136]: pgmap v781: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:45:58.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:58 smithi078 ceph-mon[99959]: pgmap v782: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:58.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:58.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:58 smithi078 ceph-mon[91136]: pgmap v782: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:58.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:45:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:58 smithi146 ceph-mon[103278]: pgmap v782: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:45:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:00.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:00.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:00.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:46:00] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[99959]: pgmap v783: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[91136]: pgmap v783: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:01.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:00 smithi146 ceph-mon[103278]: pgmap v783: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:02 smithi146 ceph-mon[103278]: pgmap v784: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:02 smithi078 ceph-mon[91136]: pgmap v784: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:02 smithi078 ceph-mon[99959]: pgmap v784: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:03.623Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:03.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:03.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:04 smithi146 ceph-mon[103278]: pgmap v785: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:04 smithi078 ceph-mon[91136]: pgmap v785: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:04 smithi078 ceph-mon[99959]: pgmap v785: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:06 smithi146 ceph-mon[103278]: pgmap v786: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:06 smithi078 ceph-mon[91136]: pgmap v786: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:06 smithi078 ceph-mon[99959]: pgmap v786: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:08.139 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:46:08.669 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:46:08.669 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 36m - - 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 36m - - 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (30m) 12s ago 31m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (31m) 8s ago 31m 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (24m) 12s ago 29m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (23m) 8s ago 35m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (25m) 12s ago 37m 536M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (23m) 12s ago 37m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:46:08.670 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (21m) 8s ago 36m 69.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (22m) 12s ago 36m 74.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (31m) 12s ago 31m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (31m) 8s ago 31m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (20m) 12s ago 35m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (20m) 12s ago 35m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (19m) 12s ago 34m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (18m) 12s ago 34m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (18m) 8s ago 34m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (17m) 8s ago 33m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (16m) 8s ago 33m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (15m) 8s ago 32m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (24m) 8s ago 32m 61.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (15m) 12s ago 30m 84.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (15m) 8s ago 30m 84.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:46:08.671 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (15m) 12s ago 30m 85.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:46:08.672 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (15m) 8s ago 29m 86.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:08.976 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:46:08.977 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:46:08.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:08 smithi078 ceph-mon[91136]: pgmap v787: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:08.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:08 smithi078 ceph-mon[99959]: pgmap v787: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:46:09.244 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:46:09.245 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:46:09.245 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:46:09.245 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:46:09.245 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:46:09.245 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:46:09.245 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:46:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:08 smithi146 ceph-mon[103278]: pgmap v787: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:09.555 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:46:09.555 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:09.556 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:10.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[91136]: from='client.65729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[91136]: from='client.45816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[91136]: from='client.45822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1965381918' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1759967563' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[99959]: from='client.65729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[99959]: from='client.45816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[99959]: from='client.45822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1965381918' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:10.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1759967563' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:09 smithi146 ceph-mon[103278]: from='client.65729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:09 smithi146 ceph-mon[103278]: from='client.45816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:09 smithi146 ceph-mon[103278]: from='client.45822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1965381918' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1759967563' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:10.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:10.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:10.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:10.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:46:10] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T20:46:11.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:10 smithi146 ceph-mon[103278]: from='client.45834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:11.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:10 smithi146 ceph-mon[103278]: pgmap v788: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:11.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-mon[91136]: from='client.45834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-mon[91136]: pgmap v788: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-mon[99959]: from='client.45834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-mon[99959]: pgmap v788: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:12 smithi146 ceph-mon[103278]: pgmap v789: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:12 smithi078 ceph-mon[91136]: pgmap v789: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:12 smithi078 ceph-mon[99959]: pgmap v789: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:13.876 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:13.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:13.877 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:13.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:13.877 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:13.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:14.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:14 smithi146 ceph-mon[103278]: pgmap v790: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:14 smithi078 ceph-mon[91136]: pgmap v790: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:14 smithi078 ceph-mon[99959]: pgmap v790: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:16 smithi146 ceph-mon[103278]: pgmap v791: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[91136]: pgmap v791: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[99959]: pgmap v791: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:17.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:18 smithi146 ceph-mon[103278]: pgmap v792: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:18 smithi078 ceph-mon[91136]: pgmap v792: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:18 smithi078 ceph-mon[99959]: pgmap v792: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:20.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:20.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:20.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:20.873 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:46:20] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[91136]: pgmap v793: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[99959]: pgmap v793: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:21.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:21.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:20 smithi146 ceph-mon[103278]: pgmap v793: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:22 smithi146 ceph-mon[103278]: pgmap v794: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:22 smithi078 ceph-mon[91136]: pgmap v794: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:22 smithi078 ceph-mon[99959]: pgmap v794: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:23.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:23.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:23.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:24 smithi146 ceph-mon[103278]: pgmap v795: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:24 smithi078 ceph-mon[91136]: pgmap v795: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:24 smithi078 ceph-mon[99959]: pgmap v795: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:26 smithi146 ceph-mon[103278]: pgmap v796: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:26 smithi078 ceph-mon[91136]: pgmap v796: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:26 smithi078 ceph-mon[99959]: pgmap v796: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:28 smithi146 ceph-mon[103278]: pgmap v797: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:28 smithi078 ceph-mon[91136]: pgmap v797: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:28 smithi078 ceph-mon[99959]: pgmap v797: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:30.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:30.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:30.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:30.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:30.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:30.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:30.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:46:30] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:46:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:30 smithi146 ceph-mon[103278]: pgmap v798: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-mon[91136]: pgmap v798: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-mon[99959]: pgmap v798: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:32 smithi146 ceph-mon[103278]: pgmap v799: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:33 smithi078 ceph-mon[99959]: pgmap v799: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:32 smithi078 ceph-mon[91136]: pgmap v799: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:33.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:33.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:33.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:35.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:35 smithi146 ceph-mon[103278]: pgmap v800: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:35 smithi078 ceph-mon[99959]: pgmap v800: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:35 smithi078 ceph-mon[91136]: pgmap v800: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:37 smithi146 ceph-mon[103278]: pgmap v801: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[99959]: pgmap v801: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[91136]: pgmap v801: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:37.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:39 smithi146 ceph-mon[103278]: pgmap v802: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:39 smithi078 ceph-mon[99959]: pgmap v802: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:39 smithi078 ceph-mon[91136]: pgmap v802: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:46:39.856 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:46:40.394 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 37m - - 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 37m - - 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (31m) 4s ago 32m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (32m) 19s ago 32m 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (24m) 4s ago 30m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (23m) 19s ago 36m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (25m) 4s ago 38m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (23m) 4s ago 38m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (22m) 19s ago 36m 70.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (22m) 4s ago 36m 75.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (32m) 4s ago 31m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (31m) 19s ago 31m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:46:40.395 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (21m) 4s ago 36m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (20m) 4s ago 35m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (19m) 4s ago 35m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (19m) 4s ago 34m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (18m) 19s ago 34m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (17m) 19s ago 34m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (17m) 19s ago 33m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (16m) 19s ago 33m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (25m) 19s ago 32m 61.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (15m) 4s ago 30m 84.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (15m) 19s ago 30m 84.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (15m) 4s ago 30m 85.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:46:40.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (15m) 19s ago 30m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:46:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:40.183Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:40.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:40.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:40.711 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:46:40.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:46:40.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:46:40.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:40.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:46:40.892 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:46:40] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:46:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:46:40.987 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:46:40.987 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:46:40.987 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:46:40.987 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:46:40.987 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:46:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: pgmap v803: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: from='client.65765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/262425679' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: pgmap v803: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: from='client.65765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/262425679' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: pgmap v803: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: from='client.65765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/262425679' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:46:41.304 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:46:41.304 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:41.304 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:46:41.305 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:41.305 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[91136]: from='client.65771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[91136]: from='client.65777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[91136]: from='client.45870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2613404740' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[99959]: from='client.65771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[99959]: from='client.65777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[99959]: from='client.45870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2613404740' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:42 smithi146 ceph-mon[103278]: from='client.65771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:42 smithi146 ceph-mon[103278]: from='client.65777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:42 smithi146 ceph-mon[103278]: from='client.45870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:46:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2613404740' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:46:43.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:43 smithi078 ceph-mon[91136]: pgmap v804: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:43.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:43 smithi078 ceph-mon[99959]: pgmap v804: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:43 smithi146 ceph-mon[103278]: pgmap v804: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:43.985 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:43.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:43.985 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:43.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:43.985 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:43.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:45 smithi078 ceph-mon[91136]: pgmap v805: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:45 smithi078 ceph-mon[99959]: pgmap v805: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:45 smithi146 ceph-mon[103278]: pgmap v805: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:47 smithi078 ceph-mon[91136]: pgmap v806: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:47 smithi078 ceph-mon[99959]: pgmap v806: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:47 smithi146 ceph-mon[103278]: pgmap v806: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:49 smithi078 ceph-mon[99959]: pgmap v807: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:49 smithi078 ceph-mon[91136]: pgmap v807: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:49 smithi146 ceph-mon[103278]: pgmap v807: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:50.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:50.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:50.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:50.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:46:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T20:46:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:51 smithi078 ceph-mon[91136]: pgmap v808: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:51 smithi078 ceph-mon[99959]: pgmap v808: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:51 smithi146 ceph-mon[103278]: pgmap v808: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:46:53.492 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:53 smithi078 ceph-mon[99959]: pgmap v809: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:53.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:53 smithi078 ceph-mon[91136]: pgmap v809: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:53 smithi146 ceph-mon[103278]: pgmap v809: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:53.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:53.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:46:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:46:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:46:53.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:46:55.359 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:55 smithi146 ceph-mon[103278]: pgmap v810: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:55 smithi078 ceph-mon[91136]: pgmap v810: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:55 smithi078 ceph-mon[99959]: pgmap v810: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:46:57.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:57 smithi146 ceph-mon[103278]: pgmap v811: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:57.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[91136]: pgmap v811: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[99959]: pgmap v811: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:57.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:46:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:59 smithi146 ceph-mon[103278]: pgmap v812: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:59 smithi078 ceph-mon[91136]: pgmap v812: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:46:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:59 smithi078 ceph-mon[99959]: pgmap v812: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:46:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:00.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:00.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:00.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:00.908 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:47:00] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[91136]: pgmap v813: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[99959]: pgmap v813: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:01 smithi146 ceph-mon[103278]: pgmap v813: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:03.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:03 smithi078 ceph-mon[91136]: pgmap v814: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:03.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:03 smithi078 ceph-mon[99959]: pgmap v814: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:03 smithi146 ceph-mon[103278]: pgmap v814: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:03.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:03.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:03.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:05 smithi146 ceph-mon[103278]: pgmap v815: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:05 smithi078 ceph-mon[91136]: pgmap v815: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:05 smithi078 ceph-mon[99959]: pgmap v815: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:07 smithi146 ceph-mon[103278]: pgmap v816: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:07 smithi078 ceph-mon[91136]: pgmap v816: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:07 smithi078 ceph-mon[99959]: pgmap v816: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:08 smithi146 ceph-mon[103278]: pgmap v817: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:08 smithi078 ceph-mon[91136]: pgmap v817: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:08 smithi078 ceph-mon[99959]: pgmap v817: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:10.338 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:10.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:10.339 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:10.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:10.339 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:10.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-mon[91136]: pgmap v818: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-mon[99959]: pgmap v818: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:10 smithi146 ceph-mon[103278]: pgmap v818: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:47:10] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T20:47:11.603 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:47:12.129 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:47:12.129 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 37m - - 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 37m - - 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (31m) 15s ago 33m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (32m) 11s ago 32m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (25m) 15s ago 30m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (24m) 11s ago 36m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (26m) 15s ago 38m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (24m) 15s ago 38m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (23m) 11s ago 37m 71.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (23m) 15s ago 37m 77.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (32m) 15s ago 32m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (32m) 11s ago 32m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (22m) 15s ago 36m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:47:12.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (21m) 15s ago 36m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (20m) 15s ago 35m 57.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (19m) 15s ago 35m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (19m) 11s ago 35m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (18m) 11s ago 34m 55.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (17m) 11s ago 34m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (16m) 11s ago 33m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (25m) 11s ago 33m 62.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (16m) 15s ago 31m 84.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (16m) 11s ago 31m 85.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (16m) 15s ago 31m 85.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:47:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (16m) 11s ago 31m 86.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:47:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:47:12.443 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:47:12.710 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:47:12.711 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:47:12.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:12 smithi078 ceph-mon[99959]: pgmap v819: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:12.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:12 smithi078 ceph-mon[99959]: from='client.45882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:12.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:12 smithi078 ceph-mon[99959]: from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:12.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:12 smithi078 ceph-mon[91136]: pgmap v819: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:12.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:12 smithi078 ceph-mon[91136]: from='client.45882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:12.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:12 smithi078 ceph-mon[91136]: from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:12 smithi146 ceph-mon[103278]: pgmap v819: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:12 smithi146 ceph-mon[103278]: from='client.45882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:12 smithi146 ceph-mon[103278]: from='client.45888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:47:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:13.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:13 smithi146 ceph-mon[103278]: from='client.45894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1863071117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:13.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:13 smithi146 ceph-mon[103278]: from='client.45906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1802020151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:13.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[91136]: from='client.45894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1863071117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[91136]: from='client.45906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1802020151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[99959]: from='client.45894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1863071117' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[99959]: from='client.45906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1802020151' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:13.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:13.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:13.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:13.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:13.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:13.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:14 smithi146 ceph-mon[103278]: pgmap v820: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:14 smithi078 ceph-mon[91136]: pgmap v820: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:14 smithi078 ceph-mon[99959]: pgmap v820: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:16 smithi146 ceph-mon[103278]: pgmap v821: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[91136]: pgmap v821: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[99959]: pgmap v821: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:16.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:18 smithi146 ceph-mon[103278]: pgmap v822: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:18 smithi078 ceph-mon[91136]: pgmap v822: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:18 smithi078 ceph-mon[99959]: pgmap v822: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:20.411 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:20.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:20.411 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:20.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:20.411 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:20.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:20 smithi146 ceph-mon[103278]: pgmap v823: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:20 smithi078 ceph-mon[91136]: pgmap v823: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:20 smithi078 ceph-mon[99959]: pgmap v823: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:47:20] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:22 smithi078 ceph-mon[91136]: pgmap v824: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:22 smithi078 ceph-mon[99959]: pgmap v824: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:23.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:22 smithi146 ceph-mon[103278]: pgmap v824: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:23.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:23.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:23.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:23.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:23.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:23.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:24 smithi146 ceph-mon[103278]: pgmap v825: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:24 smithi078 ceph-mon[91136]: pgmap v825: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:24 smithi078 ceph-mon[99959]: pgmap v825: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:26 smithi146 ceph-mon[103278]: pgmap v826: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:26 smithi078 ceph-mon[91136]: pgmap v826: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:26 smithi078 ceph-mon[99959]: pgmap v826: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:28 smithi146 ceph-mon[103278]: pgmap v827: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:29.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:28 smithi078 ceph-mon[99959]: pgmap v827: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:28 smithi078 ceph-mon[91136]: pgmap v827: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:30.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:30.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:30.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:30.865 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:47:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:47:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:30 smithi146 ceph-mon[103278]: pgmap v828: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:30 smithi078 ceph-mon[91136]: pgmap v828: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:30 smithi078 ceph-mon[99959]: pgmap v828: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:32 smithi146 ceph-mon[103278]: pgmap v829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:32 smithi078 ceph-mon[91136]: pgmap v829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:32 smithi078 ceph-mon[99959]: pgmap v829: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:33.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:33.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:33.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:34 smithi146 ceph-mon[103278]: pgmap v830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:34 smithi078 ceph-mon[99959]: pgmap v830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:34 smithi078 ceph-mon[91136]: pgmap v830: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:36 smithi146 ceph-mon[103278]: pgmap v831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[91136]: pgmap v831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[99959]: pgmap v831: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:38 smithi146 ceph-mon[103278]: pgmap v832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:38 smithi078 ceph-mon[91136]: pgmap v832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:38 smithi078 ceph-mon[99959]: pgmap v832: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:40.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:40.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:40.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:40.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:47:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:47:41.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:40 smithi078 ceph-mon[99959]: pgmap v833: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:41.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:40 smithi078 ceph-mon[91136]: pgmap v833: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:40 smithi146 ceph-mon[103278]: pgmap v833: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:43.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:42 smithi078 ceph-mon[91136]: pgmap v834: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:43.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:42 smithi078 ceph-mon[99959]: pgmap v834: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:42 smithi146 ceph-mon[103278]: pgmap v834: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:43.313 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:47:43.711 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:43.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:43.712 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:43.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:43.712 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:43.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 38m - - 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 38m - - 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (32m) 7s ago 33m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (33m) 2s ago 33m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (25m) 7s ago 31m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (24m) 2s ago 37m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (26m) 7s ago 39m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (24m) 7s ago 39m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (23m) 2s ago 37m 73.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (23m) 7s ago 37m 78.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (33m) 7s ago 33m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (32m) 2s ago 32m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (22m) 7s ago 37m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:47:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (21m) 7s ago 36m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (20m) 7s ago 36m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (20m) 7s ago 36m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (19m) 2s ago 35m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (18m) 2s ago 35m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (18m) 2s ago 34m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (17m) 2s ago 34m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (26m) 2s ago 33m 65.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (16m) 7s ago 31m 85.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (16m) 2s ago 31m 85.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (16m) 7s ago 31m 86.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:47:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (16m) 2s ago 31m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:47:44.182 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:47:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:47:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:47:44.183 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:47:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:47:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:47:44.466 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:47:44.788 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:47:44.788 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:47:44.788 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:47:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:44 smithi146 ceph-mon[103278]: from='client.65837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:44 smithi146 ceph-mon[103278]: pgmap v835: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:44 smithi146 ceph-mon[103278]: from='client.65843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:44 smithi146 ceph-mon[103278]: from='client.65849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1605753631' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1793592987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[91136]: from='client.65837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[91136]: pgmap v835: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[91136]: from='client.65843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[91136]: from='client.65849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1605753631' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1793592987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[99959]: from='client.65837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[99959]: pgmap v835: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[99959]: from='client.65843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[99959]: from='client.65849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1605753631' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:47:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1793592987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:47:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:46 smithi146 ceph-mon[103278]: from='client.45942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:46 smithi078 ceph-mon[91136]: from='client.45942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:46 smithi078 ceph-mon[99959]: from='client.45942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:47:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:47 smithi146 ceph-mon[103278]: pgmap v836: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:47 smithi078 ceph-mon[91136]: pgmap v836: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:47 smithi078 ceph-mon[99959]: pgmap v836: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:49 smithi146 ceph-mon[103278]: pgmap v837: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:49 smithi078 ceph-mon[91136]: pgmap v837: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:49 smithi078 ceph-mon[99959]: pgmap v837: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:50.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:50.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:50.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:51.010 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:47:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:47:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:51 smithi146 ceph-mon[103278]: pgmap v838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:51 smithi078 ceph-mon[91136]: pgmap v838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:51 smithi078 ceph-mon[99959]: pgmap v838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:47:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:53 smithi146 ceph-mon[103278]: pgmap v839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:53 smithi078 ceph-mon[91136]: pgmap v839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:53 smithi078 ceph-mon[99959]: pgmap v839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:53.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:53.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:47:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:47:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:47:53.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:47:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:55 smithi078 ceph-mon[91136]: pgmap v840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:55 smithi078 ceph-mon[99959]: pgmap v840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:55 smithi146 ceph-mon[103278]: pgmap v840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:47:57.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:57 smithi146 ceph-mon[103278]: pgmap v841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:57.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[99959]: pgmap v841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[91136]: pgmap v841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:57.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:47:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:59 smithi078 ceph-mon[91136]: pgmap v842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:47:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:59 smithi078 ceph-mon[99959]: pgmap v842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:47:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:47:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:59 smithi146 ceph-mon[103278]: pgmap v842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:47:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:47:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:00.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:00.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:00.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:01.031 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:48:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:48:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:01 smithi078 ceph-mon[91136]: pgmap v843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:01 smithi078 ceph-mon[99959]: pgmap v843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:01 smithi146 ceph-mon[103278]: pgmap v843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:02.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:02.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:03 smithi078 ceph-mon[91136]: pgmap v844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:03 smithi078 ceph-mon[99959]: pgmap v844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:03 smithi146 ceph-mon[103278]: pgmap v844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:03.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:03.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:03.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:05 smithi078 ceph-mon[91136]: pgmap v845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:05 smithi078 ceph-mon[99959]: pgmap v845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:05 smithi146 ceph-mon[103278]: pgmap v845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:07 smithi146 ceph-mon[103278]: pgmap v846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:07 smithi078 ceph-mon[91136]: pgmap v846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:07 smithi078 ceph-mon[99959]: pgmap v846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:09.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:09 smithi146 ceph-mon[103278]: pgmap v847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:09 smithi078 ceph-mon[91136]: pgmap v847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:09 smithi078 ceph-mon[99959]: pgmap v847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:10.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:10.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:10.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:10.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:10.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:10.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:48:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:48:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:11 smithi078 ceph-mon[99959]: pgmap v848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:11 smithi078 ceph-mon[91136]: pgmap v848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:11 smithi146 ceph-mon[103278]: pgmap v848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:12.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:13.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:13 smithi146 ceph-mon[103278]: pgmap v849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:13 smithi078 ceph-mon[91136]: pgmap v849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:13 smithi078 ceph-mon[99959]: pgmap v849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:13.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:13.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:13.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:48:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:15 smithi146 ceph-mon[103278]: pgmap v850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:15 smithi078 ceph-mon[99959]: pgmap v850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:15 smithi078 ceph-mon[91136]: pgmap v850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 38m - - 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 38m - - 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (33m) 18s ago 34m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (33m) 14s ago 33m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (26m) 18s ago 31m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (25m) 14s ago 38m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (27m) 18s ago 39m 537M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:48:15.610 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (25m) 18s ago 39m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (24m) 14s ago 38m 73.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (24m) 18s ago 38m 78.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (33m) 18s ago 33m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (33m) 14s ago 33m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (23m) 18s ago 37m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (22m) 18s ago 37m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (21m) 18s ago 36m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (20m) 18s ago 36m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (20m) 14s ago 36m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (19m) 14s ago 35m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (18m) 14s ago 35m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (17m) 14s ago 35m 54.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:48:15.611 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (26m) 14s ago 34m 65.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:48:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (17m) 18s ago 32m 85.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:48:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (17m) 14s ago 32m 86.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:48:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (17m) 18s ago 32m 86.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:48:15.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (17m) 14s ago 32m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:48:15.915 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:48:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:48:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:48:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:48:15.917 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:48:16.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:48:16.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:48:16.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3375603732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:16.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3375603732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:16.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3375603732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:16.498 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:48:16.498 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:16.498 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='client.65873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: pgmap v851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='client.65885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='client.45978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4182897320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='client.65873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: pgmap v851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='client.65885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='client.45978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4182897320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='client.65873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='client.45960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: pgmap v851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:17.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='client.65885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='client.45978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:17.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4182897320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:17.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:17.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:18 smithi146 ceph-mon[103278]: pgmap v852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:18 smithi078 ceph-mon[91136]: pgmap v852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:18 smithi078 ceph-mon[99959]: pgmap v852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:20.413 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:20.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:20.413 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:20.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:20.413 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:20.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:20 smithi146 ceph-mon[103278]: pgmap v853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:20 smithi078 ceph-mon[91136]: pgmap v853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:20 smithi078 ceph-mon[99959]: pgmap v853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:48:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T20:48:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:21.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:22.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:22 smithi146 ceph-mon[103278]: pgmap v854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:22 smithi078 ceph-mon[91136]: pgmap v854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:22 smithi078 ceph-mon[99959]: pgmap v854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:23.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:23.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:23.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:24 smithi146 ceph-mon[103278]: pgmap v855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:24 smithi078 ceph-mon[99959]: pgmap v855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:24 smithi078 ceph-mon[91136]: pgmap v855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:26 smithi146 ceph-mon[103278]: pgmap v856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:26 smithi078 ceph-mon[91136]: pgmap v856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:26 smithi078 ceph-mon[99959]: pgmap v856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:28 smithi146 ceph-mon[103278]: pgmap v857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:28 smithi078 ceph-mon[91136]: pgmap v857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:28 smithi078 ceph-mon[99959]: pgmap v857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:30.190Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:30.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:30.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:30 smithi146 ceph-mon[103278]: pgmap v858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:30 smithi078 ceph-mon[91136]: pgmap v858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:30 smithi078 ceph-mon[99959]: pgmap v858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:48:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:48:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:32 smithi146 ceph-mon[103278]: pgmap v859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:32 smithi078 ceph-mon[99959]: pgmap v859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:32 smithi078 ceph-mon[91136]: pgmap v859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:33.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:33.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:33.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:34 smithi146 ceph-mon[103278]: pgmap v860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:34 smithi078 ceph-mon[91136]: pgmap v860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:34 smithi078 ceph-mon[99959]: pgmap v860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:36.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:36 smithi078 ceph-mon[91136]: pgmap v861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:36.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:36 smithi078 ceph-mon[99959]: pgmap v861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:36 smithi146 ceph-mon[103278]: pgmap v861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:38.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:39.238 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:38 smithi146 ceph-mon[103278]: pgmap v862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:38 smithi078 ceph-mon[91136]: pgmap v862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:38 smithi078 ceph-mon[99959]: pgmap v862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:40.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:40.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:40.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:40.891 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:48:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:48:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:40 smithi146 ceph-mon[103278]: pgmap v863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:40 smithi078 ceph-mon[91136]: pgmap v863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:40 smithi078 ceph-mon[99959]: pgmap v863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:42.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:42 smithi146 ceph-mon[103278]: pgmap v864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:42 smithi078 ceph-mon[91136]: pgmap v864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:42 smithi078 ceph-mon[99959]: pgmap v864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:43.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:43.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:43.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:43.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:43.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:43.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:44.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:44.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:44 smithi146 ceph-mon[103278]: pgmap v865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:44 smithi078 ceph-mon[91136]: pgmap v865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:44 smithi078 ceph-mon[99959]: pgmap v865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:46.784 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:48:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:46 smithi078 ceph-mon[91136]: pgmap v866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:46 smithi078 ceph-mon[99959]: pgmap v866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:46 smithi146 ceph-mon[103278]: pgmap v866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 39m - - 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 39m - - 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (33m) 10s ago 34m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (34m) 5s ago 34m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (26m) 10s ago 32m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (25m) 5s ago 38m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (27m) 10s ago 40m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (25m) 10s ago 40m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (24m) 5s ago 39m 71.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (24m) 10s ago 38m 77.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (34m) 10s ago 34m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:48:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (34m) 5s ago 34m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (23m) 10s ago 38m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (22m) 10s ago 37m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (22m) 10s ago 37m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (21m) 10s ago 37m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (20m) 5s ago 36m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (19m) 5s ago 36m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (19m) 5s ago 35m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (18m) 5s ago 35m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (27m) 5s ago 34m 67.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (17m) 10s ago 32m 85.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (17m) 5s ago 32m 86.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (17m) 10s ago 32m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:48:47.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (17m) 5s ago 32m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:48:47.623 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:48:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:48:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:48:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:48:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:48:47.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:48:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:48:47.899 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:48:47.899 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:48:47.899 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:48:47.899 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:48:48.208 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:48:48.208 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:48.208 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:48:48.208 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:48.208 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:48:48.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:47 smithi078 ceph-mon[99959]: from='client.45990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:48.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:47 smithi078 ceph-mon[99959]: from='client.45996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:48.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3721262223' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:48.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:47 smithi078 ceph-mon[91136]: from='client.45990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:48.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:47 smithi078 ceph-mon[91136]: from='client.45996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:48.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3721262223' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:47 smithi146 ceph-mon[103278]: from='client.45990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:47 smithi146 ceph-mon[103278]: from='client.45996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3721262223' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:48:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:48 smithi146 ceph-mon[103278]: from='client.65921 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:48 smithi146 ceph-mon[103278]: pgmap v867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:48 smithi146 ceph-mon[103278]: from='client.65933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4183220939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[99959]: from='client.65921 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[99959]: pgmap v867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[99959]: from='client.65933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4183220939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[91136]: from='client.65921 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[91136]: pgmap v867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[91136]: from='client.65933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:48:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4183220939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:48:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:50.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:50.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:50.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:50.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:48:50] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:48:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:50 smithi146 ceph-mon[103278]: pgmap v868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:50 smithi078 ceph-mon[91136]: pgmap v868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:50 smithi078 ceph-mon[99959]: pgmap v868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:48:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:52 smithi146 ceph-mon[103278]: pgmap v869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:52 smithi078 ceph-mon[91136]: pgmap v869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:52 smithi078 ceph-mon[99959]: pgmap v869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:53.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:53.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:48:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:48:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:48:53.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:48:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:54 smithi146 ceph-mon[103278]: pgmap v870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:54 smithi078 ceph-mon[99959]: pgmap v870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:54 smithi078 ceph-mon[91136]: pgmap v870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:56 smithi146 ceph-mon[103278]: pgmap v871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:56 smithi078 ceph-mon[91136]: pgmap v871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:56 smithi078 ceph-mon[99959]: pgmap v871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:48:59.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:59 smithi146 ceph-mon[103278]: pgmap v872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:59 smithi078 ceph-mon[91136]: pgmap v872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:48:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:59 smithi078 ceph-mon[99959]: pgmap v872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:48:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:00.192Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:00.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:00.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:01.020 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:49:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:49:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:01 smithi078 ceph-mon[91136]: pgmap v873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:01 smithi078 ceph-mon[99959]: pgmap v873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:01.503 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:01 smithi146 ceph-mon[103278]: pgmap v873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:02.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:03 smithi078 ceph-mon[91136]: pgmap v874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:03 smithi078 ceph-mon[99959]: pgmap v874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:03 smithi146 ceph-mon[103278]: pgmap v874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:03.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:03.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:03.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:05.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:05 smithi078 ceph-mon[91136]: pgmap v875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:05.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:05 smithi078 ceph-mon[99959]: pgmap v875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:05 smithi146 ceph-mon[103278]: pgmap v875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:07 smithi078 ceph-mon[91136]: pgmap v876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:07 smithi078 ceph-mon[99959]: pgmap v876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:07 smithi146 ceph-mon[103278]: pgmap v876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:09 smithi146 ceph-mon[103278]: pgmap v877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:09 smithi078 ceph-mon[91136]: pgmap v877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:09 smithi078 ceph-mon[99959]: pgmap v877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:10.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:10.192Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:10.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:10.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:10.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:10.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:49:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:49:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:11 smithi146 ceph-mon[103278]: pgmap v878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:11 smithi078 ceph-mon[99959]: pgmap v878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:11 smithi078 ceph-mon[91136]: pgmap v878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:12.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:13 smithi146 ceph-mon[103278]: pgmap v879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:13 smithi078 ceph-mon[91136]: pgmap v879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:13 smithi078 ceph-mon[99959]: pgmap v879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:13.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:13.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:13.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:15 smithi146 ceph-mon[103278]: pgmap v880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:15 smithi078 ceph-mon[91136]: pgmap v880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:15 smithi078 ceph-mon[99959]: pgmap v880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:17 smithi146 ceph-mon[103278]: pgmap v881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:17 smithi078 ceph-mon[91136]: pgmap v881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:17 smithi078 ceph-mon[99959]: pgmap v881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:18.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:18 smithi146 ceph-mon[103278]: pgmap v882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[99959]: pgmap v882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:18.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:18 smithi078 ceph-mon[91136]: pgmap v882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 39m - - 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 39m - - 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (34m) 1s ago 35m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (34m) 17s ago 34m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (27m) 1s ago 32m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:49:19.029 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (26m) 17s ago 39m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (28m) 1s ago 40m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (26m) 1s ago 40m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (25m) 17s ago 39m 72.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (25m) 1s ago 39m 78.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (34m) 1s ago 34m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (34m) 17s ago 34m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (24m) 1s ago 38m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (23m) 1s ago 38m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (22m) 1s ago 38m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (22m) 1s ago 37m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (21m) 17s ago 37m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (20m) 17s ago 36m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:49:19.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (19m) 17s ago 36m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:49:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (18m) 17s ago 36m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:49:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (27m) 17s ago 35m 67.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:49:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (18m) 1s ago 33m 86.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:49:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (18m) 17s ago 33m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:49:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (18m) 1s ago 33m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:49:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (18m) 17s ago 33m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:49:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:19 smithi146 ceph-mon[103278]: from='client.65945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:19 smithi146 ceph-mon[103278]: from='client.65951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:19 smithi146 ceph-mon[103278]: from='client.65957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:19 smithi078 ceph-mon[91136]: from='client.65945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:19 smithi078 ceph-mon[91136]: from='client.65951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:19 smithi078 ceph-mon[91136]: from='client.65957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:19 smithi078 ceph-mon[99959]: from='client.65945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:19 smithi078 ceph-mon[99959]: from='client.65951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:19 smithi078 ceph-mon[99959]: from='client.65957 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:19.338 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:19.339 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:49:19.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:49:19.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:49:19.340 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:49:19.608 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:49:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:49:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:49:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:49:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:49:19.609 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:49:19.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:49:19.915 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:49:19.915 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:19.915 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:19.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:20.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3570475363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:20.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[91136]: pgmap v883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:20.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[91136]: from='client.65969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:20.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/782997325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:20.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3570475363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:20.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[99959]: pgmap v883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:20.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[99959]: from='client.65969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:20.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/782997325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:20.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3570475363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:20.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:20 smithi146 ceph-mon[103278]: pgmap v883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:20.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:20 smithi146 ceph-mon[103278]: from='client.65969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:20.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/782997325' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:20.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:20.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:20.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:20.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:49:20] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:49:21.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:22 smithi146 ceph-mon[103278]: pgmap v884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[99959]: pgmap v884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[91136]: pgmap v884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:23.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:23.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:23.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:24.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:24 smithi078 ceph-mon[91136]: pgmap v885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:24.875 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:24 smithi078 ceph-mon[99959]: pgmap v885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:24 smithi146 ceph-mon[103278]: pgmap v885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:26 smithi146 ceph-mon[103278]: pgmap v886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:26 smithi078 ceph-mon[91136]: pgmap v886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:26 smithi078 ceph-mon[99959]: pgmap v886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:28 smithi146 ceph-mon[103278]: pgmap v887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:28 smithi078 ceph-mon[91136]: pgmap v887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:28 smithi078 ceph-mon[99959]: pgmap v887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:30.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:30.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:30.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:30 smithi146 ceph-mon[103278]: pgmap v888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:30 smithi078 ceph-mon[91136]: pgmap v888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:30 smithi078 ceph-mon[99959]: pgmap v888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:49:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:49:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:32 smithi146 ceph-mon[103278]: pgmap v889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:32 smithi078 ceph-mon[91136]: pgmap v889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:32 smithi078 ceph-mon[99959]: pgmap v889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:33.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:33.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:33.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:34 smithi146 ceph-mon[103278]: pgmap v890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:34 smithi078 ceph-mon[91136]: pgmap v890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:34 smithi078 ceph-mon[99959]: pgmap v890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:36 smithi146 ceph-mon[103278]: pgmap v891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:36 smithi078 ceph-mon[99959]: pgmap v891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:36 smithi078 ceph-mon[91136]: pgmap v891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:38 smithi146 ceph-mon[103278]: pgmap v892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[99959]: pgmap v892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:38 smithi078 ceph-mon[91136]: pgmap v892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:40.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:40.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:40.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:40.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:40.440 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:40.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:40 smithi146 ceph-mon[103278]: pgmap v893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:40 smithi078 ceph-mon[91136]: pgmap v893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:40 smithi078 ceph-mon[99959]: pgmap v893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:49:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:49:41.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:42 smithi146 ceph-mon[103278]: pgmap v894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[91136]: pgmap v894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[99959]: pgmap v894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:43.638Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:43.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:43.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:44 smithi146 ceph-mon[103278]: pgmap v895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:44 smithi078 ceph-mon[91136]: pgmap v895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:44 smithi078 ceph-mon[99959]: pgmap v895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:46 smithi146 ceph-mon[103278]: pgmap v896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:46 smithi078 ceph-mon[99959]: pgmap v896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:46 smithi078 ceph-mon[91136]: pgmap v896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:48 smithi146 ceph-mon[103278]: pgmap v897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:48 smithi078 ceph-mon[91136]: pgmap v897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:48 smithi078 ceph-mon[99959]: pgmap v897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:50.208 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:49:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:50.195Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:50.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:50.196Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 40m - - 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 40m - - 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (34m) 13s ago 35m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (35m) 8s ago 35m 88.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (27m) 13s ago 33m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (26m) 8s ago 39m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (28m) 13s ago 41m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (26m) 13s ago 41m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:49:50.738 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (25m) 8s ago 40m 73.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (26m) 13s ago 39m 78.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (35m) 13s ago 35m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (35m) 8s ago 35m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (24m) 13s ago 39m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (23m) 13s ago 38m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (23m) 13s ago 38m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (22m) 13s ago 38m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (22m) 8s ago 37m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (20m) 8s ago 37m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (20m) 8s ago 36m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (19m) 8s ago 36m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (28m) 8s ago 36m 67.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (18m) 13s ago 33m 86.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:49:50.739 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (18m) 8s ago 33m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:49:50.740 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (18m) 13s ago 33m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:49:50.740 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (18m) 8s ago 33m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:49:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:50 smithi146 ceph-mon[103278]: pgmap v898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:49:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:49:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:50 smithi078 ceph-mon[99959]: pgmap v898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:50 smithi078 ceph-mon[91136]: pgmap v898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:51.046 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:49:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:49:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:49:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:49:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:49:51.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:49:51.047 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:49:51.314 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:49:51.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:49:51.315 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:49:51.315 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:49:51.315 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:49:51.315 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:49:51.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:49:51.623 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:49:51.624 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:49:51.624 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:49:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:51 smithi146 ceph-mon[103278]: from='client.65981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:51 smithi146 ceph-mon[103278]: from='client.46068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:51 smithi146 ceph-mon[103278]: from='client.65993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2103718795' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4212209159' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[99959]: from='client.65981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[99959]: from='client.46068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[99959]: from='client.65993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2103718795' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4212209159' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[91136]: from='client.65981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[91136]: from='client.46068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[91136]: from='client.65993 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2103718795' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:49:52.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4212209159' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:49:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:52 smithi146 ceph-mon[103278]: from='client.46086 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:52 smithi146 ceph-mon[103278]: pgmap v899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:52 smithi078 ceph-mon[99959]: from='client.46086 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:52 smithi078 ceph-mon[99959]: pgmap v899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:52 smithi078 ceph-mon[91136]: from='client.46086 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:49:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:52 smithi078 ceph-mon[91136]: pgmap v899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:53.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:53.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:49:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:49:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:49:53.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:49:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:54 smithi078 ceph-mon[99959]: pgmap v900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:54 smithi078 ceph-mon[91136]: pgmap v900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:54 smithi146 ceph-mon[103278]: pgmap v900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:49:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:56 smithi078 ceph-mon[99959]: pgmap v901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:56 smithi078 ceph-mon[91136]: pgmap v901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:56 smithi146 ceph-mon[103278]: pgmap v901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:58 smithi146 ceph-mon[103278]: pgmap v902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[91136]: pgmap v902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[99959]: pgmap v902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:49:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:49:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:00.196Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:00.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:00.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:00 smithi146 ceph-mon[103278]: pgmap v903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:50:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:50:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-mon[99959]: pgmap v903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:00.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-mon[91136]: pgmap v903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:00.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:01.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:02 smithi146 ceph-mon[103278]: pgmap v904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[91136]: pgmap v904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[99959]: pgmap v904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:03.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:03.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:03.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:04 smithi146 ceph-mon[103278]: pgmap v905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:04 smithi078 ceph-mon[91136]: pgmap v905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:04 smithi078 ceph-mon[99959]: pgmap v905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:07.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:06 smithi078 ceph-mon[91136]: pgmap v906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:07.211 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:06 smithi078 ceph-mon[99959]: pgmap v906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:06 smithi146 ceph-mon[103278]: pgmap v906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:08 smithi146 ceph-mon[103278]: pgmap v907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:08 smithi078 ceph-mon[91136]: pgmap v907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:08 smithi078 ceph-mon[99959]: pgmap v907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:10.197Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:10.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:10.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:10.873 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:50:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:50:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:10 smithi146 ceph-mon[103278]: pgmap v908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:10 smithi078 ceph-mon[99959]: pgmap v908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:10 smithi078 ceph-mon[91136]: pgmap v908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:12.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:13.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:12 smithi146 ceph-mon[103278]: pgmap v909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:12 smithi078 ceph-mon[91136]: pgmap v909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:12 smithi078 ceph-mon[99959]: pgmap v909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:13.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:13.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:13.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:14 smithi146 ceph-mon[103278]: pgmap v910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:14 smithi078 ceph-mon[91136]: pgmap v910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:14 smithi078 ceph-mon[99959]: pgmap v910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:16 smithi146 ceph-mon[103278]: pgmap v911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:16 smithi078 ceph-mon[91136]: pgmap v911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:16 smithi078 ceph-mon[99959]: pgmap v911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:18 smithi146 ceph-mon[103278]: pgmap v912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[91136]: pgmap v912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[99959]: pgmap v912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:20.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:20.198Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:20.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:20.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:20.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:20.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:20 smithi146 ceph-mon[103278]: pgmap v913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:20 smithi078 ceph-mon[91136]: pgmap v913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:20 smithi078 ceph-mon[99959]: pgmap v913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:50:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:50:21.552 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:21.926 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:50:22.458 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:50:22.458 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 41m - - 2024-09-20T20:50:22.458 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 40m - - 2024-09-20T20:50:22.458 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (35m) 4s ago 36m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (35m) 0s ago 35m 88.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (28m) 4s ago 33m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (27m) 0s ago 40m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (29m) 4s ago 41m 538M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (27m) 4s ago 42m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (26m) 0s ago 40m 75.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (26m) 4s ago 40m 80.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (35m) 4s ago 35m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (35m) 0s ago 35m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (25m) 4s ago 39m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (24m) 4s ago 39m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:50:22.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (23m) 4s ago 39m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (23m) 4s ago 38m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (22m) 0s ago 38m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (21m) 0s ago 37m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (20m) 0s ago 37m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (19m) 0s ago 37m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (28m) 0s ago 36m 67.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (19m) 4s ago 34m 86.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (19m) 0s ago 34m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (19m) 4s ago 34m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:50:22.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (19m) 0s ago 34m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:50:22.768 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:22.769 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:50:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:50:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:50:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:50:22.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:50:22.770 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[99959]: pgmap v914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[99959]: from='client.66017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2032847791' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[91136]: pgmap v914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[91136]: from='client.66017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2032847791' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:23.037 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:50:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:50:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:50:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:50:23.037 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:50:23.038 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:50:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:22 smithi146 ceph-mon[103278]: pgmap v914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:22 smithi146 ceph-mon[103278]: from='client.66017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2032847791' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:23.349 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:23.349 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:23.349 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:23.349 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:23.997 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:23.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:23.998 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:23.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:23.998 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:23.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:23 smithi146 ceph-mon[103278]: from='client.66023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:23 smithi146 ceph-mon[103278]: from='client.46110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:23 smithi146 ceph-mon[103278]: from='client.66041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1940202145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[99959]: from='client.66023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[99959]: from='client.46110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[99959]: from='client.66041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1940202145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[91136]: from='client.66023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[91136]: from='client.46110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[91136]: from='client.66041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1940202145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:24 smithi146 ceph-mon[103278]: pgmap v915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:24 smithi078 ceph-mon[91136]: pgmap v915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:25 smithi078 ceph-mon[99959]: pgmap v915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:27.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:27 smithi078 ceph-mon[99959]: pgmap v916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:27.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:27 smithi078 ceph-mon[91136]: pgmap v916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:27 smithi146 ceph-mon[103278]: pgmap v916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:29 smithi146 ceph-mon[103278]: pgmap v917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:29 smithi078 ceph-mon[91136]: pgmap v917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:29 smithi078 ceph-mon[99959]: pgmap v917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:30.538 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:30.199Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:30.539 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:30.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:30.539 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:30.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:31.010 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:50:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:50:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:31 smithi146 ceph-mon[103278]: pgmap v918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:31 smithi078 ceph-mon[91136]: pgmap v918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:31 smithi078 ceph-mon[99959]: pgmap v918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:33 smithi078 ceph-mon[91136]: pgmap v919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:33 smithi078 ceph-mon[99959]: pgmap v919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:33 smithi146 ceph-mon[103278]: pgmap v919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:33.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:33.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:33.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:35 smithi078 ceph-mon[99959]: pgmap v920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:35 smithi078 ceph-mon[91136]: pgmap v920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:35 smithi146 ceph-mon[103278]: pgmap v920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:37 smithi078 ceph-mon[99959]: pgmap v921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:37 smithi078 ceph-mon[91136]: pgmap v921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:37 smithi146 ceph-mon[103278]: pgmap v921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:38.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[91136]: pgmap v922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:38.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:38.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:38.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:38.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[99959]: pgmap v922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:38.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:38.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:38.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:38 smithi146 ceph-mon[103278]: pgmap v922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:40.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:40.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:40.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:40.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:40 smithi146 ceph-mon[103278]: pgmap v923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:40 smithi078 ceph-mon[99959]: pgmap v923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:40 smithi078 ceph-mon[91136]: pgmap v923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:50:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:50:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:41.854 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:42 smithi146 ceph-mon[103278]: pgmap v924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[99959]: pgmap v924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:43.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[91136]: pgmap v924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:43.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:43.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:43.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:43.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:43.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:43.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:44 smithi146 ceph-mon[103278]: pgmap v925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:44 smithi078 ceph-mon[91136]: pgmap v925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:44 smithi078 ceph-mon[99959]: pgmap v925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:46 smithi146 ceph-mon[103278]: pgmap v926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:46 smithi078 ceph-mon[99959]: pgmap v926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:46 smithi078 ceph-mon[91136]: pgmap v926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:48 smithi078 ceph-mon[91136]: pgmap v927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:48 smithi078 ceph-mon[99959]: pgmap v927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:48 smithi146 ceph-mon[103278]: pgmap v927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:50.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:50.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:50.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:50 smithi146 ceph-mon[103278]: pgmap v928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:50 smithi078 ceph-mon[91136]: pgmap v928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:50 smithi078 ceph-mon[99959]: pgmap v928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:50:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:50:50] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:50:51.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:50:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:52 smithi146 ceph-mon[103278]: pgmap v929: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:52 smithi078 ceph-mon[99959]: pgmap v929: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:52 smithi078 ceph-mon[91136]: pgmap v929: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:50:53.635 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:50:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:53.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:53.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:50:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:50:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:50:53.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 41m - - 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 41m - - 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (35m) 16s ago 36m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (36m) 12s ago 36m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (29m) 16s ago 34m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (27m) 12s ago 40m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (29m) 16s ago 42m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:50:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (28m) 16s ago 42m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (26m) 12s ago 41m 75.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (27m) 16s ago 40m 80.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (36m) 16s ago 36m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (36m) 12s ago 36m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (25m) 16s ago 40m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (24m) 16s ago 40m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (24m) 16s ago 39m 57.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (23m) 16s ago 39m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (23m) 12s ago 38m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (22m) 12s ago 38m 55.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (21m) 12s ago 38m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (20m) 12s ago 37m 53.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (29m) 12s ago 37m 67.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:50:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (20m) 16s ago 34m 86.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:50:54.162 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (19m) 12s ago 34m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:50:54.162 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (19m) 16s ago 34m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:50:54.162 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (19m) 12s ago 34m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:50:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:50:54.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:50:54.742 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:50:54.742 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:50:54.742 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:50:54.742 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:50:54.742 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:50:54.742 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:50:54.743 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:50:54.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[99959]: pgmap v930: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:50:54.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[99959]: from='client.66053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:54.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[99959]: from='client.46140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:54.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/522674155' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:54.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[91136]: pgmap v930: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:50:54.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[91136]: from='client.66053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:54.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[91136]: from='client.46140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:54.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/522674155' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:54 smithi146 ceph-mon[103278]: pgmap v930: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:50:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:54 smithi146 ceph-mon[103278]: from='client.66053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:54 smithi146 ceph-mon[103278]: from='client.46140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/522674155' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:50:55.047 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:50:55.047 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:55.047 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:50:55.047 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:55.047 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:50:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:55 smithi146 ceph-mon[103278]: from='client.46146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:55 smithi146 ceph-mon[103278]: from='client.66077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/452680919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:55 smithi078 ceph-mon[91136]: from='client.46146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:55 smithi078 ceph-mon[91136]: from='client.66077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/452680919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:55 smithi078 ceph-mon[99959]: from='client.46146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:55 smithi078 ceph-mon[99959]: from='client.66077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:50:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/452680919' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:50:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:56 smithi146 ceph-mon[103278]: pgmap v931: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T20:50:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:56 smithi078 ceph-mon[91136]: pgmap v931: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T20:50:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:56 smithi078 ceph-mon[99959]: pgmap v931: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 50 op/s 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[91136]: pgmap v932: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[99959]: pgmap v932: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:50:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:58 smithi146 ceph-mon[103278]: pgmap v932: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:50:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:50:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:00.201Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:00.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:00.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:00.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:00 smithi146 ceph-mon[103278]: pgmap v933: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:51:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:00 smithi078 ceph-mon[91136]: pgmap v933: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:51:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:00 smithi078 ceph-mon[99959]: pgmap v933: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T20:51:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:51:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:51:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:02 smithi146 ceph-mon[103278]: pgmap v934: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:51:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[91136]: pgmap v934: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:51:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[99959]: pgmap v934: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T20:51:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:03.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:03.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:03.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:03.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:04 smithi146 ceph-mon[103278]: pgmap v935: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:51:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:04 smithi078 ceph-mon[91136]: pgmap v935: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:51:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:04 smithi078 ceph-mon[99959]: pgmap v935: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-20T20:51:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:06 smithi146 ceph-mon[103278]: pgmap v936: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T20:51:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:06 smithi078 ceph-mon[91136]: pgmap v936: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T20:51:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:06 smithi078 ceph-mon[99959]: pgmap v936: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-20T20:51:08.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:08 smithi078 ceph-mon[99959]: pgmap v937: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:51:08.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:08 smithi078 ceph-mon[91136]: pgmap v937: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:51:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:08 smithi146 ceph-mon[103278]: pgmap v937: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T20:51:10.482 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:10.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:10.482 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:10.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:10.482 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:10.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:51:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:51:11.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:10 smithi078 ceph-mon[99959]: pgmap v938: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:51:11.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:10 smithi078 ceph-mon[91136]: pgmap v938: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:51:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:10 smithi146 ceph-mon[103278]: pgmap v938: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:51:12.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:12 smithi146 ceph-mon[103278]: pgmap v939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T20:51:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:12 smithi078 ceph-mon[91136]: pgmap v939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T20:51:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:12 smithi078 ceph-mon[99959]: pgmap v939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T20:51:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:13.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:13.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:13.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:14 smithi146 ceph-mon[103278]: pgmap v940: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:51:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:14 smithi078 ceph-mon[91136]: pgmap v940: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:51:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:14 smithi078 ceph-mon[99959]: pgmap v940: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T20:51:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:16 smithi146 ceph-mon[103278]: pgmap v941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T20:51:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:16 smithi078 ceph-mon[91136]: pgmap v941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T20:51:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:16 smithi078 ceph-mon[99959]: pgmap v941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-20T20:51:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:18 smithi146 ceph-mon[103278]: pgmap v942: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-20T20:51:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[91136]: pgmap v942: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-20T20:51:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[99959]: pgmap v942: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-20T20:51:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:20.203Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:20.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:20.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:20.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:51:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:51:21.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:20 smithi146 ceph-mon[103278]: pgmap v943: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:51:21.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:20 smithi078 ceph-mon[99959]: pgmap v943: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:51:21.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:20 smithi078 ceph-mon[91136]: pgmap v943: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:51:21.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:23.098 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:22 smithi146 ceph-mon[103278]: pgmap v944: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:51:23.098 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.099 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.099 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[91136]: pgmap v944: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:51:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[99959]: pgmap v944: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T20:51:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:23.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:23.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:23.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:24 smithi146 ceph-mon[103278]: pgmap v945: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:51:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:24 smithi078 ceph-mon[91136]: pgmap v945: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:51:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:24 smithi078 ceph-mon[99959]: pgmap v945: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T20:51:25.344 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:51:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:51:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 42m - - 2024-09-20T20:51:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 41m - - 2024-09-20T20:51:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (36m) 7s ago 37m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:51:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (37m) 3s ago 37m 87.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:51:25.872 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (29m) 7s ago 34m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (28m) 3s ago 41m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (30m) 7s ago 42m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (28m) 7s ago 43m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (27m) 3s ago 41m 80.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (27m) 7s ago 41m 78.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (36m) 7s ago 36m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (36m) 3s ago 36m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (26m) 7s ago 40m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (25m) 7s ago 40m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (24m) 7s ago 40m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:51:25.873 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (24m) 7s ago 39m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (23m) 3s ago 39m 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (22m) 3s ago 38m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (21m) 3s ago 38m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (20m) 3s ago 38m 56.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (29m) 3s ago 37m 67.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (20m) 7s ago 35m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (20m) 3s ago 35m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (20m) 7s ago 35m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:51:25.874 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (20m) 3s ago 35m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:51:26.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:51:26.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:26.181 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:51:26.181 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:51:26.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:51:26.181 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:51:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:51:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:51:26.757 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:51:26.757 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:26.758 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:26.758 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[91136]: from='client.66089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[91136]: pgmap v946: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[91136]: from='client.46176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[91136]: from='client.66101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1183295832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1944603211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[99959]: from='client.66089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[99959]: pgmap v946: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[99959]: from='client.46176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[99959]: from='client.66101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1183295832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1944603211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:26 smithi146 ceph-mon[103278]: from='client.66089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:26 smithi146 ceph-mon[103278]: pgmap v946: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:26 smithi146 ceph-mon[103278]: from='client.46176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:26 smithi146 ceph-mon[103278]: from='client.66101 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1183295832' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1944603211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:27 smithi146 ceph-mon[103278]: from='client.66113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:27 smithi078 ceph-mon[91136]: from='client.66113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:27 smithi078 ceph-mon[99959]: from='client.66113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:28 smithi146 ceph-mon[103278]: pgmap v947: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:28 smithi078 ceph-mon[91136]: pgmap v947: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:28 smithi078 ceph-mon[99959]: pgmap v947: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:30.203Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:30.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:30.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:30.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:51:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:51:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:30 smithi146 ceph-mon[103278]: pgmap v948: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:30 smithi078 ceph-mon[91136]: pgmap v948: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:30 smithi078 ceph-mon[99959]: pgmap v948: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:32 smithi146 ceph-mon[103278]: pgmap v949: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:32 smithi078 ceph-mon[91136]: pgmap v949: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:32 smithi078 ceph-mon[99959]: pgmap v949: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:33.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:33.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:33.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:34 smithi146 ceph-mon[103278]: pgmap v950: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:34 smithi078 ceph-mon[91136]: pgmap v950: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:34 smithi078 ceph-mon[99959]: pgmap v950: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:36 smithi146 ceph-mon[103278]: pgmap v951: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:36 smithi078 ceph-mon[91136]: pgmap v951: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:36 smithi078 ceph-mon[99959]: pgmap v951: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[91136]: pgmap v952: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[99959]: pgmap v952: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:39 smithi146 ceph-mon[103278]: pgmap v952: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:40.204Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:40.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:40.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:40.985 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:51:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:51:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:41 smithi078 ceph-mon[91136]: pgmap v953: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:41 smithi078 ceph-mon[99959]: pgmap v953: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:41.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:41 smithi146 ceph-mon[103278]: pgmap v953: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:42.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[99959]: pgmap v954: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[91136]: pgmap v954: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:43 smithi146 ceph-mon[103278]: pgmap v954: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:43.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:43.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:43.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:43.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:43.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:45 smithi146 ceph-mon[103278]: pgmap v955: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:45 smithi078 ceph-mon[91136]: pgmap v955: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:45 smithi078 ceph-mon[99959]: pgmap v955: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:47 smithi146 ceph-mon[103278]: pgmap v956: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:47 smithi078 ceph-mon[91136]: pgmap v956: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:47 smithi078 ceph-mon[99959]: pgmap v956: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:49 smithi078 ceph-mon[99959]: pgmap v957: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:49 smithi078 ceph-mon[91136]: pgmap v957: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:49 smithi146 ceph-mon[103278]: pgmap v957: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:50.265 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:50.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:50.265 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:50.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:50.265 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:50.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:51:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:51:51.349 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:51 smithi078 ceph-mon[99959]: pgmap v958: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:51.349 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:51 smithi078 ceph-mon[91136]: pgmap v958: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:51 smithi146 ceph-mon[103278]: pgmap v958: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:52.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:51:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:53 smithi146 ceph-mon[103278]: pgmap v959: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:53 smithi078 ceph-mon[99959]: pgmap v959: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:53 smithi078 ceph-mon[91136]: pgmap v959: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:53.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:53.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:51:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:51:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:51:53.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:51:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:55 smithi146 ceph-mon[103278]: pgmap v960: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:55 smithi078 ceph-mon[91136]: pgmap v960: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:55 smithi078 ceph-mon[99959]: pgmap v960: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:51:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:51:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:57 smithi146 ceph-mon[103278]: pgmap v961: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:57 smithi078 ceph-mon[99959]: pgmap v961: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:57 smithi078 ceph-mon[91136]: pgmap v961: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:57.583 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:51:57.583 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 42m - - 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 42m - - 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (36m) 19s ago 37m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (37m) 15s ago 37m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (30m) 19s ago 35m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (28m) 15s ago 41m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (30m) 19s ago 43m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (29m) 19s ago 43m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (27m) 15s ago 42m 74.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (28m) 19s ago 42m 79.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (37m) 19s ago 37m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (37m) 15s ago 37m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (26m) 19s ago 41m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (25m) 19s ago 41m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:51:57.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (25m) 19s ago 40m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (24m) 19s ago 40m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (24m) 15s ago 39m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (23m) 15s ago 39m 57.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (22m) 15s ago 39m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (21m) 15s ago 38m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (30m) 15s ago 38m 67.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (21m) 19s ago 35m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (20m) 15s ago 35m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (21m) 19s ago 35m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:51:57.585 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (20m) 15s ago 35m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:51:57.892 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:51:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:51:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:51:57.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:57.893 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:51:57.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:51:57.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:57.894 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:51:57.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:51:57.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:57.894 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:51:57.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:51:57.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:51:57.895 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:51:57.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:51:57.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:51:57.895 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:51:58.171 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:51:58.172 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:51:58.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:58 smithi078 ceph-mon[99959]: from='client.66125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:58.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/708745094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:58.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:58 smithi078 ceph-mon[91136]: from='client.66125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:58.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/708745094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:58.477 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:51:58.477 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:51:58.478 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:51:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:58 smithi146 ceph-mon[103278]: from='client.66125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/708745094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:51:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='client.66131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: pgmap v962: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='client.46218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:51:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4256938079' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='client.66131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: pgmap v962: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='client.46218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4256938079' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='client.66131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: pgmap v962: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:51:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='client.46218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:51:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:51:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:51:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:51:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4256938079' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:00 smithi146 ceph-mon[103278]: from='client.66149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:00 smithi078 ceph-mon[91136]: from='client.66149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:00 smithi078 ceph-mon[99959]: from='client.66149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:00.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:00.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:00.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:52:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:52:01.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:01 smithi078 ceph-mon[91136]: pgmap v963: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:01.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:01 smithi078 ceph-mon[99959]: pgmap v963: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:01 smithi146 ceph-mon[103278]: pgmap v963: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:02.297 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:03.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:03 smithi146 ceph-mon[103278]: pgmap v964: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:03.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[91136]: pgmap v964: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[99959]: pgmap v964: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:03.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:03.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:03.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:05 smithi146 ceph-mon[103278]: pgmap v965: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:05 smithi078 ceph-mon[91136]: pgmap v965: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:05 smithi078 ceph-mon[99959]: pgmap v965: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:07 smithi146 ceph-mon[103278]: pgmap v966: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:07 smithi078 ceph-mon[91136]: pgmap v966: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:07 smithi078 ceph-mon[99959]: pgmap v966: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:09.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:09 smithi078 ceph-mon[91136]: pgmap v967: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:09.479 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:09 smithi078 ceph-mon[99959]: pgmap v967: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:09 smithi146 ceph-mon[103278]: pgmap v967: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:10.206Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:10.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:10.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:10.837 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:52:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:52:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:11 smithi146 ceph-mon[103278]: pgmap v968: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:11 smithi078 ceph-mon[91136]: pgmap v968: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:11 smithi078 ceph-mon[99959]: pgmap v968: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:12.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:13 smithi146 ceph-mon[103278]: pgmap v969: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:13 smithi078 ceph-mon[91136]: pgmap v969: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:13 smithi078 ceph-mon[99959]: pgmap v969: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:13.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:13.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:13.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:14 smithi146 ceph-mon[103278]: pgmap v970: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:14 smithi078 ceph-mon[91136]: pgmap v970: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:14 smithi078 ceph-mon[99959]: pgmap v970: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:16 smithi146 ceph-mon[103278]: pgmap v971: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:16 smithi078 ceph-mon[91136]: pgmap v971: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:16 smithi078 ceph-mon[99959]: pgmap v971: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:18 smithi146 ceph-mon[103278]: pgmap v972: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[91136]: pgmap v972: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[99959]: pgmap v972: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:18.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:20.500 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:20.207Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:20.500 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:20.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:20.500 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:20.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:20 smithi146 ceph-mon[103278]: pgmap v973: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:20 smithi078 ceph-mon[91136]: pgmap v973: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:20 smithi078 ceph-mon[99959]: pgmap v973: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:52:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-20T20:52:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:22.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:22 smithi146 ceph-mon[103278]: pgmap v974: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:22.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:22.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:22.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[91136]: pgmap v974: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[99959]: pgmap v974: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:23.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:23.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:23.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:24 smithi146 ceph-mon[103278]: pgmap v975: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:24 smithi078 ceph-mon[91136]: pgmap v975: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:24 smithi078 ceph-mon[99959]: pgmap v975: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:26 smithi146 ceph-mon[103278]: pgmap v976: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:26 smithi078 ceph-mon[91136]: pgmap v976: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:26 smithi078 ceph-mon[99959]: pgmap v976: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:28.775 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:52:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:28 smithi078 ceph-mon[99959]: pgmap v977: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:28 smithi078 ceph-mon[91136]: pgmap v977: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:28 smithi146 ceph-mon[103278]: pgmap v977: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 43m - - 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 42m - - 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (37m) 11s ago 38m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (38m) 6s ago 38m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (30m) 11s ago 36m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:52:29.303 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (29m) 6s ago 42m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (31m) 11s ago 43m 539M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (29m) 11s ago 44m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (28m) 6s ago 42m 75.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (28m) 11s ago 42m 80.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (37m) 11s ago 37m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (37m) 6s ago 37m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (27m) 11s ago 41m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (26m) 11s ago 41m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (25m) 11s ago 41m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (25m) 11s ago 40m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (24m) 6s ago 40m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (23m) 6s ago 40m 57.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (22m) 6s ago 39m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:52:29.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (21m) 6s ago 39m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:52:29.305 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (30m) 6s ago 38m 67.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:52:29.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (21m) 11s ago 36m 86.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:52:29.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (21m) 6s ago 36m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:52:29.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (21m) 11s ago 36m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:52:29.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (21m) 6s ago 36m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:52:29.610 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:52:29.611 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:52:29.612 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:52:29.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:52:29.612 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:52:29.878 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:52:29.879 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:52:29.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:29 smithi078 ceph-mon[99959]: from='client.46242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:29.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:29 smithi078 ceph-mon[99959]: from='client.66167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:29.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4175513478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:29 smithi078 ceph-mon[91136]: from='client.46242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:29 smithi078 ceph-mon[91136]: from='client.66167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4175513478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:30.200 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:52:30.201 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:52:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:52:30.201 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:52:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:52:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:29 smithi146 ceph-mon[103278]: from='client.46242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:29 smithi146 ceph-mon[103278]: from='client.66167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4175513478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:52:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:30.208Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:30.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:30.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[99959]: from='client.46254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[99959]: pgmap v978: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[99959]: from='client.46266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/431512102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[91136]: from='client.46254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[91136]: pgmap v978: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[91136]: from='client.46266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/431512102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:31.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:52:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:52:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:30 smithi146 ceph-mon[103278]: from='client.46254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:30 smithi146 ceph-mon[103278]: pgmap v978: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:30 smithi146 ceph-mon[103278]: from='client.46266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:52:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/431512102' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:52:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:32.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:32.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:32 smithi078 ceph-mon[91136]: pgmap v979: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:32 smithi078 ceph-mon[99959]: pgmap v979: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:32 smithi146 ceph-mon[103278]: pgmap v979: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:34.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:33.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:34.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:33.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:34.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:33.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:34 smithi146 ceph-mon[103278]: pgmap v980: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:34 smithi078 ceph-mon[91136]: pgmap v980: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:34 smithi078 ceph-mon[99959]: pgmap v980: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:36 smithi146 ceph-mon[103278]: pgmap v981: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:36 smithi078 ceph-mon[91136]: pgmap v981: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:36 smithi078 ceph-mon[99959]: pgmap v981: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:38 smithi146 ceph-mon[103278]: pgmap v982: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[91136]: pgmap v982: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[99959]: pgmap v982: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:40.209Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:40.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:40.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:40.825 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:52:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:52:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:40 smithi146 ceph-mon[103278]: pgmap v983: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:40 smithi078 ceph-mon[91136]: pgmap v983: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:40 smithi078 ceph-mon[99959]: pgmap v983: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:43.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:42 smithi146 ceph-mon[103278]: pgmap v984: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:42 smithi078 ceph-mon[91136]: pgmap v984: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:42 smithi078 ceph-mon[99959]: pgmap v984: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:43.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:43.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:43.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:45.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:44 smithi146 ceph-mon[103278]: pgmap v985: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:44 smithi078 ceph-mon[91136]: pgmap v985: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:44 smithi078 ceph-mon[99959]: pgmap v985: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:46 smithi146 ceph-mon[103278]: pgmap v986: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:46 smithi078 ceph-mon[91136]: pgmap v986: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:46 smithi078 ceph-mon[99959]: pgmap v986: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:48 smithi146 ceph-mon[103278]: pgmap v987: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:48 smithi078 ceph-mon[91136]: pgmap v987: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:48 smithi078 ceph-mon[99959]: pgmap v987: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:50.210Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:50.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:50.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:50.854 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:52:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-20T20:52:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:50 smithi146 ceph-mon[103278]: pgmap v988: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:50 smithi078 ceph-mon[91136]: pgmap v988: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:50 smithi078 ceph-mon[99959]: pgmap v988: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:52.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:52.205 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:52:53.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:52 smithi078 ceph-mon[99959]: pgmap v989: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:53.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:52 smithi078 ceph-mon[91136]: pgmap v989: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:52 smithi146 ceph-mon[103278]: pgmap v989: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:53.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:53.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:52:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:52:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:52:53.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:52:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:54 smithi146 ceph-mon[103278]: pgmap v990: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:54 smithi078 ceph-mon[91136]: pgmap v990: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:54 smithi078 ceph-mon[99959]: pgmap v990: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:52:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:56 smithi146 ceph-mon[103278]: pgmap v991: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:56 smithi078 ceph-mon[91136]: pgmap v991: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:56 smithi078 ceph-mon[99959]: pgmap v991: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:58 smithi146 ceph-mon[103278]: pgmap v992: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[91136]: pgmap v992: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[99959]: pgmap v992: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:52:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:00.494 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:00.212Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:00.494 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:00.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:00.495 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:00.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:00.495 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:53:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:53:00] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:53:01.023 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:53:01.023 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 43m - - 2024-09-20T20:53:01.023 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 43m - - 2024-09-20T20:53:01.023 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (37m) 2s ago 38m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (38m) 18s ago 38m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (31m) 2s ago 36m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (29m) 18s ago 42m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (31m) 2s ago 44m 540M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (30m) 2s ago 44m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (28m) 18s ago 43m 76.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (29m) 2s ago 43m 81.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (38m) 2s ago 38m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (38m) 18s ago 38m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (27m) 2s ago 42m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (26m) 2s ago 42m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (26m) 2s ago 41m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (25m) 2s ago 41m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:53:01.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (25m) 18s ago 40m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (24m) 18s ago 40m 57.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (23m) 18s ago 40m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (22m) 18s ago 39m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (31m) 18s ago 39m 63.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (22m) 2s ago 36m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (21m) 18s ago 37m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (22m) 2s ago 36m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:53:01.025 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (21m) 18s ago 36m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:53:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:00 smithi146 ceph-mon[103278]: pgmap v993: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:00 smithi078 ceph-mon[91136]: pgmap v993: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:00 smithi078 ceph-mon[99959]: pgmap v993: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:53:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:53:01.338 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:53:01.609 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:53:01.609 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:53:01.609 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:53:01.609 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:53:01.609 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:53:01.609 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:53:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:53:01.918 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:53:01.918 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:01.918 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:01.918 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:01 smithi146 ceph-mon[103278]: from='client.66197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:01 smithi146 ceph-mon[103278]: from='client.66203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:01 smithi146 ceph-mon[103278]: from='client.66209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1871431211' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3658543247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[91136]: from='client.66197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[91136]: from='client.66203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[91136]: from='client.66209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1871431211' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3658543247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[99959]: from='client.66197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[99959]: from='client.66203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[99959]: from='client.66209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1871431211' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3658543247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:02 smithi146 ceph-mon[103278]: pgmap v994: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:02 smithi146 ceph-mon[103278]: from='client.46302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:02 smithi078 ceph-mon[91136]: pgmap v994: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:02 smithi078 ceph-mon[91136]: from='client.46302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:02 smithi078 ceph-mon[99959]: pgmap v994: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:02 smithi078 ceph-mon[99959]: from='client.46302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:04.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:03.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:04.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:03.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:04.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:03.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:04.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:05 smithi146 ceph-mon[103278]: pgmap v995: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:05 smithi078 ceph-mon[91136]: pgmap v995: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:05 smithi078 ceph-mon[99959]: pgmap v995: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:07 smithi146 ceph-mon[103278]: pgmap v996: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:07 smithi078 ceph-mon[91136]: pgmap v996: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:07 smithi078 ceph-mon[99959]: pgmap v996: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:09 smithi146 ceph-mon[103278]: pgmap v997: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:09 smithi078 ceph-mon[91136]: pgmap v997: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:09 smithi078 ceph-mon[99959]: pgmap v997: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:10.212Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:10.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:10.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:10.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:53:10] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:53:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:11 smithi078 ceph-mon[91136]: pgmap v998: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:11 smithi078 ceph-mon[99959]: pgmap v998: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:11 smithi146 ceph-mon[103278]: pgmap v998: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:12.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:12.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:12.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:13 smithi146 ceph-mon[103278]: pgmap v999: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:13.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:13 smithi078 ceph-mon[99959]: pgmap v999: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:13.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:13 smithi078 ceph-mon[91136]: pgmap v999: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:13.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:13.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:13.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:13.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:13.948 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:13.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:15 smithi146 ceph-mon[103278]: pgmap v1000: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:15 smithi078 ceph-mon[91136]: pgmap v1000: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:15 smithi078 ceph-mon[99959]: pgmap v1000: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:17 smithi146 ceph-mon[103278]: pgmap v1001: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:17 smithi078 ceph-mon[99959]: pgmap v1001: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:17 smithi078 ceph-mon[91136]: pgmap v1001: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:19 smithi146 ceph-mon[103278]: pgmap v1002: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[91136]: pgmap v1002: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[99959]: pgmap v1002: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:20.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:20.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:20.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:53:20] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:53:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:21 smithi146 ceph-mon[103278]: pgmap v1003: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:21 smithi078 ceph-mon[91136]: pgmap v1003: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:21 smithi078 ceph-mon[99959]: pgmap v1003: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:23.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:23 smithi078 ceph-mon[99959]: pgmap v1004: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:23.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:23 smithi078 ceph-mon[91136]: pgmap v1004: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:23 smithi146 ceph-mon[103278]: pgmap v1004: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:23.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:23.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:23.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:24 smithi146 ceph-mon[103278]: pgmap v1005: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[99959]: pgmap v1005: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:24 smithi078 ceph-mon[91136]: pgmap v1005: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:26.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:26 smithi146 ceph-mon[103278]: pgmap v1006: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:26 smithi078 ceph-mon[91136]: pgmap v1006: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:26 smithi078 ceph-mon[99959]: pgmap v1006: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:28 smithi078 ceph-mon[91136]: pgmap v1007: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:28 smithi078 ceph-mon[99959]: pgmap v1007: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:28 smithi146 ceph-mon[103278]: pgmap v1007: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:30.521 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:30.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:30.521 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:30.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:30.521 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:30.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:30 smithi078 ceph-mon[91136]: pgmap v1008: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:53:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:53:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:30 smithi078 ceph-mon[99959]: pgmap v1008: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:30 smithi146 ceph-mon[103278]: pgmap v1008: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:32.222 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:53:32.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:32 smithi078 ceph-mon[99959]: pgmap v1009: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:32.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:32 smithi078 ceph-mon[91136]: pgmap v1009: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:32.757 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 44m - - 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 43m - - 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (38m) 14s ago 39m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (39m) 9s ago 39m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (31m) 14s ago 37m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (30m) 9s ago 43m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (32m) 14s ago 44m 540M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (30m) 14s ago 45m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (29m) 9s ago 43m 77.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:53:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (29m) 14s ago 43m 82.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (38m) 14s ago 38m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (38m) 9s ago 38m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (28m) 14s ago 43m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (27m) 14s ago 42m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (26m) 14s ago 42m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (26m) 14s ago 41m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (25m) 9s ago 41m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (24m) 9s ago 41m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (23m) 9s ago 40m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (22m) 9s ago 40m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (31m) 9s ago 39m 63.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (22m) 14s ago 37m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:53:32.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (22m) 9s ago 37m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:53:32.760 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (22m) 14s ago 37m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:53:32.760 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (22m) 9s ago 37m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:53:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:32 smithi146 ceph-mon[103278]: pgmap v1009: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:33.070 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:53:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:53:33.070 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:53:33.071 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:53:33.352 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:53:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:53:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:53:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:53:33.352 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:53:33.353 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[99959]: from='client.46314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[99959]: from='client.46320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[99959]: from='client.46326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3367670684' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[91136]: from='client.46314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[91136]: from='client.46320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[91136]: from='client.46326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:33.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3367670684' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:33.679 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:53:33.679 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:33.679 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:53:33.679 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:33.679 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:53:33.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:33.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:33.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:33.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:33.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:33.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:33 smithi146 ceph-mon[103278]: from='client.46314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:33 smithi146 ceph-mon[103278]: from='client.46320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:33 smithi146 ceph-mon[103278]: from='client.46326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3367670684' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:53:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:34 smithi146 ceph-mon[103278]: from='client.66257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:34 smithi146 ceph-mon[103278]: pgmap v1010: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3728961349' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:34 smithi078 ceph-mon[91136]: from='client.66257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:34 smithi078 ceph-mon[91136]: pgmap v1010: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3728961349' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:34 smithi078 ceph-mon[99959]: from='client.66257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:53:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:34 smithi078 ceph-mon[99959]: pgmap v1010: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3728961349' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:53:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:36 smithi146 ceph-mon[103278]: pgmap v1011: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:36 smithi078 ceph-mon[91136]: pgmap v1011: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:36 smithi078 ceph-mon[99959]: pgmap v1011: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:38.827 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:38 smithi078 ceph-mon[99959]: pgmap v1012: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:38.827 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:38 smithi078 ceph-mon[91136]: pgmap v1012: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:38 smithi146 ceph-mon[103278]: pgmap v1012: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:40.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:40.214Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:40.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:40.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:40 smithi078 ceph-mon[99959]: pgmap v1013: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:40 smithi078 ceph-mon[91136]: pgmap v1013: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:53:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:53:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:40 smithi146 ceph-mon[103278]: pgmap v1013: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:42 smithi146 ceph-mon[103278]: pgmap v1014: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:42 smithi078 ceph-mon[99959]: pgmap v1014: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:42 smithi078 ceph-mon[91136]: pgmap v1014: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:43.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:43.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:43.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:53:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:44 smithi146 ceph-mon[103278]: pgmap v1015: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:44 smithi078 ceph-mon[91136]: pgmap v1015: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:44 smithi078 ceph-mon[99959]: pgmap v1015: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:47.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:46 smithi146 ceph-mon[103278]: pgmap v1016: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:46 smithi078 ceph-mon[91136]: pgmap v1016: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:46 smithi078 ceph-mon[99959]: pgmap v1016: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:48 smithi146 ceph-mon[103278]: pgmap v1017: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:48 smithi078 ceph-mon[91136]: pgmap v1017: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:48 smithi078 ceph-mon[99959]: pgmap v1017: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:50.215Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:50.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:50.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:50 smithi078 ceph-mon[91136]: pgmap v1018: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:50 smithi078 ceph-mon[99959]: pgmap v1018: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:53:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:53:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:50 smithi146 ceph-mon[103278]: pgmap v1018: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:53:53.015 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:52 smithi078 ceph-mon[99959]: pgmap v1019: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:53.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:52 smithi078 ceph-mon[91136]: pgmap v1019: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:53.272 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:52 smithi146 ceph-mon[103278]: pgmap v1019: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:53.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:53.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:53.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:53.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:53:53.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:53:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:53:53.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:53:55.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:54 smithi078 ceph-mon[91136]: pgmap v1020: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:55.152 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:54 smithi078 ceph-mon[99959]: pgmap v1020: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:54 smithi146 ceph-mon[103278]: pgmap v1020: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:53:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:56 smithi146 ceph-mon[103278]: pgmap v1021: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:56 smithi078 ceph-mon[91136]: pgmap v1021: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:56 smithi078 ceph-mon[99959]: pgmap v1021: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:58 smithi146 ceph-mon[103278]: pgmap v1022: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:58 smithi078 ceph-mon[91136]: pgmap v1022: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:53:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:58 smithi078 ceph-mon[99959]: pgmap v1022: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:53:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:00.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:53:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:53:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:53:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:00.216Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:00.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:00.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:00.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:54:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:54:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:00 smithi146 ceph-mon[103278]: pgmap v1023: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:00 smithi078 ceph-mon[91136]: pgmap v1023: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:00 smithi078 ceph-mon[99959]: pgmap v1023: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:02 smithi146 ceph-mon[103278]: pgmap v1024: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:02 smithi078 ceph-mon[91136]: pgmap v1024: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:02 smithi078 ceph-mon[99959]: pgmap v1024: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:03.975 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:03.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:03.975 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:03.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:03.975 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:03.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:03.976 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:54:04.505 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:54:04.505 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 44m - - 2024-09-20T20:54:04.505 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 44m - - 2024-09-20T20:54:04.506 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (38m) 5s ago 39m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:54:04.506 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (39m) 1s ago 39m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:54:04.506 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (32m) 5s ago 37m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:54:04.506 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (31m) 1s ago 43m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:54:04.506 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (32m) 5s ago 45m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:54:04.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (31m) 5s ago 45m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:54:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (29m) 1s ago 44m 79.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:54:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (30m) 5s ago 44m 82.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:54:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (39m) 5s ago 39m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:54:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (39m) 1s ago 39m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:54:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (28m) 5s ago 43m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:54:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (27m) 5s ago 43m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:54:04.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (27m) 5s ago 42m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:54:04.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (26m) 5s ago 42m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:54:04.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (26m) 1s ago 41m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:54:04.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (25m) 1s ago 41m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:54:04.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (24m) 1s ago 41m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:54:04.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (23m) 1s ago 40m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:54:04.509 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (32m) 1s ago 40m 66.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:54:04.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (23m) 5s ago 38m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:54:04.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (23m) 1s ago 38m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:54:04.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (23m) 5s ago 37m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:54:04.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (22m) 1s ago 37m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:54:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:04 smithi146 ceph-mon[103278]: pgmap v1025: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:04 smithi146 ceph-mon[103278]: from='client.46350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[91136]: pgmap v1025: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[91136]: from='client.46350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[99959]: pgmap v1025: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:04 smithi078 ceph-mon[99959]: from='client.46350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:04.813 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:54:04.813 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:54:04.813 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:54:04.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:04.813 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:54:04.813 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:54:04.814 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:54:05.085 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:54:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:54:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:54:05.086 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:54:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:54:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:54:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:54:05.397 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:54:05.397 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:54:05.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:05 smithi146 ceph-mon[103278]: from='client.46356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:05 smithi146 ceph-mon[103278]: from='client.46362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/27759126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:05.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:05 smithi146 ceph-mon[103278]: from='client.46374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1172873549' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[91136]: from='client.46356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[91136]: from='client.46362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/27759126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[91136]: from='client.46374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1172873549' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[99959]: from='client.46356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[99959]: from='client.46362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/27759126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[99959]: from='client.46374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1172873549' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:06.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:06 smithi146 ceph-mon[103278]: pgmap v1026: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:06 smithi078 ceph-mon[91136]: pgmap v1026: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:06 smithi078 ceph-mon[99959]: pgmap v1026: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:08 smithi078 ceph-mon[91136]: pgmap v1027: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:08 smithi078 ceph-mon[99959]: pgmap v1027: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:09.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:08 smithi146 ceph-mon[103278]: pgmap v1027: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:10.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:10.217Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:10.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:10.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:10.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:10.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:10 smithi078 ceph-mon[91136]: pgmap v1028: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:10 smithi078 ceph-mon[99959]: pgmap v1028: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:54:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:54:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:10 smithi146 ceph-mon[103278]: pgmap v1028: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:12.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:12 smithi146 ceph-mon[103278]: pgmap v1029: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:12 smithi078 ceph-mon[91136]: pgmap v1029: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:12 smithi078 ceph-mon[99959]: pgmap v1029: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:14.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:14.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:14.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:13.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:14.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:13.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:14.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:13.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:14.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:14 smithi078 ceph-mon[91136]: pgmap v1030: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:14.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:14 smithi078 ceph-mon[99959]: pgmap v1030: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:14 smithi146 ceph-mon[103278]: pgmap v1030: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:17.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:16 smithi146 ceph-mon[103278]: pgmap v1031: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:16 smithi078 ceph-mon[91136]: pgmap v1031: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:16 smithi078 ceph-mon[99959]: pgmap v1031: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:18 smithi146 ceph-mon[103278]: pgmap v1032: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:18 smithi078 ceph-mon[91136]: pgmap v1032: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:18 smithi078 ceph-mon[99959]: pgmap v1032: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:20.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.447 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:20.218Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:20.447 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:20.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:20.448 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:20.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:54:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:54:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:21 smithi146 ceph-mon[103278]: pgmap v1033: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:21 smithi078 ceph-mon[91136]: pgmap v1033: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:21 smithi078 ceph-mon[99959]: pgmap v1033: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:23 smithi146 ceph-mon[103278]: pgmap v1034: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:23 smithi078 ceph-mon[99959]: pgmap v1034: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:23 smithi078 ceph-mon[91136]: pgmap v1034: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:23.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:23.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:23.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:24 smithi146 ceph-mon[103278]: pgmap v1035: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[99959]: pgmap v1035: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[91136]: pgmap v1035: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:26.115 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 20:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T20:54:25.849973314Z level=info msg="Completed cleanup jobs" duration=14.050239ms 2024-09-20T20:54:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:26 smithi146 ceph-mon[103278]: pgmap v1036: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:26 smithi078 ceph-mon[99959]: pgmap v1036: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:26 smithi078 ceph-mon[91136]: pgmap v1036: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:28 smithi146 ceph-mon[103278]: pgmap v1037: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:28 smithi078 ceph-mon[91136]: pgmap v1037: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:28 smithi078 ceph-mon[99959]: pgmap v1037: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:30.219Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:30.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:30.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:30 smithi146 ceph-mon[103278]: pgmap v1038: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:30 smithi078 ceph-mon[91136]: pgmap v1038: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:30 smithi078 ceph-mon[99959]: pgmap v1038: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:54:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:54:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:32 smithi146 ceph-mon[103278]: pgmap v1039: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:32 smithi078 ceph-mon[99959]: pgmap v1039: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:32 smithi078 ceph-mon[91136]: pgmap v1039: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:33.955 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:33.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:33.955 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:33.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:33.955 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:33.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:34.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:34 smithi078 ceph-mon[91136]: pgmap v1040: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:34.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:34 smithi078 ceph-mon[99959]: pgmap v1040: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:34 smithi146 ceph-mon[103278]: pgmap v1040: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:35.734 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:54:36.281 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 45m - - 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 44m - - 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (39m) 17s ago 40m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (40m) 12s ago 40m 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (32m) 17s ago 38m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (31m) 12s ago 44m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:54:36.282 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (33m) 17s ago 46m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (31m) 17s ago 46m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (30m) 12s ago 44m 77.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (30m) 17s ago 44m 82.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (39m) 17s ago 39m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (39m) 12s ago 39m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (29m) 17s ago 44m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (28m) 17s ago 43m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (27m) 17s ago 43m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:54:36.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (27m) 17s ago 42m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (26m) 12s ago 42m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (25m) 12s ago 42m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (24m) 12s ago 41m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (24m) 12s ago 41m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (32m) 12s ago 40m 66.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (23m) 17s ago 38m 86.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (23m) 12s ago 38m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (23m) 17s ago 38m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:54:36.284 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (23m) 12s ago 38m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:54:36.601 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:54:36.870 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:36 smithi078 ceph-mon[99959]: pgmap v1041: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:36.870 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:36 smithi078 ceph-mon[99959]: from='client.46386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:36.870 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:36 smithi078 ceph-mon[99959]: from='client.46392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:36.870 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:36 smithi078 ceph-mon[91136]: pgmap v1041: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:36.870 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:36 smithi078 ceph-mon[91136]: from='client.46386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:36.871 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:36 smithi078 ceph-mon[91136]: from='client.46392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:54:36.871 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:54:36.872 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:54:36.872 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:54:36.872 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:54:36.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:54:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:36 smithi146 ceph-mon[103278]: pgmap v1041: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:36 smithi146 ceph-mon[103278]: from='client.46386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:36 smithi146 ceph-mon[103278]: from='client.46392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:54:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:54:37.195 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:54:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:54:37.196 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:54:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:37 smithi146 ceph-mon[103278]: from='client.66317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/24309692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:37 smithi146 ceph-mon[103278]: from='client.46410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1243535599' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[91136]: from='client.66317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/24309692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[91136]: from='client.46410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1243535599' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[99959]: from='client.66317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/24309692' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[99959]: from='client.46410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:54:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1243535599' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:54:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:38 smithi146 ceph-mon[103278]: pgmap v1042: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:38 smithi078 ceph-mon[91136]: pgmap v1042: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:38 smithi078 ceph-mon[99959]: pgmap v1042: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:40.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[91136]: pgmap v1043: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:40.455 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.455 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.455 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.455 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-mon[99959]: pgmap v1043: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:40.455 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:40.220Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:40.455 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:40.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:40.455 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:40.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:40 smithi146 ceph-mon[103278]: pgmap v1043: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:54:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:54:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:42 smithi146 ceph-mon[103278]: pgmap v1044: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:42 smithi078 ceph-mon[99959]: pgmap v1044: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:42 smithi078 ceph-mon[91136]: pgmap v1044: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:43.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:43.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:43.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:44 smithi146 ceph-mon[103278]: pgmap v1045: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[91136]: pgmap v1045: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[99959]: pgmap v1045: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:54:46.875 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:46 smithi146 ceph-mon[103278]: pgmap v1046: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:46 smithi078 ceph-mon[91136]: pgmap v1046: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:46 smithi078 ceph-mon[99959]: pgmap v1046: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:48.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:48 smithi146 ceph-mon[103278]: pgmap v1047: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:48 smithi078 ceph-mon[91136]: pgmap v1047: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:48 smithi078 ceph-mon[99959]: pgmap v1047: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:50.221Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:50.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:50.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:50 smithi146 ceph-mon[103278]: pgmap v1048: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:50 smithi078 ceph-mon[91136]: pgmap v1048: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:54:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:54:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:50 smithi078 ceph-mon[99959]: pgmap v1048: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:54:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:52 smithi146 ceph-mon[103278]: pgmap v1049: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:52 smithi078 ceph-mon[91136]: pgmap v1049: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:52 smithi078 ceph-mon[99959]: pgmap v1049: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:53.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:53.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:54:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:54:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:54:53.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:54:54.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:54 smithi078 ceph-mon[91136]: pgmap v1050: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:54.866 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:54 smithi078 ceph-mon[99959]: pgmap v1050: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:54 smithi146 ceph-mon[103278]: pgmap v1050: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:54:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:56 smithi146 ceph-mon[103278]: pgmap v1051: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:56 smithi078 ceph-mon[91136]: pgmap v1051: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:56 smithi078 ceph-mon[99959]: pgmap v1051: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:58 smithi146 ceph-mon[103278]: pgmap v1052: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:54:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:58 smithi078 ceph-mon[91136]: pgmap v1052: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:54:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:54:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:58 smithi078 ceph-mon[99959]: pgmap v1052: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:54:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:54:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[91136]: pgmap v1053: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-mon[99959]: pgmap v1053: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:00.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:00.222Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:00.538 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:00.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:00.538 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:00.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:00 smithi146 ceph-mon[103278]: pgmap v1053: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:55:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:55:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:02 smithi146 ceph-mon[103278]: pgmap v1054: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:02 smithi078 ceph-mon[91136]: pgmap v1054: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:02 smithi078 ceph-mon[99959]: pgmap v1054: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:03.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:03.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:03.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:04 smithi146 ceph-mon[103278]: pgmap v1055: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[91136]: pgmap v1055: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[99959]: pgmap v1055: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:07.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:06 smithi146 ceph-mon[103278]: pgmap v1056: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:06 smithi078 ceph-mon[91136]: pgmap v1056: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:06 smithi078 ceph-mon[99959]: pgmap v1056: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:07.491 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:55:08.020 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:55:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 45m - - 2024-09-20T20:55:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 45m - - 2024-09-20T20:55:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (39m) 8s ago 40m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:55:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (40m) 4s ago 40m 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:55:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (33m) 8s ago 38m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:55:08.022 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (32m) 4s ago 44m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:55:08.022 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (33m) 8s ago 46m 541M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:55:08.022 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (32m) 8s ago 46m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:55:08.022 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (30m) 4s ago 45m 78.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:55:08.022 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (31m) 8s ago 45m 83.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:55:08.022 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (40m) 8s ago 40m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:55:08.023 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (40m) 4s ago 40m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:55:08.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (29m) 8s ago 44m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:55:08.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (28m) 8s ago 44m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:55:08.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (28m) 8s ago 43m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:55:08.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (27m) 8s ago 43m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:55:08.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (27m) 4s ago 43m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:55:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (26m) 4s ago 42m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:55:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (25m) 4s ago 42m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:55:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (24m) 4s ago 41m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:55:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (33m) 4s ago 41m 66.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:55:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (24m) 8s ago 39m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:55:08.024 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (24m) 4s ago 39m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:55:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (24m) 8s ago 39m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:55:08.025 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (24m) 4s ago 38m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:55:08.331 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:08.332 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:55:08.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:55:08.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:55:08.333 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:55:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:55:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:55:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:55:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:55:08.602 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:55:08.911 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:55:08.911 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:08.911 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:08.911 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:08.911 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[99959]: from='client.46422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[99959]: pgmap v1057: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[99959]: from='client.46428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[99959]: from='client.46434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3680443316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[91136]: from='client.46422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[91136]: pgmap v1057: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[91136]: from='client.46428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[91136]: from='client.46434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:08.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3680443316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:08 smithi146 ceph-mon[103278]: from='client.46422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:08 smithi146 ceph-mon[103278]: pgmap v1057: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:08 smithi146 ceph-mon[103278]: from='client.46428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:08 smithi146 ceph-mon[103278]: from='client.46434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3680443316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:09 smithi146 ceph-mon[103278]: from='client.66365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2044189356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:10.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:09 smithi078 ceph-mon[99959]: from='client.66365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:10.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2044189356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:10.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:09 smithi078 ceph-mon[91136]: from='client.66365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:10.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2044189356' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:10.478 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:10.223Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:10.478 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:10.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:10.478 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:10.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:55:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:55:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:10 smithi146 ceph-mon[103278]: pgmap v1058: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:10 smithi078 ceph-mon[91136]: pgmap v1058: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:10 smithi078 ceph-mon[99959]: pgmap v1058: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:12.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:12 smithi146 ceph-mon[103278]: pgmap v1059: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:12 smithi078 ceph-mon[91136]: pgmap v1059: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:12 smithi078 ceph-mon[99959]: pgmap v1059: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:13.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:13.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:13.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:14 smithi146 ceph-mon[103278]: pgmap v1060: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:14 smithi078 ceph-mon[91136]: pgmap v1060: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:14 smithi078 ceph-mon[99959]: pgmap v1060: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:17.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:16 smithi078 ceph-mon[99959]: pgmap v1061: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:17.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:16 smithi078 ceph-mon[91136]: pgmap v1061: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:16 smithi146 ceph-mon[103278]: pgmap v1061: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:18 smithi146 ceph-mon[103278]: pgmap v1062: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:18 smithi078 ceph-mon[91136]: pgmap v1062: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:18 smithi078 ceph-mon[99959]: pgmap v1062: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[91136]: pgmap v1063: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-mon[99959]: pgmap v1063: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:20.485 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:20.224Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:20.485 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:20.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:20.485 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:20.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:20 smithi146 ceph-mon[103278]: pgmap v1063: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:55:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-20T20:55:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:22 smithi146 ceph-mon[103278]: pgmap v1064: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:22 smithi078 ceph-mon[91136]: pgmap v1064: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:22 smithi078 ceph-mon[99959]: pgmap v1064: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:23.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:23.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:23.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:24 smithi146 ceph-mon[103278]: pgmap v1065: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[91136]: pgmap v1065: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[99959]: pgmap v1065: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:26 smithi146 ceph-mon[103278]: pgmap v1066: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:26 smithi078 ceph-mon[91136]: pgmap v1066: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:26 smithi078 ceph-mon[99959]: pgmap v1066: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:28 smithi146 ceph-mon[103278]: pgmap v1067: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:28 smithi078 ceph-mon[99959]: pgmap v1067: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:28 smithi078 ceph-mon[91136]: pgmap v1067: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:30.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:30.225Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:30.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:30.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:30.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:30.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:55:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:55:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:30 smithi146 ceph-mon[103278]: pgmap v1068: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:30 smithi078 ceph-mon[91136]: pgmap v1068: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:30 smithi078 ceph-mon[99959]: pgmap v1068: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:32 smithi146 ceph-mon[103278]: pgmap v1069: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:32 smithi078 ceph-mon[91136]: pgmap v1069: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:32 smithi078 ceph-mon[99959]: pgmap v1069: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:33.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:33.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:33.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:34 smithi146 ceph-mon[103278]: pgmap v1070: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:34 smithi078 ceph-mon[91136]: pgmap v1070: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:34 smithi078 ceph-mon[99959]: pgmap v1070: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:37.245 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:36 smithi078 ceph-mon[99959]: pgmap v1071: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:37.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:36 smithi078 ceph-mon[91136]: pgmap v1071: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:36 smithi146 ceph-mon[103278]: pgmap v1071: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:39.202 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:55:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:38 smithi146 ceph-mon[103278]: pgmap v1072: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:38 smithi078 ceph-mon[99959]: pgmap v1072: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:38 smithi078 ceph-mon[91136]: pgmap v1072: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:39.744 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:55:39.744 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 46m - - 2024-09-20T20:55:39.744 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 46m - - 2024-09-20T20:55:39.744 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (40m) 0s ago 41m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:55:39.744 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (41m) 15s ago 41m 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:55:39.744 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (33m) 0s ago 39m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (32m) 15s ago 45m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (34m) 0s ago 47m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (32m) 0s ago 47m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (31m) 15s ago 45m 79.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (31m) 0s ago 45m 84.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (41m) 0s ago 40m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (40m) 15s ago 40m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (30m) 0s ago 45m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (29m) 0s ago 44m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (28m) 0s ago 44m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (28m) 0s ago 43m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (27m) 15s ago 43m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (26m) 15s ago 43m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:55:39.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (26m) 15s ago 42m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:55:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (25m) 15s ago 42m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:55:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (34m) 15s ago 41m 66.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:55:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (24m) 0s ago 39m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:55:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (24m) 15s ago 39m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:55:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (24m) 0s ago 39m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:55:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (24m) 15s ago 39m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:55:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:55:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:55:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:55:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:55:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:55:40.060 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:55:40.330 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:55:40.330 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:55:40.330 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:55:40.330 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:55:40.331 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:55:40.331 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:40.225Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:40.331 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:40.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:40.331 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:40.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:40.642 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:55:40.642 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:40.642 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:55:40.642 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:40.642 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:55:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='client.66377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='client.46464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: pgmap v1073: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='client.46470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/362921251' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:55:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='client.66377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='client.46464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: pgmap v1073: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='client.46470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/362921251' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='client.66377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='client.46464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: pgmap v1073: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='client.46470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/362921251' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:55:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:41 smithi078 ceph-mon[91136]: from='client.46482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3857306213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:41 smithi078 ceph-mon[99959]: from='client.46482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3857306213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:41 smithi146 ceph-mon[103278]: from='client.46482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:55:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3857306213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:55:42.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:42 smithi146 ceph-mon[103278]: pgmap v1074: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:42 smithi078 ceph-mon[91136]: pgmap v1074: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:42 smithi078 ceph-mon[99959]: pgmap v1074: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:43.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:43.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:43.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:45 smithi146 ceph-mon[103278]: pgmap v1075: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[91136]: pgmap v1075: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[99959]: pgmap v1075: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:55:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:47 smithi146 ceph-mon[103278]: pgmap v1076: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:47 smithi078 ceph-mon[99959]: pgmap v1076: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:47 smithi078 ceph-mon[91136]: pgmap v1076: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:49.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:49 smithi146 ceph-mon[103278]: pgmap v1077: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:49 smithi078 ceph-mon[99959]: pgmap v1077: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:49 smithi078 ceph-mon[91136]: pgmap v1077: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:50.502 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:50.226Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:50.502 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:50.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:50.502 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:50.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:55:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:55:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:51 smithi146 ceph-mon[103278]: pgmap v1078: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:51 smithi078 ceph-mon[91136]: pgmap v1078: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:51 smithi078 ceph-mon[99959]: pgmap v1078: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:55:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:53 smithi146 ceph-mon[103278]: pgmap v1079: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:53 smithi078 ceph-mon[91136]: pgmap v1079: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:53 smithi078 ceph-mon[99959]: pgmap v1079: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:53.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:53.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:55:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:55:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:55:53.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:55:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:55 smithi146 ceph-mon[103278]: pgmap v1080: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:55 smithi078 ceph-mon[91136]: pgmap v1080: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:55 smithi078 ceph-mon[99959]: pgmap v1080: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:55:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:57 smithi146 ceph-mon[103278]: pgmap v1081: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:57 smithi078 ceph-mon[99959]: pgmap v1081: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:57 smithi078 ceph-mon[91136]: pgmap v1081: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:59.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:59 smithi078 ceph-mon[91136]: pgmap v1082: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:59.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:59.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:59 smithi078 ceph-mon[99959]: pgmap v1082: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:59.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:55:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:59 smithi146 ceph-mon[103278]: pgmap v1082: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:55:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:00.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:00.227Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:00.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:00.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:00.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:00.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:56:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:56:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[99959]: pgmap v1083: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[91136]: pgmap v1083: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:00 smithi146 ceph-mon[103278]: pgmap v1083: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:02 smithi146 ceph-mon[103278]: pgmap v1084: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:02 smithi078 ceph-mon[99959]: pgmap v1084: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:02 smithi078 ceph-mon[91136]: pgmap v1084: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:03.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:03.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:03.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:04 smithi146 ceph-mon[103278]: pgmap v1085: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[99959]: pgmap v1085: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[91136]: pgmap v1085: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:06 smithi146 ceph-mon[103278]: pgmap v1086: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:06 smithi078 ceph-mon[91136]: pgmap v1086: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:06 smithi078 ceph-mon[99959]: pgmap v1086: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:08.898 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:08 smithi146 ceph-mon[103278]: pgmap v1087: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:08.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:08 smithi078 ceph-mon[99959]: pgmap v1087: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:08.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:08 smithi078 ceph-mon[91136]: pgmap v1087: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:10.301 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:10.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:10.301 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:10.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:10.301 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:10.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:10.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:10 smithi146 ceph-mon[103278]: pgmap v1088: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:10.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:56:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:56:10.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:10 smithi078 ceph-mon[99959]: pgmap v1088: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:10.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:10 smithi078 ceph-mon[91136]: pgmap v1088: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:10.939 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 46m - - 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 46m - - 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (40m) 11s ago 42m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (41m) 7s ago 41m 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (34m) 11s ago 39m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:56:11.487 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (33m) 7s ago 45m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (35m) 11s ago 47m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (33m) 11s ago 47m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (32m) 7s ago 46m 80.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (32m) 11s ago 46m 85.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (41m) 11s ago 41m 14.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (41m) 7s ago 41m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (31m) 11s ago 45m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (30m) 11s ago 45m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (29m) 11s ago 44m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (28m) 11s ago 44m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (28m) 7s ago 44m 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (27m) 7s ago 43m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (26m) 7s ago 43m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (25m) 7s ago 42m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (34m) 7s ago 42m 66.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:56:11.488 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (25m) 11s ago 40m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:56:11.489 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (25m) 7s ago 40m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:56:11.489 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (25m) 11s ago 40m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:56:11.489 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (25m) 7s ago 40m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:56:11.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:56:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:56:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:56:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:56:11.809 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:56:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:11 smithi146 ceph-mon[103278]: from='client.66413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:11 smithi078 ceph-mon[91136]: from='client.66413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:11 smithi078 ceph-mon[99959]: from='client.66413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:56:12.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:56:12.080 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:56:12.383 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:56:12.384 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:12.384 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:12 smithi146 ceph-mon[103278]: from='client.66419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:12 smithi146 ceph-mon[103278]: from='client.66425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:12 smithi146 ceph-mon[103278]: pgmap v1089: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4201999998' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:12 smithi146 ceph-mon[103278]: from='client.46515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/146480152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[91136]: from='client.66419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[91136]: from='client.66425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[91136]: pgmap v1089: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4201999998' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[91136]: from='client.46515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/146480152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[99959]: from='client.66419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[99959]: from='client.66425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[99959]: pgmap v1089: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4201999998' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:13.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[99959]: from='client.46515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:13.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/146480152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:13.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:13.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:13.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:14 smithi146 ceph-mon[103278]: pgmap v1090: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:14 smithi078 ceph-mon[91136]: pgmap v1090: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:14 smithi078 ceph-mon[99959]: pgmap v1090: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:16 smithi146 ceph-mon[103278]: pgmap v1091: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:16 smithi078 ceph-mon[99959]: pgmap v1091: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:16 smithi078 ceph-mon[91136]: pgmap v1091: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:18.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:18 smithi078 ceph-mon[99959]: pgmap v1092: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:18.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:18 smithi078 ceph-mon[91136]: pgmap v1092: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:18 smithi146 ceph-mon[103278]: pgmap v1092: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:20.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:20.230Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:20.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:20.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:20.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:20.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:56:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:56:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:20 smithi146 ceph-mon[103278]: pgmap v1093: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[99959]: pgmap v1093: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[91136]: pgmap v1093: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:22 smithi146 ceph-mon[103278]: pgmap v1094: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:22 smithi078 ceph-mon[99959]: pgmap v1094: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:22 smithi078 ceph-mon[91136]: pgmap v1094: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:24.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:23.669Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:23.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:23.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:24 smithi146 ceph-mon[103278]: pgmap v1095: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[91136]: pgmap v1095: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[99959]: pgmap v1095: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:26 smithi146 ceph-mon[103278]: pgmap v1096: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:26 smithi078 ceph-mon[91136]: pgmap v1096: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:26 smithi078 ceph-mon[99959]: pgmap v1096: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:29.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:28 smithi146 ceph-mon[103278]: pgmap v1097: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:29.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:28 smithi078 ceph-mon[91136]: pgmap v1097: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:28 smithi078 ceph-mon[99959]: pgmap v1097: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:30.231Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:30.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:30.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:30.844 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:56:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:56:31.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:30 smithi146 ceph-mon[103278]: pgmap v1098: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:30 smithi078 ceph-mon[91136]: pgmap v1098: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:30 smithi078 ceph-mon[99959]: pgmap v1098: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:32 smithi146 ceph-mon[103278]: pgmap v1099: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:32 smithi078 ceph-mon[99959]: pgmap v1099: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:32 smithi078 ceph-mon[91136]: pgmap v1099: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:33.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:33.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:33.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:34 smithi146 ceph-mon[103278]: pgmap v1100: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:34 smithi078 ceph-mon[99959]: pgmap v1100: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:34 smithi078 ceph-mon[91136]: pgmap v1100: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:36 smithi146 ceph-mon[103278]: pgmap v1101: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:36 smithi078 ceph-mon[91136]: pgmap v1101: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:36 smithi078 ceph-mon[99959]: pgmap v1101: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:39.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:38 smithi078 ceph-mon[91136]: pgmap v1102: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:39.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:38 smithi078 ceph-mon[99959]: pgmap v1102: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:38 smithi146 ceph-mon[103278]: pgmap v1102: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:40.231Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:40.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:40.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:40.945 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:56:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:40 smithi146 ceph-mon[103278]: pgmap v1103: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[99959]: pgmap v1103: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[91136]: pgmap v1103: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:42.669 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:56:43.197 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:56:43.197 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 47m - - 2024-09-20T20:56:43.197 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 47m - - 2024-09-20T20:56:43.197 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (41m) 3s ago 42m 29.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:56:43.197 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (42m) 18s ago 42m 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (34m) 3s ago 40m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (33m) 18s ago 46m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (35m) 3s ago 48m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (33m) 3s ago 48m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (32m) 18s ago 46m 81.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (32m) 3s ago 46m 86.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (42m) 3s ago 42m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (41m) 18s ago 41m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (31m) 3s ago 46m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (30m) 3s ago 45m 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (29m) 3s ago 45m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (29m) 3s ago 45m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (28m) 18s ago 44m 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:56:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (27m) 18s ago 44m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (27m) 18s ago 43m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (26m) 18s ago 43m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (35m) 18s ago 42m 66.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (25m) 3s ago 40m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (25m) 18s ago 40m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (25m) 3s ago 40m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:56:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (25m) 18s ago 40m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:56:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:42 smithi146 ceph-mon[103278]: pgmap v1104: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:42 smithi078 ceph-mon[99959]: pgmap v1104: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:42 smithi078 ceph-mon[91136]: pgmap v1104: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:56:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:56:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:56:43.783 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:56:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:56:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:56:43.784 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:56:43.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:43.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:43.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:43.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:43.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:43.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[91136]: from='client.66449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[91136]: from='client.66455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1977297862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[99959]: from='client.66449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[99959]: from='client.66455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1977297862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:44.095 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:56:44.095 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:56:44.095 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:56:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:43 smithi146 ceph-mon[103278]: from='client.66449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:43 smithi146 ceph-mon[103278]: from='client.66455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1977297862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:56:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: from='client.46539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: pgmap v1105: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: from='client.66473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/916488255' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: from='client.46539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: pgmap v1105: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: from='client.66473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/916488255' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: from='client.46539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: pgmap v1105: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: from='client.66473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/916488255' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:56:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:56:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:46 smithi146 ceph-mon[103278]: pgmap v1106: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:46 smithi078 ceph-mon[91136]: pgmap v1106: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:46 smithi078 ceph-mon[99959]: pgmap v1106: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:48 smithi146 ceph-mon[103278]: pgmap v1107: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:48 smithi078 ceph-mon[91136]: pgmap v1107: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:48 smithi078 ceph-mon[99959]: pgmap v1107: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:50.233Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:50.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:50.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:50.972 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:56:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T20:56:51.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:50 smithi146 ceph-mon[103278]: pgmap v1108: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:50 smithi078 ceph-mon[91136]: pgmap v1108: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:50 smithi078 ceph-mon[99959]: pgmap v1108: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:56:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:52 smithi146 ceph-mon[103278]: pgmap v1109: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:52 smithi078 ceph-mon[99959]: pgmap v1109: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:52 smithi078 ceph-mon[91136]: pgmap v1109: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:53.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:53.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:56:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:56:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:56:53.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:56:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:54 smithi146 ceph-mon[103278]: pgmap v1110: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:54 smithi078 ceph-mon[91136]: pgmap v1110: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:54 smithi078 ceph-mon[99959]: pgmap v1110: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:56:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:56 smithi146 ceph-mon[103278]: pgmap v1111: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:56 smithi078 ceph-mon[99959]: pgmap v1111: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:56 smithi078 ceph-mon[91136]: pgmap v1111: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:58 smithi078 ceph-mon[91136]: pgmap v1112: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:56:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:58 smithi078 ceph-mon[99959]: pgmap v1112: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:56:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:56:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:58 smithi146 ceph-mon[103278]: pgmap v1112: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:56:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:56:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:00.234Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:00.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:00.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:00.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:57:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[99959]: pgmap v1113: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[91136]: pgmap v1113: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:01 smithi146 ceph-mon[103278]: pgmap v1113: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:01.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:01.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:03 smithi146 ceph-mon[103278]: pgmap v1114: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:03 smithi078 ceph-mon[91136]: pgmap v1114: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:03 smithi078 ceph-mon[99959]: pgmap v1114: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:03.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:03.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:03.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:05 smithi146 ceph-mon[103278]: pgmap v1115: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[91136]: pgmap v1115: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[99959]: pgmap v1115: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:07 smithi146 ceph-mon[103278]: pgmap v1116: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:07 smithi078 ceph-mon[91136]: pgmap v1116: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:07 smithi078 ceph-mon[99959]: pgmap v1116: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:09 smithi146 ceph-mon[103278]: pgmap v1117: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:09 smithi078 ceph-mon[91136]: pgmap v1117: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:09 smithi078 ceph-mon[99959]: pgmap v1117: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:10.234Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:10.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:10.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:10.892 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:57:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T20:57:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:11 smithi078 ceph-mon[91136]: pgmap v1118: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:11 smithi078 ceph-mon[99959]: pgmap v1118: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:11.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:11 smithi146 ceph-mon[103278]: pgmap v1118: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:12.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:13 smithi146 ceph-mon[103278]: pgmap v1119: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:13 smithi078 ceph-mon[91136]: pgmap v1119: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:13 smithi078 ceph-mon[99959]: pgmap v1119: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:14.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:13.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:13.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:13.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:14.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:14.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:14.394 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:57:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:14.918 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:57:14.918 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 47m - - 2024-09-20T20:57:14.918 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 47m - - 2024-09-20T20:57:14.918 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (42m) 14s ago 43m 29.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (42m) 10s ago 42m 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (35m) 14s ago 40m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (34m) 10s ago 47m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (36m) 14s ago 48m 542M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (34m) 14s ago 48m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (33m) 10s ago 47m 80.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (33m) 14s ago 47m 86.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (42m) 14s ago 42m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (42m) 10s ago 42m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (32m) 14s ago 46m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (31m) 14s ago 46m 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (30m) 14s ago 45m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (29m) 14s ago 45m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:57:14.919 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (29m) 10s ago 45m 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (28m) 10s ago 44m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (27m) 10s ago 44m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (26m) 10s ago 43m 56.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (35m) 10s ago 43m 66.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (26m) 14s ago 41m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (26m) 10s ago 41m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (26m) 14s ago 41m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:57:14.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (26m) 10s ago 41m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:57:15.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:57:15.232 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:57:15.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:15 smithi078 ceph-mon[99959]: pgmap v1120: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:15.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:15 smithi078 ceph-mon[91136]: pgmap v1120: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:57:15.499 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:57:15.500 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:57:15.500 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:57:15.500 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:57:15.500 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:57:15.500 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:57:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:15 smithi146 ceph-mon[103278]: pgmap v1120: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:15.815 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:57:15.815 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:15.815 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:15.815 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:15.815 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:16 smithi146 ceph-mon[103278]: from='client.66485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:16 smithi146 ceph-mon[103278]: from='client.66491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:16 smithi146 ceph-mon[103278]: from='client.66497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/197139236' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1089681444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[91136]: from='client.66485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[91136]: from='client.66491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[91136]: from='client.66497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/197139236' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1089681444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[99959]: from='client.66485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[99959]: from='client.66491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[99959]: from='client.66497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:16.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/197139236' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:16.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1089681444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:17 smithi146 ceph-mon[103278]: from='client.66509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:17 smithi146 ceph-mon[103278]: pgmap v1121: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:17 smithi078 ceph-mon[91136]: from='client.66509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:17 smithi078 ceph-mon[91136]: pgmap v1121: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:17 smithi078 ceph-mon[99959]: from='client.66509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:17 smithi078 ceph-mon[99959]: pgmap v1121: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:19.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:19 smithi078 ceph-mon[91136]: pgmap v1122: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:19.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:19 smithi078 ceph-mon[99959]: pgmap v1122: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:19 smithi146 ceph-mon[103278]: pgmap v1122: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:20.417 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:20.234Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:20.418 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:20.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:20.418 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:20.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:20.669 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:57:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T20:57:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[91136]: pgmap v1123: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[99959]: pgmap v1123: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:21 smithi146 ceph-mon[103278]: pgmap v1123: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:23 smithi146 ceph-mon[103278]: pgmap v1124: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:23 smithi078 ceph-mon[91136]: pgmap v1124: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:23 smithi078 ceph-mon[99959]: pgmap v1124: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:23.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:23.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:23.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:25 smithi146 ceph-mon[103278]: pgmap v1125: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[91136]: pgmap v1125: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[99959]: pgmap v1125: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:27 smithi146 ceph-mon[103278]: pgmap v1126: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:27 smithi078 ceph-mon[91136]: pgmap v1126: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:27 smithi078 ceph-mon[99959]: pgmap v1126: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:29 smithi146 ceph-mon[103278]: pgmap v1127: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:29 smithi078 ceph-mon[91136]: pgmap v1127: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:29 smithi078 ceph-mon[99959]: pgmap v1127: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:30.235Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:30.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:30.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:57:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:57:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:31 smithi146 ceph-mon[103278]: pgmap v1128: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:31 smithi078 ceph-mon[91136]: pgmap v1128: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:31 smithi078 ceph-mon[99959]: pgmap v1128: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:33 smithi146 ceph-mon[103278]: pgmap v1129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:33 smithi078 ceph-mon[91136]: pgmap v1129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:33 smithi078 ceph-mon[99959]: pgmap v1129: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T20:57:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:33.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:33.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:33.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:34 smithi146 ceph-mon[103278]: pgmap v1130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:34 smithi078 ceph-mon[91136]: pgmap v1130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:34 smithi078 ceph-mon[99959]: pgmap v1130: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:36 smithi146 ceph-mon[103278]: pgmap v1131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:36 smithi078 ceph-mon[91136]: pgmap v1131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:36 smithi078 ceph-mon[99959]: pgmap v1131: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:38 smithi146 ceph-mon[103278]: pgmap v1132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:38 smithi078 ceph-mon[91136]: pgmap v1132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:38 smithi078 ceph-mon[99959]: pgmap v1132: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:40.236Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:40.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:40.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:40.894 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:57:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[99959]: pgmap v1133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[91136]: pgmap v1133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:40.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:40 smithi146 ceph-mon[103278]: pgmap v1133: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:41.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:41.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:42 smithi078 ceph-mon[91136]: pgmap v1134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:42 smithi078 ceph-mon[99959]: pgmap v1134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:42.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:42 smithi146 ceph-mon[103278]: pgmap v1134: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:43.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:43.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:43.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:44 smithi146 ceph-mon[103278]: pgmap v1135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:44 smithi078 ceph-mon[91136]: pgmap v1135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:44 smithi078 ceph-mon[99959]: pgmap v1135: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:46.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.115 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:57:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:57:46.646 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:57:46.646 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 48m - - 2024-09-20T20:57:46.646 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 48m - - 2024-09-20T20:57:46.646 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (42m) 6s ago 43m 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:57:46.646 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (43m) 1s ago 43m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (35m) 6s ago 41m 62.1M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (34m) 1s ago 47m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (36m) 6s ago 49m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (34m) 6s ago 49m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (33m) 1s ago 48m 80.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (33m) 6s ago 47m 86.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (43m) 6s ago 43m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (43m) 1s ago 43m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (32m) 6s ago 47m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (31m) 6s ago 46m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (31m) 6s ago 46m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (30m) 6s ago 46m 64.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (29m) 1s ago 45m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:57:46.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (28m) 1s ago 45m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (28m) 1s ago 44m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (27m) 1s ago 44m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (36m) 1s ago 43m 67.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (26m) 6s ago 41m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (26m) 1s ago 41m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (26m) 6s ago 41m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:57:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (26m) 1s ago 41m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:57:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:57:46.961 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:57:46.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:46 smithi078 ceph-mon[99959]: pgmap v1136: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:46.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:46 smithi078 ceph-mon[99959]: from='client.66521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:46.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:46 smithi078 ceph-mon[91136]: pgmap v1136: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:46.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:46 smithi078 ceph-mon[91136]: from='client.66521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:47.229 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:57:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:57:47.230 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:57:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:46 smithi146 ceph-mon[103278]: pgmap v1136: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:46 smithi146 ceph-mon[103278]: from='client.66521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:47.539 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:57:47.540 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:47.540 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:57:47.540 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:47.540 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:57:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:47 smithi146 ceph-mon[103278]: from='client.66527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:47 smithi146 ceph-mon[103278]: from='client.46611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2942474392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3833743232' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[91136]: from='client.66527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[91136]: from='client.46611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2942474392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3833743232' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[99959]: from='client.66527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[99959]: from='client.46611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2942474392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:57:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3833743232' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:57:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:48 smithi146 ceph-mon[103278]: from='client.66545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:48 smithi146 ceph-mon[103278]: pgmap v1137: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:48 smithi078 ceph-mon[91136]: from='client.66545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:48 smithi078 ceph-mon[91136]: pgmap v1137: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:48 smithi078 ceph-mon[99959]: from='client.66545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:57:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:48 smithi078 ceph-mon[99959]: pgmap v1137: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:50.237Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:50.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:50.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:50 smithi078 ceph-mon[91136]: pgmap v1138: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:50 smithi078 ceph-mon[99959]: pgmap v1138: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:57:50] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T20:57:51.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:50 smithi146 ceph-mon[103278]: pgmap v1138: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:52.117 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:57:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:52 smithi146 ceph-mon[103278]: pgmap v1139: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:52 smithi078 ceph-mon[91136]: pgmap v1139: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:52 smithi078 ceph-mon[99959]: pgmap v1139: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:53.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:53.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:57:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:57:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:57:53.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:57:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:54 smithi146 ceph-mon[103278]: pgmap v1140: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:54 smithi078 ceph-mon[91136]: pgmap v1140: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:54 smithi078 ceph-mon[99959]: pgmap v1140: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:57:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:56 smithi146 ceph-mon[103278]: pgmap v1141: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:56 smithi078 ceph-mon[99959]: pgmap v1141: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:56 smithi078 ceph-mon[91136]: pgmap v1141: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:58 smithi146 ceph-mon[103278]: pgmap v1142: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:58 smithi078 ceph-mon[91136]: pgmap v1142: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:57:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:58 smithi078 ceph-mon[99959]: pgmap v1142: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:57:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:00.237Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:00.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:00.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:00.867 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:58:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:58:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:00 smithi146 ceph-mon[103278]: pgmap v1143: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[99959]: pgmap v1143: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[91136]: pgmap v1143: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:01.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:01.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:02 smithi078 ceph-mon[91136]: pgmap v1144: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:02 smithi078 ceph-mon[99959]: pgmap v1144: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:02 smithi146 ceph-mon[103278]: pgmap v1144: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:03.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:03.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:03.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:04 smithi146 ceph-mon[103278]: pgmap v1145: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:04 smithi078 ceph-mon[91136]: pgmap v1145: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:04 smithi078 ceph-mon[99959]: pgmap v1145: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:06 smithi146 ceph-mon[103278]: pgmap v1146: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:06 smithi078 ceph-mon[91136]: pgmap v1146: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:06 smithi078 ceph-mon[99959]: pgmap v1146: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:08 smithi146 ceph-mon[103278]: pgmap v1147: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:08 smithi078 ceph-mon[91136]: pgmap v1147: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:08 smithi078 ceph-mon[99959]: pgmap v1147: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:10.238Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:10.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:10.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:10.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:58:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:58:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:10 smithi146 ceph-mon[103278]: pgmap v1148: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:10 smithi078 ceph-mon[91136]: pgmap v1148: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:10 smithi078 ceph-mon[99959]: pgmap v1148: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:12 smithi146 ceph-mon[103278]: pgmap v1149: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:12 smithi078 ceph-mon[91136]: pgmap v1149: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:12 smithi078 ceph-mon[99959]: pgmap v1149: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:13.995 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:13.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:13.995 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:13.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:13.995 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:13.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:14 smithi146 ceph-mon[103278]: pgmap v1150: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:14 smithi078 ceph-mon[99959]: pgmap v1150: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:14 smithi078 ceph-mon[91136]: pgmap v1150: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:16 smithi146 ceph-mon[103278]: pgmap v1151: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:16 smithi078 ceph-mon[91136]: pgmap v1151: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:16 smithi078 ceph-mon[99959]: pgmap v1151: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:17.835 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 48m - - 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 48m - - 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (43m) 17s ago 44m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (43m) 13s ago 43m 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (36m) 17s ago 41m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (35m) 13s ago 48m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (37m) 17s ago 49m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (35m) 17s ago 49m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (34m) 13s ago 48m 81.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:58:18.365 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (34m) 17s ago 48m 85.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (43m) 17s ago 43m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (43m) 13s ago 43m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (33m) 17s ago 47m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (32m) 17s ago 47m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (31m) 17s ago 47m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (31m) 17s ago 46m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (30m) 13s ago 46m 68.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (29m) 13s ago 45m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (28m) 13s ago 45m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (27m) 13s ago 45m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (36m) 13s ago 44m 67.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:58:18.366 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (27m) 17s ago 42m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:58:18.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (27m) 13s ago 42m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:58:18.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (27m) 17s ago 42m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:58:18.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (27m) 13s ago 42m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:58:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:58:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:58:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:58:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:58:18.677 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:58:18.949 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:58:18.950 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:58:19.257 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:58:19.257 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:19.257 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:19.257 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:19.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[91136]: pgmap v1152: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:19.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[91136]: from='client.66557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:19.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[91136]: from='client.66563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:19.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3753327343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:19.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[99959]: pgmap v1152: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:19.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[99959]: from='client.66557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:19.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[99959]: from='client.66563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:19.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3753327343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:19 smithi146 ceph-mon[103278]: pgmap v1152: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:19 smithi146 ceph-mon[103278]: from='client.66557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:19 smithi146 ceph-mon[103278]: from='client.66563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3753327343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:20 smithi146 ceph-mon[103278]: from='client.46647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:20 smithi146 ceph-mon[103278]: from='client.66581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/753372996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-mon[91136]: from='client.46647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-mon[91136]: from='client.66581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/753372996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-mon[99959]: from='client.46647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-mon[99959]: from='client.66581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/753372996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:20.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:20.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:20.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:20.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:20.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:21.012 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:58:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T20:58:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:21 smithi146 ceph-mon[103278]: pgmap v1153: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[91136]: pgmap v1153: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[99959]: pgmap v1153: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:21.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:23 smithi078 ceph-mon[91136]: pgmap v1154: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:23 smithi078 ceph-mon[99959]: pgmap v1154: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:23 smithi146 ceph-mon[103278]: pgmap v1154: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:23.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:23.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:23.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:25 smithi078 ceph-mon[91136]: pgmap v1155: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:25 smithi078 ceph-mon[99959]: pgmap v1155: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:25 smithi146 ceph-mon[103278]: pgmap v1155: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:27 smithi146 ceph-mon[103278]: pgmap v1156: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:27 smithi078 ceph-mon[91136]: pgmap v1156: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:27 smithi078 ceph-mon[99959]: pgmap v1156: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:29 smithi146 ceph-mon[103278]: pgmap v1157: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:29 smithi078 ceph-mon[91136]: pgmap v1157: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:29 smithi078 ceph-mon[99959]: pgmap v1157: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:30.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:30.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:30.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:58:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T20:58:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:31 smithi146 ceph-mon[103278]: pgmap v1158: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:31 smithi078 ceph-mon[91136]: pgmap v1158: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:31 smithi078 ceph-mon[99959]: pgmap v1158: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:32.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:33.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:33 smithi146 ceph-mon[103278]: pgmap v1159: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:33 smithi078 ceph-mon[91136]: pgmap v1159: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:33 smithi078 ceph-mon[99959]: pgmap v1159: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:33.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:33.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:33.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:35 smithi146 ceph-mon[103278]: pgmap v1160: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:35 smithi078 ceph-mon[99959]: pgmap v1160: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:35 smithi078 ceph-mon[91136]: pgmap v1160: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:37 smithi146 ceph-mon[103278]: pgmap v1161: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:37 smithi078 ceph-mon[99959]: pgmap v1161: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:37 smithi078 ceph-mon[91136]: pgmap v1161: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:39 smithi146 ceph-mon[103278]: pgmap v1162: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:39 smithi078 ceph-mon[91136]: pgmap v1162: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:39 smithi078 ceph-mon[99959]: pgmap v1162: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:40.240Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:40.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:40.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:58:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T20:58:41.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[91136]: pgmap v1163: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:41.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[99959]: pgmap v1163: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:41.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:41 smithi146 ceph-mon[103278]: pgmap v1163: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:43.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-mon[91136]: pgmap v1164: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:43.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:43.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-mon[99959]: pgmap v1164: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:43.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:43.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:43 smithi146 ceph-mon[103278]: pgmap v1164: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:43.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:43.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:43.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:43.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:45 smithi146 ceph-mon[103278]: pgmap v1165: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:45 smithi078 ceph-mon[91136]: pgmap v1165: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:45 smithi078 ceph-mon[99959]: pgmap v1165: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:46 smithi146 ceph-mon[103278]: pgmap v1166: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[91136]: pgmap v1166: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:58:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:46 smithi078 ceph-mon[99959]: pgmap v1166: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:48 smithi146 ceph-mon[103278]: pgmap v1167: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:48 smithi078 ceph-mon[91136]: pgmap v1167: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:48 smithi078 ceph-mon[99959]: pgmap v1167: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:49.552 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:58:50.082 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:58:50.082 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 49m - - 2024-09-20T20:58:50.082 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 49m - - 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (43m) 9s ago 44m 29.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (44m) 4s ago 44m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (36m) 9s ago 42m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (35m) 4s ago 48m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (37m) 9s ago 50m 543M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (35m) 9s ago 50m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (34m) 4s ago 49m 82.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (35m) 9s ago 48m 87.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (44m) 9s ago 44m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (44m) 4s ago 44m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (33m) 9s ago 48m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (32m) 9s ago 47m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:58:50.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (32m) 9s ago 47m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (31m) 9s ago 47m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (31m) 4s ago 46m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (29m) 4s ago 46m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (29m) 4s ago 45m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (28m) 4s ago 45m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (37m) 4s ago 44m 67.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (27m) 9s ago 42m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (27m) 4s ago 42m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (27m) 9s ago 42m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:58:50.084 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (27m) 4s ago 42m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:58:50.396 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:58:50.396 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:58:50.396 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:58:50.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:58:50.397 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:58:50.398 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:50.240Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:50.398 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:50.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:50.398 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:50.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:50.667 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:58:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T20:58:50.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[99959]: from='client.66593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[99959]: pgmap v1168: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[99959]: from='client.66599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[99959]: from='client.46683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1182363815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[91136]: from='client.66593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[91136]: pgmap v1168: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[91136]: from='client.66599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[91136]: from='client.46683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:50.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1182363815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:58:50.669 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:58:50.670 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:58:50.670 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:58:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:58:50.977 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:58:50.977 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:50.977 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:58:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:50 smithi146 ceph-mon[103278]: from='client.66593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:50 smithi146 ceph-mon[103278]: pgmap v1168: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:50 smithi146 ceph-mon[103278]: from='client.66599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:50 smithi146 ceph-mon[103278]: from='client.46683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1182363815' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:58:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:51 smithi146 ceph-mon[103278]: from='client.46695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2143749692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:51 smithi078 ceph-mon[91136]: from='client.46695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2143749692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:58:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:51 smithi078 ceph-mon[99959]: from='client.46695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:58:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2143749692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:58:52.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:52 smithi146 ceph-mon[103278]: pgmap v1169: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:52 smithi078 ceph-mon[99959]: pgmap v1169: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:52 smithi078 ceph-mon[91136]: pgmap v1169: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:53.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:53.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:58:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:58:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:58:53.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:58:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:54 smithi146 ceph-mon[103278]: pgmap v1170: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:54 smithi078 ceph-mon[91136]: pgmap v1170: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:54 smithi078 ceph-mon[99959]: pgmap v1170: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:58:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:56 smithi146 ceph-mon[103278]: pgmap v1171: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:56 smithi078 ceph-mon[91136]: pgmap v1171: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:56 smithi078 ceph-mon[99959]: pgmap v1171: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:58 smithi146 ceph-mon[103278]: pgmap v1172: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:58:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:58 smithi078 ceph-mon[91136]: pgmap v1172: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:58:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:58:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:58 smithi078 ceph-mon[99959]: pgmap v1172: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:58:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:58:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:00.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:00.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:00.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:00.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:00 smithi078 ceph-mon[91136]: pgmap v1173: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:00.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:00 smithi078 ceph-mon[99959]: pgmap v1173: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:00.881 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:59:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T20:59:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:00 smithi146 ceph-mon[103278]: pgmap v1173: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:02 smithi146 ceph-mon[103278]: pgmap v1174: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:02 smithi078 ceph-mon[91136]: pgmap v1174: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:02 smithi078 ceph-mon[99959]: pgmap v1174: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:03.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:03.680Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:03.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:03.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:04 smithi146 ceph-mon[103278]: pgmap v1175: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:04 smithi078 ceph-mon[91136]: pgmap v1175: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:04 smithi078 ceph-mon[99959]: pgmap v1175: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:06 smithi146 ceph-mon[103278]: pgmap v1176: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[91136]: pgmap v1176: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:06 smithi078 ceph-mon[99959]: pgmap v1176: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:08 smithi146 ceph-mon[103278]: pgmap v1177: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:08 smithi078 ceph-mon[91136]: pgmap v1177: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:08 smithi078 ceph-mon[99959]: pgmap v1177: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:10.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:10.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:10.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:10.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:10 smithi078 ceph-mon[91136]: pgmap v1178: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:10.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:59:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T20:59:10.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:10 smithi078 ceph-mon[99959]: pgmap v1178: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:10 smithi146 ceph-mon[103278]: pgmap v1178: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:12.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:12 smithi146 ceph-mon[103278]: pgmap v1179: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:12 smithi078 ceph-mon[91136]: pgmap v1179: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:12 smithi078 ceph-mon[99959]: pgmap v1179: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:13.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:13.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:13.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:14 smithi146 ceph-mon[103278]: pgmap v1180: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:14 smithi078 ceph-mon[91136]: pgmap v1180: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:14 smithi078 ceph-mon[99959]: pgmap v1180: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:16 smithi146 ceph-mon[103278]: pgmap v1181: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:16 smithi078 ceph-mon[91136]: pgmap v1181: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:16 smithi078 ceph-mon[99959]: pgmap v1181: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:18 smithi146 ceph-mon[103278]: pgmap v1182: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:18 smithi078 ceph-mon[91136]: pgmap v1182: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:18 smithi078 ceph-mon[99959]: pgmap v1182: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:20.243Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:20.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:20.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:20 smithi146 ceph-mon[103278]: pgmap v1183: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:21.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:20 smithi078 ceph-mon[99959]: pgmap v1183: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:21.018 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:59:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T20:59:21.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:20 smithi078 ceph-mon[91136]: pgmap v1183: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:21.274 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:59:21.805 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:59:21.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 50m - - 2024-09-20T20:59:21.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 49m - - 2024-09-20T20:59:21.805 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (44m) 0s ago 45m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:59:21.805 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (44m) 16s ago 44m 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:59:21.805 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (37m) 0s ago 42m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (36m) 16s ago 49m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (38m) 0s ago 50m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (36m) 0s ago 51m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (35m) 16s ago 49m 83.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (35m) 0s ago 49m 88.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (44m) 0s ago 44m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (44m) 16s ago 44m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (34m) 0s ago 48m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (33m) 0s ago 48m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (32m) 0s ago 48m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (32m) 0s ago 47m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (31m) 16s ago 47m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:59:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (30m) 16s ago 46m 57.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (29m) 16s ago 46m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (28m) 16s ago 46m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (37m) 16s ago 45m 63.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (28m) 0s ago 43m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (28m) 16s ago 43m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (28m) 0s ago 43m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:59:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (28m) 16s ago 43m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:59:22.119 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:59:22.119 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:59:22.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:59:22.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:59:22.120 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:22.391 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:59:22.391 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:59:22.391 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:59:22.391 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:59:22.392 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:59:22.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:59:22.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:59:22.392 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:59:22.392 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:59:22.392 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:59:22.393 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:59:22.393 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:59:22.393 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:59:22.393 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:59:22.702 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:59:22.702 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:22.702 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:22.702 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:22.703 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[99959]: from='client.66629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[99959]: from='client.66635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[99959]: pgmap v1184: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[99959]: from='client.46719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1076290092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1365297138' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[91136]: from='client.66629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[91136]: from='client.66635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[91136]: pgmap v1184: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[91136]: from='client.46719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1076290092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1365297138' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:23 smithi146 ceph-mon[103278]: from='client.66629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:23 smithi146 ceph-mon[103278]: from='client.66635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:23 smithi146 ceph-mon[103278]: pgmap v1184: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:23 smithi146 ceph-mon[103278]: from='client.46719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1076290092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1365297138' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:24.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:23.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": context deadline exceeded" 2024-09-20T20:59:24.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:23.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:24.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:23.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:24 smithi078 ceph-mon[91136]: from='client.66653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:24 smithi078 ceph-mon[99959]: from='client.66653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:24 smithi146 ceph-mon[103278]: from='client.66653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:25 smithi078 ceph-mon[91136]: pgmap v1185: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:25 smithi078 ceph-mon[99959]: pgmap v1185: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:25.482 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:25 smithi146 ceph-mon[103278]: pgmap v1185: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[99959]: pgmap v1186: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:26 smithi078 ceph-mon[91136]: pgmap v1186: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:26 smithi146 ceph-mon[103278]: pgmap v1186: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:28 smithi146 ceph-mon[103278]: pgmap v1187: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:28 smithi078 ceph-mon[99959]: pgmap v1187: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:28 smithi078 ceph-mon[91136]: pgmap v1187: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:30.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:30.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:30.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:30 smithi146 ceph-mon[103278]: pgmap v1188: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:30 smithi078 ceph-mon[99959]: pgmap v1188: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:30 smithi078 ceph-mon[91136]: pgmap v1188: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:59:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:59:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:32 smithi146 ceph-mon[103278]: pgmap v1189: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:32 smithi078 ceph-mon[91136]: pgmap v1189: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:32 smithi078 ceph-mon[99959]: pgmap v1189: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:33.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:33.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:33.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:34.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:34 smithi146 ceph-mon[103278]: pgmap v1190: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:34 smithi078 ceph-mon[99959]: pgmap v1190: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:34 smithi078 ceph-mon[91136]: pgmap v1190: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:36 smithi146 ceph-mon[103278]: pgmap v1191: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:36 smithi078 ceph-mon[91136]: pgmap v1191: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:36 smithi078 ceph-mon[99959]: pgmap v1191: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:38 smithi146 ceph-mon[103278]: pgmap v1192: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:38 smithi078 ceph-mon[99959]: pgmap v1192: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:38 smithi078 ceph-mon[91136]: pgmap v1192: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:40.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:40.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:40.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:40.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:40 smithi078 ceph-mon[99959]: pgmap v1193: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:40.999 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:59:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-20T20:59:40.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:40 smithi078 ceph-mon[91136]: pgmap v1193: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:40 smithi146 ceph-mon[103278]: pgmap v1193: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:43 smithi146 ceph-mon[103278]: pgmap v1194: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:43 smithi078 ceph-mon[91136]: pgmap v1194: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:43 smithi078 ceph-mon[99959]: pgmap v1194: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:43.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:43.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:43.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:43.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:43.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:43.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:45 smithi146 ceph-mon[103278]: pgmap v1195: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:45 smithi078 ceph-mon[91136]: pgmap v1195: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:45 smithi078 ceph-mon[99959]: pgmap v1195: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[91136]: pgmap v1196: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[99959]: pgmap v1196: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:46.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:46 smithi146 ceph-mon[103278]: pgmap v1196: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T20:59:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:48 smithi146 ceph-mon[103278]: pgmap v1197: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:48 smithi078 ceph-mon[91136]: pgmap v1197: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:48 smithi078 ceph-mon[99959]: pgmap v1197: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:50.246Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:50.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:50.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:50 smithi146 ceph-mon[103278]: pgmap v1198: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:50 smithi078 ceph-mon[91136]: pgmap v1198: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:50 smithi078 ceph-mon[99959]: pgmap v1198: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 20:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:20:59:50] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T20:59:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T20:59:53.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:52 smithi078 ceph-mon[99959]: pgmap v1199: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:53.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:52 smithi078 ceph-mon[91136]: pgmap v1199: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:53.003 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T20:59:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:52 smithi146 ceph-mon[103278]: pgmap v1199: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:53.529 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 50m - - 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 50m - - 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (44m) 12s ago 45m 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (45m) 7s ago 45m 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (38m) 12s ago 43m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (36m) 7s ago 49m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (38m) 12s ago 51m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (37m) 12s ago 51m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (35m) 7s ago 50m 83.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (36m) 12s ago 49m 89.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (45m) 12s ago 45m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T20:59:53.530 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (45m) 7s ago 45m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (34m) 12s ago 49m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (33m) 12s ago 48m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (33m) 12s ago 48m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (32m) 12s ago 48m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (32m) 7s ago 47m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (31m) 7s ago 47m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (30m) 7s ago 47m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (29m) 7s ago 46m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (38m) 7s ago 46m 63.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (29m) 12s ago 43m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (28m) 7s ago 43m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T20:59:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (28m) 12s ago 43m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T20:59:53.532 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (28m) 7s ago 43m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T20:59:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:53 smithi078 ceph-mon[91136]: from='client.66665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:53 smithi078 ceph-mon[99959]: from='client.66665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:53.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:53.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T20:59:53.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 20:59:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T20:59:53.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T20:59:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T20:59:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T20:59:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T20:59:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T20:59:53.848 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:59:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:53 smithi146 ceph-mon[103278]: from='client.66665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T20:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T20:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T20:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T20:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T20:59:54.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T20:59:54.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T20:59:54.426 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T20:59:54.426 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:54.426 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T20:59:54.426 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:54.426 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T20:59:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:54 smithi146 ceph-mon[103278]: from='client.46749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:54 smithi146 ceph-mon[103278]: from='client.66677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:54 smithi146 ceph-mon[103278]: pgmap v1200: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/150813582' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:54 smithi146 ceph-mon[103278]: from='client.66689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1528256648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[91136]: from='client.46749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[91136]: from='client.66677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[91136]: pgmap v1200: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/150813582' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[91136]: from='client.66689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1528256648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[99959]: from='client.46749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[99959]: from='client.66677 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[99959]: pgmap v1200: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/150813582' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T20:59:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[99959]: from='client.66689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T20:59:54.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1528256648' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T20:59:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:56 smithi146 ceph-mon[103278]: pgmap v1201: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:56 smithi078 ceph-mon[99959]: pgmap v1201: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:56 smithi078 ceph-mon[91136]: pgmap v1201: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:58 smithi146 ceph-mon[103278]: pgmap v1202: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 20:59:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:58 smithi078 ceph-mon[91136]: pgmap v1202: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 20:59:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T20:59:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:58 smithi078 ceph-mon[99959]: pgmap v1202: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T20:59:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 20:59:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T20:59:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:00.247Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:00.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:00.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:00.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:00.876 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:00:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:00:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:00 smithi146 ceph-mon[103278]: pgmap v1203: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:01.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-mon[91136]: pgmap v1203: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:01.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:01.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-mon[99959]: pgmap v1203: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:01.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:02 smithi146 ceph-mon[103278]: pgmap v1204: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:02 smithi078 ceph-mon[91136]: pgmap v1204: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:02 smithi078 ceph-mon[99959]: pgmap v1204: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:03.684Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:03.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:03.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:04 smithi146 ceph-mon[103278]: pgmap v1205: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:04 smithi078 ceph-mon[91136]: pgmap v1205: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:04 smithi078 ceph-mon[99959]: pgmap v1205: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:06 smithi146 ceph-mon[103278]: pgmap v1206: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[91136]: pgmap v1206: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[99959]: pgmap v1206: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:08 smithi146 ceph-mon[103278]: pgmap v1207: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:08 smithi078 ceph-mon[91136]: pgmap v1207: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:08 smithi078 ceph-mon[99959]: pgmap v1207: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:10.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:10.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:10.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:10 smithi146 ceph-mon[103278]: pgmap v1208: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:10 smithi078 ceph-mon[91136]: pgmap v1208: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:10 smithi078 ceph-mon[99959]: pgmap v1208: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:00:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:00:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:12 smithi146 ceph-mon[103278]: pgmap v1209: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:12 smithi078 ceph-mon[91136]: pgmap v1209: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:12 smithi078 ceph-mon[99959]: pgmap v1209: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:13.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:13.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:13.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:14.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:14 smithi146 ceph-mon[103278]: pgmap v1210: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:14 smithi078 ceph-mon[91136]: pgmap v1210: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:14 smithi078 ceph-mon[99959]: pgmap v1210: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:16.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:16 smithi146 ceph-mon[103278]: pgmap v1211: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:16 smithi078 ceph-mon[91136]: pgmap v1211: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:16 smithi078 ceph-mon[99959]: pgmap v1211: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:18 smithi146 ceph-mon[103278]: pgmap v1212: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:18 smithi078 ceph-mon[91136]: pgmap v1212: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:18 smithi078 ceph-mon[99959]: pgmap v1212: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:20.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:20.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:20.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:20 smithi146 ceph-mon[103278]: pgmap v1213: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:20 smithi078 ceph-mon[91136]: pgmap v1213: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:20 smithi078 ceph-mon[99959]: pgmap v1213: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:00:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T21:00:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:22.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:22 smithi146 ceph-mon[103278]: pgmap v1214: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:22 smithi078 ceph-mon[91136]: pgmap v1214: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:22 smithi078 ceph-mon[99959]: pgmap v1214: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:23.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:23.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:23.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:24.726 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:00:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:24 smithi146 ceph-mon[103278]: pgmap v1215: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:24 smithi078 ceph-mon[99959]: pgmap v1215: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:24 smithi078 ceph-mon[91136]: pgmap v1215: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:25.257 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:00:25.257 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 51m - - 2024-09-20T21:00:25.257 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 50m - - 2024-09-20T21:00:25.257 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (45m) 3s ago 46m 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:00:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (46m) 19s ago 46m 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:00:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (38m) 3s ago 43m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:00:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (37m) 19s ago 50m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:00:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (39m) 3s ago 51m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:00:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (37m) 3s ago 52m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:00:25.258 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (36m) 19s ago 50m 83.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:00:25.259 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (36m) 3s ago 50m 88.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:00:25.259 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (45m) 3s ago 45m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:00:25.259 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (45m) 19s ago 45m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:00:25.259 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (35m) 3s ago 49m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:00:25.259 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (34m) 3s ago 49m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:00:25.259 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (33m) 3s ago 49m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (33m) 3s ago 48m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (32m) 19s ago 48m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (31m) 19s ago 47m 57.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (30m) 19s ago 47m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (29m) 19s ago 47m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (38m) 19s ago 46m 64.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:00:25.260 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (29m) 3s ago 44m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:00:25.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (29m) 19s ago 44m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:00:25.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (29m) 3s ago 44m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:00:25.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (29m) 19s ago 44m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:00:25.575 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:00:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:00:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:00:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:00:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:00:25.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:25 smithi146 ceph-mon[103278]: from='client.66701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:25 smithi146 ceph-mon[103278]: from='client.66707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2350351642' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:25 smithi078 ceph-mon[91136]: from='client.66701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:25 smithi078 ceph-mon[91136]: from='client.66707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2350351642' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:25 smithi078 ceph-mon[99959]: from='client.66701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:25 smithi078 ceph-mon[99959]: from='client.66707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2350351642' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:26.182 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:26.182 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:26.182 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:26.183 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:26.183 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: from='client.46791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: pgmap v1216: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: from='client.66725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:26.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/679539626' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: from='client.46791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: pgmap v1216: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: from='client.66725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:26.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/679539626' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: from='client.46791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: pgmap v1216: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: from='client.66725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/679539626' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:28 smithi146 ceph-mon[103278]: pgmap v1217: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:28 smithi078 ceph-mon[91136]: pgmap v1217: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:28 smithi078 ceph-mon[99959]: pgmap v1217: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:30.250Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:30.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:30.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:30.877 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:00:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:00:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:30 smithi146 ceph-mon[103278]: pgmap v1218: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:30 smithi078 ceph-mon[91136]: pgmap v1218: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:30 smithi078 ceph-mon[99959]: pgmap v1218: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:32 smithi146 ceph-mon[103278]: pgmap v1219: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:32 smithi078 ceph-mon[91136]: pgmap v1219: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:32 smithi078 ceph-mon[99959]: pgmap v1219: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:33.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:33.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:33.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:35.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:34 smithi146 ceph-mon[103278]: pgmap v1220: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:34 smithi078 ceph-mon[91136]: pgmap v1220: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:34 smithi078 ceph-mon[99959]: pgmap v1220: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:37.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:36 smithi146 ceph-mon[103278]: pgmap v1221: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:36 smithi078 ceph-mon[91136]: pgmap v1221: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:36 smithi078 ceph-mon[99959]: pgmap v1221: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:38 smithi146 ceph-mon[103278]: pgmap v1222: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:38 smithi078 ceph-mon[91136]: pgmap v1222: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:38 smithi078 ceph-mon[99959]: pgmap v1222: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:40.250Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:40.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:40.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:40.901 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:00:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:00:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:40 smithi146 ceph-mon[103278]: pgmap v1223: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:40 smithi078 ceph-mon[91136]: pgmap v1223: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:40 smithi078 ceph-mon[99959]: pgmap v1223: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:43.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:42 smithi146 ceph-mon[103278]: pgmap v1224: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:42 smithi078 ceph-mon[99959]: pgmap v1224: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:42 smithi078 ceph-mon[91136]: pgmap v1224: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:43.688Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:43.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:43.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:44 smithi146 ceph-mon[103278]: pgmap v1225: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:44 smithi078 ceph-mon[99959]: pgmap v1225: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:44 smithi078 ceph-mon[91136]: pgmap v1225: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:46 smithi146 ceph-mon[103278]: pgmap v1226: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[91136]: pgmap v1226: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[99959]: pgmap v1226: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:00:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:48 smithi146 ceph-mon[103278]: pgmap v1227: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:48 smithi078 ceph-mon[91136]: pgmap v1227: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:48 smithi078 ceph-mon[99959]: pgmap v1227: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:50.251Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:50.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:50.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:50.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:00:50] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:00:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:50 smithi146 ceph-mon[103278]: pgmap v1228: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:50 smithi078 ceph-mon[91136]: pgmap v1228: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:50 smithi078 ceph-mon[99959]: pgmap v1228: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:00:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:00:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:52 smithi146 ceph-mon[103278]: pgmap v1229: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:52 smithi078 ceph-mon[91136]: pgmap v1229: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:52 smithi078 ceph-mon[99959]: pgmap v1229: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:00:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:53.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:53.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:00:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:00:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:00:53.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:00:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:55 smithi146 ceph-mon[103278]: pgmap v1230: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:00:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:55 smithi078 ceph-mon[91136]: pgmap v1230: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:00:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:55 smithi078 ceph-mon[99959]: pgmap v1230: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:00:56.476 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:00:57.014 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:00:57.014 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 51m - - 2024-09-20T21:00:57.014 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 51m - - 2024-09-20T21:00:57.014 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (45m) 15s ago 46m 30.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:00:57.014 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (46m) 11s ago 46m 87.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:00:57.014 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (39m) 15s ago 44m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (37m) 11s ago 50m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (39m) 15s ago 52m 544M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (38m) 15s ago 52m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (36m) 11s ago 51m 84.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (37m) 15s ago 51m 88.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (46m) 15s ago 46m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (46m) 11s ago 46m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (35m) 15s ago 50m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (34m) 15s ago 50m 65.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (34m) 15s ago 49m 59.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (33m) 15s ago 49m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (33m) 11s ago 48m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:00:57.015 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (32m) 11s ago 48m 57.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (31m) 11s ago 48m 61.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (30m) 11s ago 47m 56.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (39m) 11s ago 47m 66.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (30m) 15s ago 44m 86.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (29m) 11s ago 44m 87.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (30m) 15s ago 44m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:00:57.016 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (29m) 11s ago 44m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:00:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:57 smithi146 ceph-mon[103278]: pgmap v1231: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:00:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:57 smithi078 ceph-mon[99959]: pgmap v1231: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:00:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:57 smithi078 ceph-mon[91136]: pgmap v1231: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:00:57.324 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:00:57.324 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:00:57.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:00:57.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:00:57.325 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:00:57.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:00:57.596 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:00:57.943 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:00:57.943 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:00:57.943 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:00:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:58 smithi146 ceph-mon[103278]: from='client.66737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:58 smithi146 ceph-mon[103278]: from='client.46821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:58 smithi146 ceph-mon[103278]: from='client.46827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/744798189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2129993229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:58.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[99959]: from='client.66737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[99959]: from='client.46821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[99959]: from='client.46827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/744798189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2129993229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[91136]: from='client.66737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[91136]: from='client.46821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[91136]: from='client.46827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/744798189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:00:58.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2129993229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:00:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:59 smithi146 ceph-mon[103278]: from='client.46839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:59 smithi146 ceph-mon[103278]: pgmap v1232: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:00:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:59 smithi078 ceph-mon[91136]: from='client.46839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:59 smithi078 ceph-mon[91136]: pgmap v1232: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:00:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:00:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:59 smithi078 ceph-mon[99959]: from='client.46839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:00:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:59 smithi078 ceph-mon[99959]: pgmap v1232: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:00:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:00.252Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:00.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:00.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:01.013 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:01:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:01:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:01 smithi146 ceph-mon[103278]: pgmap v1233: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:01:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:01 smithi078 ceph-mon[91136]: pgmap v1233: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:01:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:01 smithi078 ceph-mon[99959]: pgmap v1233: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:01:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:03 smithi078 ceph-mon[91136]: pgmap v1234: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T21:01:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:03 smithi078 ceph-mon[99959]: pgmap v1234: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T21:01:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:03 smithi146 ceph-mon[103278]: pgmap v1234: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T21:01:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:03.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:03.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:03.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:05 smithi078 ceph-mon[91136]: pgmap v1235: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T21:01:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:05 smithi078 ceph-mon[99959]: pgmap v1235: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T21:01:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:05 smithi146 ceph-mon[103278]: pgmap v1235: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[99959]: pgmap v1236: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[91136]: pgmap v1236: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:07 smithi146 ceph-mon[103278]: pgmap v1236: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T21:01:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:09 smithi078 ceph-mon[91136]: pgmap v1237: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T21:01:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:09 smithi078 ceph-mon[99959]: pgmap v1237: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T21:01:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:09 smithi146 ceph-mon[103278]: pgmap v1237: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T21:01:10.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:10.252Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:10.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:10.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:10.793 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:01:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:01:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:11 smithi146 ceph-mon[103278]: pgmap v1238: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:01:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:11 smithi078 ceph-mon[91136]: pgmap v1238: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:01:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:11 smithi078 ceph-mon[99959]: pgmap v1238: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:01:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:12.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:13 smithi146 ceph-mon[103278]: pgmap v1239: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T21:01:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:13 smithi078 ceph-mon[91136]: pgmap v1239: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T21:01:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:13 smithi078 ceph-mon[99959]: pgmap v1239: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T21:01:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:13.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:13.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:13.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:15 smithi146 ceph-mon[103278]: pgmap v1240: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:01:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:15 smithi078 ceph-mon[91136]: pgmap v1240: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:15 smithi078 ceph-mon[99959]: pgmap v1240: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:01:17.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:17 smithi146 ceph-mon[103278]: pgmap v1241: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T21:01:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:17 smithi078 ceph-mon[91136]: pgmap v1241: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T21:01:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:17 smithi078 ceph-mon[99959]: pgmap v1241: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-20T21:01:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:19 smithi146 ceph-mon[103278]: pgmap v1242: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:01:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:19 smithi078 ceph-mon[91136]: pgmap v1242: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:01:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:19 smithi078 ceph-mon[99959]: pgmap v1242: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:01:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:20.252Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:20.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:20.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:01:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:01:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:21 smithi146 ceph-mon[103278]: pgmap v1243: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:01:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:21 smithi078 ceph-mon[91136]: pgmap v1243: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:01:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:21 smithi078 ceph-mon[99959]: pgmap v1243: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-20T21:01:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:23 smithi146 ceph-mon[103278]: pgmap v1244: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:01:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:23 smithi078 ceph-mon[99959]: pgmap v1244: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:01:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:23 smithi078 ceph-mon[91136]: pgmap v1244: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:01:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:23.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:23.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:23.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:25 smithi146 ceph-mon[103278]: pgmap v1245: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:01:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:25 smithi078 ceph-mon[99959]: pgmap v1245: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:01:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:25 smithi078 ceph-mon[91136]: pgmap v1245: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:01:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:27 smithi146 ceph-mon[103278]: pgmap v1246: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[99959]: pgmap v1246: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[91136]: pgmap v1246: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:28.235 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:01:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 52m - - 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 51m - - 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (46m) 7s ago 47m 30.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (47m) 2s ago 47m 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (39m) 7s ago 45m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (38m) 2s ago 51m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:01:28.793 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (40m) 7s ago 52m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (38m) 7s ago 53m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (37m) 2s ago 51m 85.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (37m) 7s ago 51m 90.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (46m) 7s ago 46m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (46m) 2s ago 46m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (36m) 7s ago 50m 65.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (35m) 7s ago 50m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (34m) 7s ago 50m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (34m) 7s ago 49m 65.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (33m) 2s ago 49m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (32m) 2s ago 49m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (31m) 2s ago 48m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (30m) 2s ago 48m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:01:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (39m) 2s ago 47m 65.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:01:28.795 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (30m) 7s ago 45m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:01:28.795 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (30m) 2s ago 45m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:01:28.795 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (30m) 7s ago 45m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:01:28.795 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (30m) 2s ago 45m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:01:29.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:01:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:01:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:29 smithi078 ceph-mon[99959]: pgmap v1247: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/911255238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:29 smithi078 ceph-mon[91136]: pgmap v1247: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/911255238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:29.402 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:01:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:01:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:01:29.403 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:01:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:29 smithi146 ceph-mon[103278]: pgmap v1247: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/911255238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:01:29.757 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:01:29.757 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:01:29.757 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:01:29.757 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:01:29.757 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:01:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[91136]: from='client.66773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[91136]: from='client.46857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[91136]: from='client.46863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3453740400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[99959]: from='client.66773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[99959]: from='client.46857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[99959]: from='client.46863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3453740400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:30.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:30.253Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:30.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:30.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:30.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:30.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:30 smithi146 ceph-mon[103278]: from='client.66773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:30 smithi146 ceph-mon[103278]: from='client.46857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:30 smithi146 ceph-mon[103278]: from='client.46863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3453740400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:01:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:01:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:01:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:31 smithi146 ceph-mon[103278]: from='client.66797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:31 smithi146 ceph-mon[103278]: pgmap v1248: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:31 smithi078 ceph-mon[91136]: from='client.66797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:31 smithi078 ceph-mon[91136]: pgmap v1248: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:31 smithi078 ceph-mon[99959]: from='client.66797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:01:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:31 smithi078 ceph-mon[99959]: pgmap v1248: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:33 smithi146 ceph-mon[103278]: pgmap v1249: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:33 smithi078 ceph-mon[99959]: pgmap v1249: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:33 smithi078 ceph-mon[91136]: pgmap v1249: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:33.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:33.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:33.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:35 smithi146 ceph-mon[103278]: pgmap v1250: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:35 smithi078 ceph-mon[99959]: pgmap v1250: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:35 smithi078 ceph-mon[91136]: pgmap v1250: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:37 smithi146 ceph-mon[103278]: pgmap v1251: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:37 smithi078 ceph-mon[91136]: pgmap v1251: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:37 smithi078 ceph-mon[99959]: pgmap v1251: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:39.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:39 smithi146 ceph-mon[103278]: pgmap v1252: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:39 smithi078 ceph-mon[91136]: pgmap v1252: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:39 smithi078 ceph-mon[99959]: pgmap v1252: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:40.254Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:40.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:40.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:01:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:01:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:41 smithi146 ceph-mon[103278]: pgmap v1253: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:41 smithi078 ceph-mon[91136]: pgmap v1253: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:41 smithi078 ceph-mon[99959]: pgmap v1253: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:43 smithi146 ceph-mon[103278]: pgmap v1254: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-mon[91136]: pgmap v1254: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-mon[99959]: pgmap v1254: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:44.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:43.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:43.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:43.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:44 smithi146 ceph-mon[103278]: pgmap v1255: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:44 smithi078 ceph-mon[91136]: pgmap v1255: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:44 smithi078 ceph-mon[99959]: pgmap v1255: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:46 smithi146 ceph-mon[103278]: pgmap v1256: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[91136]: pgmap v1256: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[99959]: pgmap v1256: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:01:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:48 smithi146 ceph-mon[103278]: pgmap v1257: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:48 smithi078 ceph-mon[91136]: pgmap v1257: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:48 smithi078 ceph-mon[99959]: pgmap v1257: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:50.255Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:50.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:50.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:50.825 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:50 smithi078 ceph-mon[99959]: pgmap v1258: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:50.825 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:01:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:01:50.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:50 smithi078 ceph-mon[91136]: pgmap v1258: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:50 smithi146 ceph-mon[103278]: pgmap v1258: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:01:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:52 smithi146 ceph-mon[103278]: pgmap v1259: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:52 smithi078 ceph-mon[91136]: pgmap v1259: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:52 smithi078 ceph-mon[99959]: pgmap v1259: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:53.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:53.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:01:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:01:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:01:53.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:01:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:54 smithi146 ceph-mon[103278]: pgmap v1260: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:54 smithi078 ceph-mon[91136]: pgmap v1260: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:54 smithi078 ceph-mon[99959]: pgmap v1260: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:01:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:56 smithi146 ceph-mon[103278]: pgmap v1261: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:56 smithi078 ceph-mon[91136]: pgmap v1261: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:56 smithi078 ceph-mon[99959]: pgmap v1261: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:58 smithi146 ceph-mon[103278]: pgmap v1262: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:58 smithi078 ceph-mon[91136]: pgmap v1262: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:01:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:58 smithi078 ceph-mon[99959]: pgmap v1262: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:01:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:00.060 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:02:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:00.256Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:00.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:00.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 52m - - 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 52m - - 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (46m) 18s ago 47m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (47m) 14s ago 47m 87.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (40m) 18s ago 45m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (38m) 14s ago 51m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (40m) 18s ago 53m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (39m) 18s ago 53m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (37m) 14s ago 52m 86.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (38m) 18s ago 52m 91.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (47m) 18s ago 47m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:02:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (47m) 14s ago 47m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (36m) 18s ago 51m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (35m) 18s ago 51m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (35m) 18s ago 50m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (34m) 18s ago 50m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (34m) 14s ago 49m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (33m) 14s ago 49m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (32m) 14s ago 49m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (31m) 14s ago 48m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (40m) 14s ago 48m 65.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (31m) 18s ago 45m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (30m) 14s ago 46m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (31m) 18s ago 45m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:02:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (30m) 14s ago 45m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:02:00.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:02:00.901 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:02:00.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-mon[99959]: pgmap v1263: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:00.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-mon[99959]: from='client.46887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:00.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:02:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:02:00.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-mon[91136]: pgmap v1263: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:00.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:00 smithi078 ceph-mon[91136]: from='client.46887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:00 smithi146 ceph-mon[103278]: pgmap v1263: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:00 smithi146 ceph-mon[103278]: from='client.46887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:02:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:02:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:02:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:02:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:02:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:02:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:02:01.170 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:02:01.477 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:02:01.477 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:01.477 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:01.477 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[91136]: from='client.66815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[91136]: from='client.46899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1669059151' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[91136]: from='client.46911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3369939096' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[99959]: from='client.66815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[99959]: from='client.46899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1669059151' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[99959]: from='client.46911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3369939096' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:01 smithi146 ceph-mon[103278]: from='client.66815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:01 smithi146 ceph-mon[103278]: from='client.46899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1669059151' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:01 smithi146 ceph-mon[103278]: from='client.46911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3369939096' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:03 smithi146 ceph-mon[103278]: pgmap v1264: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:02 smithi078 ceph-mon[91136]: pgmap v1264: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[99959]: pgmap v1264: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:03.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:03.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:03.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:05 smithi146 ceph-mon[103278]: pgmap v1265: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:05 smithi078 ceph-mon[91136]: pgmap v1265: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:05 smithi078 ceph-mon[99959]: pgmap v1265: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:07 smithi146 ceph-mon[103278]: pgmap v1266: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[91136]: pgmap v1266: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[99959]: pgmap v1266: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:09 smithi146 ceph-mon[103278]: pgmap v1267: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:09.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:09 smithi078 ceph-mon[99959]: pgmap v1267: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:09.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:09 smithi078 ceph-mon[91136]: pgmap v1267: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:10.256Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:10.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:10.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:10.844 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:02:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:02:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:11 smithi146 ceph-mon[103278]: pgmap v1268: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:11 smithi078 ceph-mon[91136]: pgmap v1268: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:11 smithi078 ceph-mon[99959]: pgmap v1268: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:13 smithi078 ceph-mon[91136]: pgmap v1269: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:13 smithi078 ceph-mon[99959]: pgmap v1269: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:13.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:13 smithi146 ceph-mon[103278]: pgmap v1269: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:14.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:13.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:14.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:13.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:14.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:13.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:14.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:15 smithi078 ceph-mon[91136]: pgmap v1270: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:15 smithi078 ceph-mon[99959]: pgmap v1270: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:15 smithi146 ceph-mon[103278]: pgmap v1270: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:17 smithi146 ceph-mon[103278]: pgmap v1271: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:17 smithi078 ceph-mon[91136]: pgmap v1271: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:17 smithi078 ceph-mon[99959]: pgmap v1271: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:19 smithi146 ceph-mon[103278]: pgmap v1272: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:19 smithi078 ceph-mon[91136]: pgmap v1272: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:19 smithi078 ceph-mon[99959]: pgmap v1272: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:20.257Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:20.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:20.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:02:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:02:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:21 smithi146 ceph-mon[103278]: pgmap v1273: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:21 smithi078 ceph-mon[99959]: pgmap v1273: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:21 smithi078 ceph-mon[91136]: pgmap v1273: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:22.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:22.392 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[91136]: pgmap v1274: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[99959]: pgmap v1274: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:23 smithi146 ceph-mon[103278]: pgmap v1274: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:23.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:23.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:23.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:25 smithi146 ceph-mon[103278]: pgmap v1275: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:25 smithi078 ceph-mon[91136]: pgmap v1275: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:25 smithi078 ceph-mon[99959]: pgmap v1275: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:27 smithi146 ceph-mon[103278]: pgmap v1276: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[99959]: pgmap v1276: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[91136]: pgmap v1276: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:29.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:29 smithi078 ceph-mon[91136]: pgmap v1277: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:02:29.479 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:29 smithi078 ceph-mon[99959]: pgmap v1277: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:02:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:29 smithi146 ceph-mon[103278]: pgmap v1277: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:02:30.512 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:30.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:30.512 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:30.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:30.512 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:30.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:30.783 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:02:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:02:31.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:31 smithi078 ceph-mon[91136]: pgmap v1278: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:31.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:31 smithi078 ceph-mon[99959]: pgmap v1278: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:31 smithi146 ceph-mon[103278]: pgmap v1278: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:31.767 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:02:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:02:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 53m - - 2024-09-20T21:02:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 52m - - 2024-09-20T21:02:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (47m) 10s ago 48m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:02:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (48m) 5s ago 48m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:02:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (40m) 10s ago 46m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (39m) 5s ago 52m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (41m) 10s ago 53m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (39m) 10s ago 54m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (38m) 5s ago 52m 87.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (38m) 10s ago 52m 92.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (47m) 10s ago 47m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (47m) 5s ago 47m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (37m) 10s ago 52m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (36m) 10s ago 51m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:02:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (35m) 10s ago 51m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (35m) 10s ago 50m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (34m) 5s ago 50m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (33m) 5s ago 50m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (32m) 5s ago 49m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (31m) 5s ago 49m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (40m) 5s ago 48m 67.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (31m) 10s ago 46m 87.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:02:32.302 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (31m) 5s ago 46m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:02:32.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (31m) 10s ago 46m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:02:32.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (31m) 5s ago 46m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:02:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:32.610 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:02:32.611 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:02:32.877 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:02:32.877 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:02:32.878 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:02:33.185 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:02:33.186 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:33.186 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:02:33.186 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:33.186 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: pgmap v1279: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: from='client.46923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: from='client.66851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: from='client.66857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3844435950' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: from='client.46947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3654561598' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: pgmap v1279: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: from='client.46923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: from='client.66851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: from='client.66857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3844435950' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: from='client.46947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3654561598' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: pgmap v1279: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: from='client.46923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: from='client.66851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: from='client.66857 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3844435950' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: from='client.46947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3654561598' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:02:34.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:33.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:33.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:33.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:34 smithi146 ceph-mon[103278]: pgmap v1280: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:34 smithi078 ceph-mon[91136]: pgmap v1280: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:34 smithi078 ceph-mon[99959]: pgmap v1280: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:36 smithi146 ceph-mon[103278]: pgmap v1281: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:36 smithi078 ceph-mon[91136]: pgmap v1281: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:36 smithi078 ceph-mon[99959]: pgmap v1281: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:38 smithi146 ceph-mon[103278]: pgmap v1282: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:38 smithi078 ceph-mon[91136]: pgmap v1282: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:38 smithi078 ceph-mon[99959]: pgmap v1282: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:40.259Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:40.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:40.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:40 smithi146 ceph-mon[103278]: pgmap v1283: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:40.870 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:40 smithi078 ceph-mon[91136]: pgmap v1283: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:40.871 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:40 smithi078 ceph-mon[99959]: pgmap v1283: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:40.871 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:02:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:02:42.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:42.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:42 smithi146 ceph-mon[103278]: pgmap v1284: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[91136]: pgmap v1284: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[99959]: pgmap v1284: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:43.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:43.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:43.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:44 smithi146 ceph-mon[103278]: pgmap v1285: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:44 smithi078 ceph-mon[91136]: pgmap v1285: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:44 smithi078 ceph-mon[99959]: pgmap v1285: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:46 smithi146 ceph-mon[103278]: pgmap v1286: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:46 smithi078 ceph-mon[91136]: pgmap v1286: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:46 smithi078 ceph-mon[99959]: pgmap v1286: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:02:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:48 smithi078 ceph-mon[91136]: pgmap v1287: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:48 smithi078 ceph-mon[99959]: pgmap v1287: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:48 smithi146 ceph-mon[103278]: pgmap v1287: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:50.389 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:50.259Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:50.390 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:50.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:50.390 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:50.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:02:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:02:51.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:50 smithi078 ceph-mon[99959]: pgmap v1288: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:51.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:50 smithi078 ceph-mon[91136]: pgmap v1288: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:50 smithi146 ceph-mon[103278]: pgmap v1288: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:52.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:52.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:02:52.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:52 smithi078 ceph-mon[99959]: pgmap v1289: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:52.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:52 smithi078 ceph-mon[91136]: pgmap v1289: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:52 smithi146 ceph-mon[103278]: pgmap v1289: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:53.701Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:53.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:02:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:02:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:02:53.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:02:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:54 smithi146 ceph-mon[103278]: pgmap v1290: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:54 smithi078 ceph-mon[91136]: pgmap v1290: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:54 smithi078 ceph-mon[99959]: pgmap v1290: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:02:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:56 smithi146 ceph-mon[103278]: pgmap v1291: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:56 smithi078 ceph-mon[91136]: pgmap v1291: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:56 smithi078 ceph-mon[99959]: pgmap v1291: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:58.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:58 smithi146 ceph-mon[103278]: pgmap v1292: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:58.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:02:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:58 smithi078 ceph-mon[99959]: pgmap v1292: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:02:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:02:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:58 smithi078 ceph-mon[91136]: pgmap v1292: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:02:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:02:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:00.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:00.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:00.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:00.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:00.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:00 smithi078 ceph-mon[91136]: pgmap v1293: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:00.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:00 smithi078 ceph-mon[99959]: pgmap v1293: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:00.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:03:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:03:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:00 smithi146 ceph-mon[103278]: pgmap v1293: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:02 smithi146 ceph-mon[103278]: pgmap v1294: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[99959]: pgmap v1294: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[91136]: pgmap v1294: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:03.472 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:03:03.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:03.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:03.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:03.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:03.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 53m - - 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 53m - - 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (47m) 1s ago 48m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (48m) 17s ago 48m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (41m) 1s ago 46m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:03:03.999 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (40m) 17s ago 52m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (41m) 1s ago 54m 545M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (40m) 1s ago 54m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (38m) 17s ago 53m 85.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (39m) 1s ago 53m 91.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (48m) 1s ago 48m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (48m) 17s ago 48m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (37m) 1s ago 52m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (36m) 1s ago 52m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (36m) 1s ago 51m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (35m) 1s ago 51m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (35m) 17s ago 50m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (34m) 17s ago 50m 59.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (33m) 17s ago 50m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:03:04.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (32m) 17s ago 49m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:03:04.001 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (41m) 17s ago 49m 67.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:03:04.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (32m) 1s ago 47m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:03:04.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (32m) 17s ago 47m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:03:04.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (32m) 1s ago 46m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:03:04.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (31m) 17s ago 46m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:03:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:04.310 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:03:04.310 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:03:04.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:03:04.310 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:03:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:03:04.579 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:03:04.889 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:03:04.890 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:03:04.890 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:03:04.890 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:03:04.890 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:03:04.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[99959]: from='client.46959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:04.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[99959]: pgmap v1295: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:04.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[99959]: from='client.46965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:04.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[99959]: from='client.46971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:04.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1119280811' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:04.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[91136]: from='client.46959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:04.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[91136]: pgmap v1295: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:04.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[91136]: from='client.46965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:04.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[91136]: from='client.46971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:04.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1119280811' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:04 smithi146 ceph-mon[103278]: from='client.46959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:04 smithi146 ceph-mon[103278]: pgmap v1295: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:04 smithi146 ceph-mon[103278]: from='client.46965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:04 smithi146 ceph-mon[103278]: from='client.46971 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1119280811' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:05 smithi146 ceph-mon[103278]: from='client.66905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/912317410' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:05 smithi078 ceph-mon[91136]: from='client.66905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/912317410' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:05 smithi078 ceph-mon[99959]: from='client.66905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/912317410' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:06 smithi146 ceph-mon[103278]: pgmap v1296: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:06 smithi078 ceph-mon[91136]: pgmap v1296: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:06 smithi078 ceph-mon[99959]: pgmap v1296: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:08 smithi146 ceph-mon[103278]: pgmap v1297: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:08 smithi078 ceph-mon[91136]: pgmap v1297: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:08 smithi078 ceph-mon[99959]: pgmap v1297: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:10.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:10.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:10.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:10.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:10.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:10.790 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:03:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:03:11.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:10 smithi078 ceph-mon[99959]: pgmap v1298: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:11.068 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:10 smithi078 ceph-mon[91136]: pgmap v1298: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:10 smithi146 ceph-mon[103278]: pgmap v1298: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:12.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:12.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:12 smithi146 ceph-mon[103278]: pgmap v1299: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:12 smithi078 ceph-mon[91136]: pgmap v1299: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:12 smithi078 ceph-mon[99959]: pgmap v1299: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:13.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:13.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:13.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:14 smithi146 ceph-mon[103278]: pgmap v1300: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:14 smithi078 ceph-mon[91136]: pgmap v1300: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:14 smithi078 ceph-mon[99959]: pgmap v1300: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:16 smithi146 ceph-mon[103278]: pgmap v1301: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:16 smithi078 ceph-mon[91136]: pgmap v1301: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:16 smithi078 ceph-mon[99959]: pgmap v1301: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:19.210 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:18 smithi146 ceph-mon[103278]: pgmap v1302: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:18 smithi078 ceph-mon[91136]: pgmap v1302: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:18 smithi078 ceph-mon[99959]: pgmap v1302: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:20.261Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:20.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:20.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:20.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:03:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:03:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:20 smithi146 ceph-mon[103278]: pgmap v1303: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:21.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:20 smithi078 ceph-mon[99959]: pgmap v1303: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:21.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:20 smithi078 ceph-mon[91136]: pgmap v1303: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:22 smithi146 ceph-mon[103278]: pgmap v1304: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[99959]: pgmap v1304: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[91136]: pgmap v1304: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:23.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:24.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:23.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:24.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:23.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:24 smithi146 ceph-mon[103278]: pgmap v1305: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:24 smithi078 ceph-mon[91136]: pgmap v1305: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:24 smithi078 ceph-mon[99959]: pgmap v1305: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:26 smithi146 ceph-mon[103278]: pgmap v1306: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:26 smithi078 ceph-mon[91136]: pgmap v1306: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:26 smithi078 ceph-mon[99959]: pgmap v1306: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:29 smithi146 ceph-mon[103278]: pgmap v1307: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:29 smithi078 ceph-mon[91136]: pgmap v1307: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:29 smithi078 ceph-mon[99959]: pgmap v1307: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:30.261Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:30.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:30.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:30.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:03:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:03:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:31 smithi078 ceph-mon[91136]: pgmap v1308: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:31 smithi078 ceph-mon[99959]: pgmap v1308: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:31 smithi146 ceph-mon[103278]: pgmap v1308: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:32.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:32.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:33 smithi146 ceph-mon[103278]: pgmap v1309: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:33.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:33 smithi078 ceph-mon[91136]: pgmap v1309: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:33.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:33 smithi078 ceph-mon[99959]: pgmap v1309: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:33.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:33.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:33.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:35.178 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:03:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:35 smithi146 ceph-mon[103278]: pgmap v1310: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:35 smithi078 ceph-mon[99959]: pgmap v1310: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:35 smithi078 ceph-mon[91136]: pgmap v1310: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:35.705 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:03:35.705 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 54m - - 2024-09-20T21:03:35.705 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 53m - - 2024-09-20T21:03:35.705 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (48m) 13s ago 49m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:03:35.705 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (49m) 8s ago 49m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:03:35.705 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (41m) 13s ago 47m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (40m) 8s ago 53m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (42m) 13s ago 54m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (40m) 13s ago 55m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (39m) 8s ago 53m 87.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (39m) 13s ago 53m 91.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (48m) 13s ago 48m 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (48m) 8s ago 48m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (38m) 13s ago 53m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (37m) 13s ago 52m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (36m) 13s ago 52m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (36m) 13s ago 51m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (35m) 8s ago 51m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (34m) 8s ago 51m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:03:35.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (33m) 8s ago 50m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:03:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (33m) 8s ago 50m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:03:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (41m) 8s ago 49m 64.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:03:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (32m) 13s ago 47m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:03:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (32m) 8s ago 47m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:03:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (32m) 13s ago 47m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:03:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (32m) 8s ago 47m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:03:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:03:36.014 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:03:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:36 smithi078 ceph-mon[99959]: from='client.46995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1744252184' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:36 smithi078 ceph-mon[91136]: from='client.46995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1744252184' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:36.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:03:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:03:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:03:36.286 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:03:36.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:03:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:36 smithi146 ceph-mon[103278]: from='client.46995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1744252184' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:03:36.594 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:03:36.595 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:03:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:03:36.595 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:03:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:03:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:37 smithi146 ceph-mon[103278]: from='client.47001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:37 smithi146 ceph-mon[103278]: pgmap v1311: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:37 smithi146 ceph-mon[103278]: from='client.47007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1604499939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[91136]: from='client.47001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[91136]: pgmap v1311: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[91136]: from='client.47007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1604499939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[99959]: from='client.47001 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[99959]: pgmap v1311: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[99959]: from='client.47007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1604499939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:03:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:38 smithi146 ceph-mon[103278]: from='client.47019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:38 smithi078 ceph-mon[91136]: from='client.47019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:38 smithi078 ceph-mon[99959]: from='client.47019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:03:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:39 smithi146 ceph-mon[103278]: pgmap v1312: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:39 smithi078 ceph-mon[99959]: pgmap v1312: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:39 smithi078 ceph-mon[91136]: pgmap v1312: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:40.262Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:40.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:40.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:40.916 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:03:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:03:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:41 smithi146 ceph-mon[103278]: pgmap v1313: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:41 smithi078 ceph-mon[91136]: pgmap v1313: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:41 smithi078 ceph-mon[99959]: pgmap v1313: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:43 smithi146 ceph-mon[103278]: pgmap v1314: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[91136]: pgmap v1314: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[99959]: pgmap v1314: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:43.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:43.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:43.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:45 smithi146 ceph-mon[103278]: pgmap v1315: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:45 smithi078 ceph-mon[91136]: pgmap v1315: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:45 smithi078 ceph-mon[99959]: pgmap v1315: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:47 smithi146 ceph-mon[103278]: pgmap v1316: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:47 smithi078 ceph-mon[99959]: pgmap v1316: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:47 smithi078 ceph-mon[91136]: pgmap v1316: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:03:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:49 smithi146 ceph-mon[103278]: pgmap v1317: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:49 smithi078 ceph-mon[91136]: pgmap v1317: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:49 smithi078 ceph-mon[99959]: pgmap v1317: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:50.263Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:50.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:50.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:50.922 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:03:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:03:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:51 smithi146 ceph-mon[103278]: pgmap v1318: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:51 smithi078 ceph-mon[91136]: pgmap v1318: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:51 smithi078 ceph-mon[99959]: pgmap v1318: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:52.500 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:52.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:03:53.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:53 smithi078 ceph-mon[99959]: pgmap v1319: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:53.436 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:53 smithi078 ceph-mon[91136]: pgmap v1319: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:53 smithi146 ceph-mon[103278]: pgmap v1319: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:53.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:53.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:53.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:53.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:03:53.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:03:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:03:53.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:03:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:55 smithi146 ceph-mon[103278]: pgmap v1320: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:55 smithi078 ceph-mon[91136]: pgmap v1320: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:55 smithi078 ceph-mon[99959]: pgmap v1320: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:03:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:57 smithi146 ceph-mon[103278]: pgmap v1321: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:57 smithi078 ceph-mon[91136]: pgmap v1321: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:57 smithi078 ceph-mon[99959]: pgmap v1321: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:03:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:03:59 smithi146 ceph-mon[103278]: pgmap v1322: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:03:59 smithi078 ceph-mon[91136]: pgmap v1322: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:03:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:03:59 smithi078 ceph-mon[99959]: pgmap v1322: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:00.264Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:00.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:00.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:00.925 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:04:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:04:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:01 smithi146 ceph-mon[103278]: pgmap v1323: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:01 smithi078 ceph-mon[91136]: pgmap v1323: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:01 smithi078 ceph-mon[99959]: pgmap v1323: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:03 smithi146 ceph-mon[103278]: pgmap v1324: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[91136]: pgmap v1324: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[99959]: pgmap v1324: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:03.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:03.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:03.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:05 smithi146 ceph-mon[103278]: pgmap v1325: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:05 smithi078 ceph-mon[91136]: pgmap v1325: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:05 smithi078 ceph-mon[99959]: pgmap v1325: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:06.889 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 54m - - 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 54m - - 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (48m) 4s ago 49m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (49m) 0s ago 49m 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (42m) 4s ago 47m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:04:07.419 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (41m) 0s ago 53m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (42m) 4s ago 55m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (41m) 4s ago 55m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (39m) 0s ago 54m 86.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (40m) 4s ago 54m 91.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (49m) 4s ago 49m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (49m) 0s ago 49m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (38m) 4s ago 53m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (37m) 4s ago 53m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (37m) 4s ago 52m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (36m) 4s ago 52m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (36m) 0s ago 52m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (35m) 0s ago 51m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (34m) 0s ago 51m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:04:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (33m) 0s ago 50m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:04:07.421 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (42m) 0s ago 50m 66.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:04:07.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (33m) 4s ago 48m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:04:07.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (33m) 0s ago 48m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:04:07.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (33m) 4s ago 48m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:04:07.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (32m) 0s ago 47m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:04:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:07 smithi146 ceph-mon[103278]: pgmap v1326: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:07 smithi078 ceph-mon[91136]: pgmap v1326: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:07 smithi078 ceph-mon[99959]: pgmap v1326: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:04:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:04:07.733 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:04:08.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:04:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:04:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:04:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:04:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:04:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:04:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:04:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:04:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:04:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:04:08.003 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:04:08.003 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:04:08.003 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:04:08.003 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='client.66953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='client.47037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='client.66965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: pgmap v1327: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2478677232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[91136]: from='client.47055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='client.66953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='client.47037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='client.66965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: pgmap v1327: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2478677232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:08 smithi078 ceph-mon[99959]: from='client.47055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.310 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:04:08.310 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:08.310 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='client.66953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='client.47037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='client.66965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: pgmap v1327: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2478677232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:08 smithi146 ceph-mon[103278]: from='client.47055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1947477588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1947477588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1947477588' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:10 smithi146 ceph-mon[103278]: pgmap v1328: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:10 smithi078 ceph-mon[91136]: pgmap v1328: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:10 smithi078 ceph-mon[99959]: pgmap v1328: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:10.265Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:10.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:10.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:10.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:04:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:04:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:12 smithi146 ceph-mon[103278]: pgmap v1329: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:12 smithi078 ceph-mon[91136]: pgmap v1329: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:12 smithi078 ceph-mon[99959]: pgmap v1329: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:13.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:13.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:13.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:14.314 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:14 smithi078 ceph-mon[91136]: pgmap v1330: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:14.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:14 smithi078 ceph-mon[99959]: pgmap v1330: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:14 smithi146 ceph-mon[103278]: pgmap v1330: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:16 smithi146 ceph-mon[103278]: pgmap v1331: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:16 smithi078 ceph-mon[91136]: pgmap v1331: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:17.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:16 smithi078 ceph-mon[99959]: pgmap v1331: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:18 smithi146 ceph-mon[103278]: pgmap v1332: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:18 smithi078 ceph-mon[91136]: pgmap v1332: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:18 smithi078 ceph-mon[99959]: pgmap v1332: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:20.265Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:20.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:20.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:20.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:20 smithi078 ceph-mon[91136]: pgmap v1333: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:20.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:20 smithi078 ceph-mon[99959]: pgmap v1333: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:20.941 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:04:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:04:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:20 smithi146 ceph-mon[103278]: pgmap v1333: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:22.867 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:22 smithi146 ceph-mon[103278]: pgmap v1334: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:22 smithi078 ceph-mon[91136]: pgmap v1334: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:22 smithi078 ceph-mon[99959]: pgmap v1334: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:23.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:24.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:23.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:24.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:23.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:24.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:24.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:24 smithi146 ceph-mon[103278]: pgmap v1335: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:24 smithi078 ceph-mon[91136]: pgmap v1335: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:24 smithi078 ceph-mon[99959]: pgmap v1335: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 21:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T21:04:25.848984692Z level=info msg="Completed cleanup jobs" duration=12.908092ms 2024-09-20T21:04:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:26 smithi146 ceph-mon[103278]: pgmap v1336: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:26 smithi078 ceph-mon[91136]: pgmap v1336: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:26 smithi078 ceph-mon[99959]: pgmap v1336: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:28 smithi146 ceph-mon[103278]: pgmap v1337: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[99959]: pgmap v1337: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[91136]: pgmap v1337: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:30.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:30.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:30.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:30.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:30 smithi078 ceph-mon[91136]: pgmap v1338: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:30.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:30 smithi078 ceph-mon[99959]: pgmap v1338: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:30.952 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:04:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:04:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:30 smithi146 ceph-mon[103278]: pgmap v1338: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:32 smithi146 ceph-mon[103278]: pgmap v1339: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:32 smithi078 ceph-mon[91136]: pgmap v1339: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:32 smithi078 ceph-mon[99959]: pgmap v1339: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:34.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:33.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:33.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:33.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:34 smithi146 ceph-mon[103278]: pgmap v1340: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:34 smithi078 ceph-mon[91136]: pgmap v1340: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:34 smithi078 ceph-mon[99959]: pgmap v1340: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:36 smithi146 ceph-mon[103278]: pgmap v1341: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:36 smithi078 ceph-mon[91136]: pgmap v1341: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:36 smithi078 ceph-mon[99959]: pgmap v1341: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:38.610 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:04:39.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:38 smithi146 ceph-mon[103278]: pgmap v1342: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:38 smithi078 ceph-mon[99959]: pgmap v1342: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:38 smithi078 ceph-mon[91136]: pgmap v1342: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 55m - - 2024-09-20T21:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 55m - - 2024-09-20T21:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (49m) 16s ago 50m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (50m) 11s ago 50m 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (42m) 16s ago 48m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (41m) 11s ago 54m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (43m) 16s ago 56m 546M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (41m) 16s ago 56m 132M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (40m) 11s ago 54m 87.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (40m) 16s ago 54m 91.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (49m) 16s ago 49m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (49m) 11s ago 49m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (39m) 16s ago 54m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (38m) 16s ago 53m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (37m) 16s ago 53m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (37m) 16s ago 52m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (36m) 11s ago 52m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (35m) 11s ago 52m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (35m) 11s ago 51m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (34m) 11s ago 51m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (42m) 11s ago 50m 67.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (33m) 16s ago 48m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (33m) 11s ago 48m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (33m) 16s ago 48m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (33m) 11s ago 48m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:04:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:39 smithi146 ceph-mon[103278]: from='client.66989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:39 smithi146 ceph-mon[103278]: from='client.47073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:39 smithi146 ceph-mon[103278]: from='client.47079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3594374252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:40.024 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:04:40.024 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:04:40.024 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[99959]: from='client.66989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[99959]: from='client.47073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[99959]: from='client.47079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3594374252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[91136]: from='client.66989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[91136]: from='client.47073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[91136]: from='client.47079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:40.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3594374252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:04:40.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:40.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:40.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:40.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:40.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:40.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:40.764 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:04:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:04:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:40 smithi146 ceph-mon[103278]: pgmap v1343: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:40 smithi146 ceph-mon[103278]: from='client.67013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1087605425' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-mon[99959]: pgmap v1343: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-mon[99959]: from='client.67013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1087605425' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-mon[91136]: pgmap v1343: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-mon[91136]: from='client.67013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:04:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1087605425' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:04:41.889 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:42 smithi078 ceph-mon[91136]: pgmap v1344: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:42 smithi078 ceph-mon[99959]: pgmap v1344: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:43.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:42 smithi146 ceph-mon[103278]: pgmap v1344: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:43.707Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:43.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:43.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:44.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:44.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:44.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:44.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:45 smithi146 ceph-mon[103278]: pgmap v1345: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:45 smithi078 ceph-mon[91136]: pgmap v1345: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:45 smithi078 ceph-mon[99959]: pgmap v1345: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:47 smithi146 ceph-mon[103278]: pgmap v1346: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:47 smithi078 ceph-mon[91136]: pgmap v1346: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:47 smithi078 ceph-mon[99959]: pgmap v1346: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:48 smithi146 ceph-mon[103278]: pgmap v1347: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[99959]: pgmap v1347: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:04:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:48 smithi078 ceph-mon[91136]: pgmap v1347: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:50.267Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:50.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:50.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:50.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:50 smithi078 ceph-mon[99959]: pgmap v1348: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:50.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:50 smithi078 ceph-mon[91136]: pgmap v1348: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:50.965 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:04:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:04:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:50 smithi146 ceph-mon[103278]: pgmap v1348: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:04:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:52 smithi146 ceph-mon[103278]: pgmap v1349: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:52 smithi078 ceph-mon[99959]: pgmap v1349: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:52 smithi078 ceph-mon[91136]: pgmap v1349: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:53.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:53.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:04:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:04:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:04:53.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:04:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:54 smithi146 ceph-mon[103278]: pgmap v1350: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:54 smithi078 ceph-mon[91136]: pgmap v1350: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:54 smithi078 ceph-mon[99959]: pgmap v1350: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:04:56.987 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:56 smithi078 ceph-mon[91136]: pgmap v1351: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:56.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:56 smithi078 ceph-mon[99959]: pgmap v1351: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:56 smithi146 ceph-mon[103278]: pgmap v1351: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:59.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:58 smithi078 ceph-mon[99959]: pgmap v1352: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:59.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:59.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:58 smithi078 ceph-mon[91136]: pgmap v1352: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:59.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:04:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:58 smithi146 ceph-mon[103278]: pgmap v1352: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:04:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:00.268Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:00.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:00.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:00.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:00 smithi078 ceph-mon[99959]: pgmap v1353: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:00.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:00 smithi078 ceph-mon[91136]: pgmap v1353: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:00.974 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:05:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:05:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:00 smithi146 ceph-mon[103278]: pgmap v1353: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:02.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:03.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:02 smithi146 ceph-mon[103278]: pgmap v1354: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:02 smithi078 ceph-mon[91136]: pgmap v1354: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:02 smithi078 ceph-mon[99959]: pgmap v1354: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:03.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:03.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:03.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:05 smithi146 ceph-mon[103278]: pgmap v1355: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:05.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:05 smithi078 ceph-mon[99959]: pgmap v1355: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:05.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:05 smithi078 ceph-mon[91136]: pgmap v1355: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:07 smithi146 ceph-mon[103278]: pgmap v1356: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:07 smithi078 ceph-mon[99959]: pgmap v1356: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:07 smithi078 ceph-mon[91136]: pgmap v1356: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:08 smithi146 ceph-mon[103278]: pgmap v1357: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:09.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[99959]: pgmap v1357: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[91136]: pgmap v1357: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:10.317 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:05:10.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:10.268Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:10.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:10.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:10.856 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:05:10.856 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 55m - - 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 55m - - 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (49m) 7s ago 51m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (50m) 3s ago 50m 87.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (43m) 7s ago 48m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (42m) 3s ago 54m 457M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (44m) 7s ago 56m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (42m) 7s ago 56m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (41m) 3s ago 55m 92.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (41m) 7s ago 55m 93.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (50m) 7s ago 50m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (50m) 3s ago 50m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (40m) 7s ago 54m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (39m) 7s ago 54m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:05:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (38m) 7s ago 53m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (37m) 7s ago 53m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (37m) 3s ago 53m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (36m) 3s ago 52m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (35m) 3s ago 52m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (34m) 3s ago 51m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (43m) 3s ago 51m 67.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (34m) 7s ago 49m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (34m) 3s ago 49m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (34m) 7s ago 49m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:05:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (34m) 3s ago 49m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:05:10.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:10 smithi078 ceph-mon[99959]: pgmap v1358: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:10.859 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:05:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:05:10.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:10 smithi078 ceph-mon[91136]: pgmap v1358: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:10 smithi146 ceph-mon[103278]: pgmap v1358: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:11.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:05:11.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:05:11.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:05:11.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:11.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:05:11.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:05:11.180 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:05:11.468 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:05:11.468 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:05:11.468 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:05:11.469 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:05:11.787 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:05:11.787 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:11.787 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:11.787 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:11.787 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:11 smithi146 ceph-mon[103278]: from='client.47103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:11 smithi146 ceph-mon[103278]: from='client.67031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:11 smithi146 ceph-mon[103278]: from='client.67037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1079542165' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[91136]: from='client.47103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[91136]: from='client.67031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[91136]: from='client.67037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1079542165' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[99959]: from='client.47103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[99959]: from='client.67031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[99959]: from='client.67037 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:12.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1079542165' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:12 smithi146 ceph-mon[103278]: from='client.47127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:12 smithi146 ceph-mon[103278]: pgmap v1359: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4174130437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:12 smithi078 ceph-mon[91136]: from='client.47127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:12 smithi078 ceph-mon[91136]: pgmap v1359: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4174130437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:12 smithi078 ceph-mon[99959]: from='client.47127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:12 smithi078 ceph-mon[99959]: pgmap v1359: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4174130437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:13.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:13.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:13.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:14 smithi078 ceph-mon[91136]: pgmap v1360: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:14 smithi078 ceph-mon[99959]: pgmap v1360: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:14 smithi146 ceph-mon[103278]: pgmap v1360: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:16 smithi078 ceph-mon[91136]: pgmap v1361: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:16 smithi078 ceph-mon[99959]: pgmap v1361: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:16 smithi146 ceph-mon[103278]: pgmap v1361: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:18 smithi078 ceph-mon[91136]: pgmap v1362: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:18 smithi078 ceph-mon[99959]: pgmap v1362: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:18 smithi146 ceph-mon[103278]: pgmap v1362: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:20.269Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:20.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:20.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:20.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:20 smithi078 ceph-mon[99959]: pgmap v1363: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:20.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:20 smithi078 ceph-mon[91136]: pgmap v1363: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:20.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:05:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:05:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:20 smithi146 ceph-mon[103278]: pgmap v1363: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:22.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:22 smithi146 ceph-mon[103278]: pgmap v1364: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:22 smithi078 ceph-mon[91136]: pgmap v1364: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:22 smithi078 ceph-mon[99959]: pgmap v1364: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:23.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:23.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:23.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:24 smithi146 ceph-mon[103278]: pgmap v1365: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[99959]: pgmap v1365: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:24 smithi078 ceph-mon[91136]: pgmap v1365: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:26 smithi146 ceph-mon[103278]: pgmap v1366: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:26 smithi078 ceph-mon[99959]: pgmap v1366: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:26 smithi078 ceph-mon[91136]: pgmap v1366: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:28 smithi146 ceph-mon[103278]: pgmap v1367: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[99959]: pgmap v1367: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[91136]: pgmap v1367: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:29.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:30.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:30.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:30.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:30.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:30 smithi078 ceph-mon[99959]: pgmap v1368: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:30.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:30 smithi078 ceph-mon[91136]: pgmap v1368: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:30.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:05:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:05:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:30 smithi146 ceph-mon[103278]: pgmap v1368: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:32 smithi146 ceph-mon[103278]: pgmap v1369: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:32 smithi078 ceph-mon[99959]: pgmap v1369: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:32 smithi078 ceph-mon[91136]: pgmap v1369: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:33.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:33.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:33.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:34 smithi146 ceph-mon[103278]: pgmap v1370: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:35.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:34 smithi078 ceph-mon[99959]: pgmap v1370: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:35.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:34 smithi078 ceph-mon[91136]: pgmap v1370: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:36 smithi078 ceph-mon[91136]: pgmap v1371: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:36 smithi078 ceph-mon[99959]: pgmap v1371: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:36 smithi146 ceph-mon[103278]: pgmap v1371: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:38 smithi146 ceph-mon[103278]: pgmap v1372: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:38 smithi078 ceph-mon[99959]: pgmap v1372: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:38 smithi078 ceph-mon[91136]: pgmap v1372: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:40.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:40.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:40.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:40.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:40.847 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:05:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:05:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:40 smithi146 ceph-mon[103278]: pgmap v1373: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:40 smithi078 ceph-mon[91136]: pgmap v1373: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:40 smithi078 ceph-mon[99959]: pgmap v1373: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:42.087 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:05:42.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 56m - - 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 56m - - 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (50m) 19s ago 51m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (51m) 14s ago 51m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (43m) 19s ago 49m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (42m) 14s ago 55m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (44m) 19s ago 57m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (42m) 19s ago 57m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (41m) 14s ago 55m 89.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:05:42.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (41m) 19s ago 55m 93.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (51m) 19s ago 51m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (50m) 14s ago 50m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (40m) 19s ago 55m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (39m) 19s ago 54m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (38m) 19s ago 54m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (38m) 19s ago 54m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (37m) 14s ago 53m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (36m) 14s ago 53m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (36m) 14s ago 52m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (35m) 14s ago 52m 58.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:05:42.613 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (44m) 14s ago 51m 67.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:05:42.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (34m) 19s ago 49m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:05:42.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (34m) 14s ago 49m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:05:42.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (34m) 19s ago 49m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:05:42.614 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (34m) 14s ago 49m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:05:42.926 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:05:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:05:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:05:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:05:42.927 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:05:42.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:42 smithi078 ceph-mon[99959]: pgmap v1374: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:42.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:42 smithi078 ceph-mon[99959]: from='client.67061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:42.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:42 smithi078 ceph-mon[91136]: pgmap v1374: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:42.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:42 smithi078 ceph-mon[91136]: from='client.67061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:43.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:42 smithi146 ceph-mon[103278]: pgmap v1374: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:43.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:42 smithi146 ceph-mon[103278]: from='client.67061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:05:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:05:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:05:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:05:43.199 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:05:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:05:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:05:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='client.47145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='client.67073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/902356962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='client.47163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4064621477' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:44.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:43.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:44.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:43.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:44.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:43.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:44.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='client.47145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='client.67073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/902356962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:44.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:44.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='client.47163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4064621477' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='client.47145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='client.67073 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/902356962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='client.47163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:44.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4064621477' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:05:45.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:44 smithi146 ceph-mon[103278]: pgmap v1375: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:44 smithi078 ceph-mon[99959]: pgmap v1375: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:44 smithi078 ceph-mon[91136]: pgmap v1375: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:46 smithi146 ceph-mon[103278]: pgmap v1376: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:46 smithi078 ceph-mon[91136]: pgmap v1376: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:46 smithi078 ceph-mon[99959]: pgmap v1376: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:48 smithi146 ceph-mon[103278]: pgmap v1377: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[99959]: pgmap v1377: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[91136]: pgmap v1377: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:05:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:50.272Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:50.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:50.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:50.961 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:05:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:05:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:50 smithi146 ceph-mon[103278]: pgmap v1378: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:50 smithi078 ceph-mon[91136]: pgmap v1378: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:50 smithi078 ceph-mon[99959]: pgmap v1378: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:05:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:52 smithi146 ceph-mon[103278]: pgmap v1379: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:52 smithi078 ceph-mon[91136]: pgmap v1379: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:52 smithi078 ceph-mon[99959]: pgmap v1379: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:53.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:53.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:05:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:05:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:05:53.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:05:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:54 smithi146 ceph-mon[103278]: pgmap v1380: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:54 smithi078 ceph-mon[91136]: pgmap v1380: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:54 smithi078 ceph-mon[99959]: pgmap v1380: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:05:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:56 smithi078 ceph-mon[91136]: pgmap v1381: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:56 smithi078 ceph-mon[99959]: pgmap v1381: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:56 smithi146 ceph-mon[103278]: pgmap v1381: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:58 smithi146 ceph-mon[103278]: pgmap v1382: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:05:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:58 smithi078 ceph-mon[91136]: pgmap v1382: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:05:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:05:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:58 smithi078 ceph-mon[99959]: pgmap v1382: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:05:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:05:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:00.273Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:00.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:00.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:00.985 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:06:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:06:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:00 smithi146 ceph-mon[103278]: pgmap v1383: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:00 smithi078 ceph-mon[91136]: pgmap v1383: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:00 smithi078 ceph-mon[99959]: pgmap v1383: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:02 smithi146 ceph-mon[103278]: pgmap v1384: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:02 smithi078 ceph-mon[91136]: pgmap v1384: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:02 smithi078 ceph-mon[99959]: pgmap v1384: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:03.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:03.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:03.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:04.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:04.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:04.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:04.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:04 smithi146 ceph-mon[103278]: pgmap v1385: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[91136]: pgmap v1385: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:04 smithi078 ceph-mon[99959]: pgmap v1385: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:06 smithi078 ceph-mon[91136]: pgmap v1386: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:06 smithi078 ceph-mon[99959]: pgmap v1386: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:06 smithi146 ceph-mon[103278]: pgmap v1386: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:09 smithi146 ceph-mon[103278]: pgmap v1387: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[99959]: pgmap v1387: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[91136]: pgmap v1387: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:10.274Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:10.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:10.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:10.894 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:06:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:06:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:11 smithi078 ceph-mon[91136]: pgmap v1388: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:11 smithi078 ceph-mon[99959]: pgmap v1388: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:11 smithi146 ceph-mon[103278]: pgmap v1388: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:12.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:13 smithi146 ceph-mon[103278]: pgmap v1389: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:13 smithi078 ceph-mon[99959]: pgmap v1389: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:13 smithi078 ceph-mon[91136]: pgmap v1389: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:13.821 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:06:14.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:13.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:13.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:13.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:14.347 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:06:14.347 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 56m - - 2024-09-20T21:06:14.347 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 56m - - 2024-09-20T21:06:14.347 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (51m) 10s ago 52m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:06:14.347 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (51m) 6s ago 51m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:06:14.347 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (44m) 10s ago 49m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (43m) 6s ago 56m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (45m) 10s ago 57m 547M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (43m) 10s ago 57m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (42m) 6s ago 56m 90.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (42m) 10s ago 56m 94.6M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (51m) 10s ago 51m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (51m) 6s ago 51m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (41m) 10s ago 55m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (40m) 10s ago 55m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (39m) 10s ago 54m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (38m) 10s ago 54m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (38m) 6s ago 54m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (37m) 6s ago 53m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (36m) 6s ago 53m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:06:14.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (35m) 6s ago 52m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:06:14.349 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (44m) 6s ago 52m 67.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:06:14.349 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (35m) 10s ago 50m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:06:14.349 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (35m) 6s ago 50m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:06:14.349 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (35m) 10s ago 50m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:06:14.349 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (35m) 6s ago 50m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:06:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:14.654 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:14.655 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:06:14.656 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:06:14.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:06:14.656 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:06:14.925 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:06:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:06:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:06:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:06:14.925 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:06:14.926 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:06:14.927 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:06:15.234 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:06:15.234 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:15.234 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[99959]: pgmap v1390: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[99959]: from='client.47175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[99959]: from='client.47181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3505597014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[91136]: pgmap v1390: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[91136]: from='client.47175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[91136]: from='client.47181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:15.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3505597014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:15 smithi146 ceph-mon[103278]: pgmap v1390: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:15 smithi146 ceph-mon[103278]: from='client.47175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:15 smithi146 ceph-mon[103278]: from='client.47181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3505597014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:16 smithi146 ceph-mon[103278]: from='client.67109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:16 smithi146 ceph-mon[103278]: from='client.67121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/388693550' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:16 smithi078 ceph-mon[91136]: from='client.67109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:16 smithi078 ceph-mon[91136]: from='client.67121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/388693550' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:16 smithi078 ceph-mon[99959]: from='client.67109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:16 smithi078 ceph-mon[99959]: from='client.67121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/388693550' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:17 smithi078 ceph-mon[91136]: pgmap v1391: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:17 smithi078 ceph-mon[99959]: pgmap v1391: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:17 smithi146 ceph-mon[103278]: pgmap v1391: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:19.146 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:19 smithi078 ceph-mon[91136]: pgmap v1392: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:19.146 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:19 smithi078 ceph-mon[99959]: pgmap v1392: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:19 smithi146 ceph-mon[103278]: pgmap v1392: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:20.275Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:20.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:20.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:06:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:06:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:21 smithi146 ceph-mon[103278]: pgmap v1393: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:21 smithi078 ceph-mon[99959]: pgmap v1393: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:21 smithi078 ceph-mon[91136]: pgmap v1393: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:23 smithi146 ceph-mon[103278]: pgmap v1394: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:23 smithi078 ceph-mon[91136]: pgmap v1394: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:23 smithi078 ceph-mon[99959]: pgmap v1394: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:23.716Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:23.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:23.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:24 smithi146 ceph-mon[103278]: pgmap v1395: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[91136]: pgmap v1395: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:25.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:24 smithi078 ceph-mon[99959]: pgmap v1395: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:26 smithi146 ceph-mon[103278]: pgmap v1396: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:26 smithi078 ceph-mon[99959]: pgmap v1396: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:26 smithi078 ceph-mon[91136]: pgmap v1396: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[99959]: pgmap v1397: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[91136]: pgmap v1397: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:28 smithi146 ceph-mon[103278]: pgmap v1397: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:30.276Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:30.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:30.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:30 smithi078 ceph-mon[91136]: pgmap v1398: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:30 smithi078 ceph-mon[99959]: pgmap v1398: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:06:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:06:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:30 smithi146 ceph-mon[103278]: pgmap v1398: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:32 smithi146 ceph-mon[103278]: pgmap v1399: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:32 smithi078 ceph-mon[91136]: pgmap v1399: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:32 smithi078 ceph-mon[99959]: pgmap v1399: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:33.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:33.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:33.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:34 smithi146 ceph-mon[103278]: pgmap v1400: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:34 smithi078 ceph-mon[99959]: pgmap v1400: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:34 smithi078 ceph-mon[91136]: pgmap v1400: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:36 smithi146 ceph-mon[103278]: pgmap v1401: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:37.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:36 smithi078 ceph-mon[91136]: pgmap v1401: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:37.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:36 smithi078 ceph-mon[99959]: pgmap v1401: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:38.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:38 smithi078 ceph-mon[91136]: pgmap v1402: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:06:38.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:38 smithi078 ceph-mon[99959]: pgmap v1402: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:06:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:38 smithi146 ceph-mon[103278]: pgmap v1402: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:06:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:40.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:40.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:40.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:40 smithi078 ceph-mon[91136]: pgmap v1403: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:40 smithi078 ceph-mon[99959]: pgmap v1403: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:06:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:06:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:40 smithi146 ceph-mon[103278]: pgmap v1403: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:43.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:42 smithi146 ceph-mon[103278]: pgmap v1404: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:42 smithi078 ceph-mon[91136]: pgmap v1404: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:42 smithi078 ceph-mon[99959]: pgmap v1404: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:43.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:43.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:43.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:44 smithi146 ceph-mon[103278]: pgmap v1405: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[99959]: pgmap v1405: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[91136]: pgmap v1405: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:45.529 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 57m - - 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 57m - - 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (51m) 2s ago 52m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (52m) 17s ago 52m 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (44m) 2s ago 50m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (43m) 17s ago 56m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (45m) 2s ago 58m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (43m) 2s ago 58m 137M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (42m) 17s ago 56m 90.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:06:46.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (42m) 2s ago 56m 96.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (52m) 2s ago 52m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (52m) 17s ago 52m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (41m) 2s ago 56m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (40m) 2s ago 55m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (40m) 2s ago 55m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (39m) 2s ago 55m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (38m) 17s ago 54m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (37m) 17s ago 54m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (37m) 17s ago 53m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (36m) 17s ago 53m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (45m) 17s ago 52m 67.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (35m) 2s ago 50m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (35m) 17s ago 50m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:06:46.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (35m) 2s ago 50m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:06:46.054 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (35m) 17s ago 50m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:06:46.359 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:06:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:06:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:06:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:06:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:06:46.361 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:06:46.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:06:46.629 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:06:46.629 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:06:46.630 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:06:46.939 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:06:46.939 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:46.939 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:06:46.939 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:46.939 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[99959]: from='client.47211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[99959]: pgmap v1406: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[99959]: from='client.67139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[99959]: from='client.67145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2605018049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[91136]: from='client.47211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[91136]: pgmap v1406: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[91136]: from='client.67139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[91136]: from='client.67145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:46.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2605018049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:46 smithi146 ceph-mon[103278]: from='client.47211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:46 smithi146 ceph-mon[103278]: pgmap v1406: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:46 smithi146 ceph-mon[103278]: from='client.67139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:46 smithi146 ceph-mon[103278]: from='client.67145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2605018049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:06:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:47 smithi146 ceph-mon[103278]: from='client.47235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4092813136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:47 smithi078 ceph-mon[91136]: from='client.47235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4092813136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:47 smithi078 ceph-mon[99959]: from='client.47235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:06:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4092813136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:06:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:48 smithi146 ceph-mon[103278]: pgmap v1407: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[91136]: pgmap v1407: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[99959]: pgmap v1407: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:06:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:50.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:50.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:50.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:50 smithi078 ceph-mon[91136]: pgmap v1408: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:50 smithi078 ceph-mon[99959]: pgmap v1408: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:06:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:06:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:50 smithi146 ceph-mon[103278]: pgmap v1408: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:06:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:52 smithi146 ceph-mon[103278]: pgmap v1409: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:52 smithi078 ceph-mon[91136]: pgmap v1409: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:52 smithi078 ceph-mon[99959]: pgmap v1409: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:53.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:53.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:06:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:06:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:06:53.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:06:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:54 smithi146 ceph-mon[103278]: pgmap v1410: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:54 smithi078 ceph-mon[91136]: pgmap v1410: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:54 smithi078 ceph-mon[99959]: pgmap v1410: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:06:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:56 smithi146 ceph-mon[103278]: pgmap v1411: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:56 smithi078 ceph-mon[91136]: pgmap v1411: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:56 smithi078 ceph-mon[99959]: pgmap v1411: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:58 smithi078 ceph-mon[91136]: pgmap v1412: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:58 smithi078 ceph-mon[99959]: pgmap v1412: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:06:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:58 smithi146 ceph-mon[103278]: pgmap v1412: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:06:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:00.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:00.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:00.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:00.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:07:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T21:07:01.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:00 smithi078 ceph-mon[91136]: pgmap v1413: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:01.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:00 smithi078 ceph-mon[99959]: pgmap v1413: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:00 smithi146 ceph-mon[103278]: pgmap v1413: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:02 smithi146 ceph-mon[103278]: pgmap v1414: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:02 smithi078 ceph-mon[91136]: pgmap v1414: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:02 smithi078 ceph-mon[99959]: pgmap v1414: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:03.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:03.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:03.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:03.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:03.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:03.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:04 smithi146 ceph-mon[103278]: pgmap v1415: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[91136]: pgmap v1415: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[99959]: pgmap v1415: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:06 smithi146 ceph-mon[103278]: pgmap v1416: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:06 smithi078 ceph-mon[99959]: pgmap v1416: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:06 smithi078 ceph-mon[91136]: pgmap v1416: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:08 smithi146 ceph-mon[103278]: pgmap v1417: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[91136]: pgmap v1417: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[99959]: pgmap v1417: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:10.279Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:10.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:10.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:10.973 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:07:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T21:07:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:10 smithi146 ceph-mon[103278]: pgmap v1418: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:10 smithi078 ceph-mon[91136]: pgmap v1418: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:10 smithi078 ceph-mon[99959]: pgmap v1418: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:12 smithi146 ceph-mon[103278]: pgmap v1419: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:12 smithi078 ceph-mon[91136]: pgmap v1419: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:12 smithi078 ceph-mon[99959]: pgmap v1419: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:13.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:13.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:13.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:13.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:13.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:13.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:14 smithi146 ceph-mon[103278]: pgmap v1420: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:14 smithi078 ceph-mon[91136]: pgmap v1420: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:14 smithi078 ceph-mon[99959]: pgmap v1420: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:07:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:16 smithi146 ceph-mon[103278]: pgmap v1421: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:16 smithi078 ceph-mon[99959]: pgmap v1421: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:16 smithi078 ceph-mon[91136]: pgmap v1421: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:17.766 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:07:17.766 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 57m - - 2024-09-20T21:07:17.766 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 57m - - 2024-09-20T21:07:17.766 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (52m) 13s ago 53m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (52m) 9s ago 52m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (45m) 13s ago 50m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (44m) 9s ago 57m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (46m) 13s ago 58m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (44m) 13s ago 58m 137M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (43m) 9s ago 57m 92.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (43m) 13s ago 57m 96.7M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (52m) 13s ago 52m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (52m) 9s ago 52m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (42m) 13s ago 56m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (41m) 13s ago 56m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (40m) 13s ago 56m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (40m) 13s ago 55m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:07:17.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (39m) 9s ago 55m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (38m) 9s ago 54m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (37m) 9s ago 54m 62.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (36m) 9s ago 54m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (45m) 9s ago 53m 67.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (36m) 13s ago 51m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (36m) 9s ago 51m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (36m) 13s ago 51m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:07:17.768 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (36m) 9s ago 51m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:07:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:18 smithi078 ceph-mon[91136]: from='client.67169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:18 smithi078 ceph-mon[99959]: from='client.67169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:18.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:07:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:07:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:07:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:07:18.079 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:07:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:18 smithi146 ceph-mon[103278]: from='client.67169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:07:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:07:18.347 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:07:18.347 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:07:18.347 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:07:18.347 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:07:18.671 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:07:18.671 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:18.671 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:19 smithi146 ceph-mon[103278]: from='client.47253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:19 smithi146 ceph-mon[103278]: pgmap v1422: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:19 smithi146 ceph-mon[103278]: from='client.67181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2559368534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/793210528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[99959]: from='client.47253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[99959]: pgmap v1422: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[99959]: from='client.67181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2559368534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/793210528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[91136]: from='client.47253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[91136]: pgmap v1422: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[91136]: from='client.67181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2559368534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/793210528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:20.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:20 smithi078 ceph-mon[99959]: from='client.47271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:20.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:20 smithi078 ceph-mon[91136]: from='client.47271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:20 smithi146 ceph-mon[103278]: from='client.47271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:20.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:20.279Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:20.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:20.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:20.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:21.010 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:07:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:07:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:21 smithi146 ceph-mon[103278]: pgmap v1423: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:21 smithi078 ceph-mon[91136]: pgmap v1423: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:21 smithi078 ceph-mon[99959]: pgmap v1423: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:23 smithi078 ceph-mon[91136]: pgmap v1424: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:23 smithi078 ceph-mon[99959]: pgmap v1424: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:23 smithi146 ceph-mon[103278]: pgmap v1424: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:07:24.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:23.720Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:23.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:23.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:25 smithi146 ceph-mon[103278]: pgmap v1425: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[99959]: pgmap v1425: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[91136]: pgmap v1425: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:27.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:27 smithi146 ceph-mon[103278]: pgmap v1426: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:27 smithi078 ceph-mon[91136]: pgmap v1426: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:27 smithi078 ceph-mon[99959]: pgmap v1426: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:29 smithi146 ceph-mon[103278]: pgmap v1427: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[91136]: pgmap v1427: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[99959]: pgmap v1427: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:30.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:30.279Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:30.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:30.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:07:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:07:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:31 smithi146 ceph-mon[103278]: pgmap v1428: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:31 smithi078 ceph-mon[91136]: pgmap v1428: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:31 smithi078 ceph-mon[99959]: pgmap v1428: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:33 smithi146 ceph-mon[103278]: pgmap v1429: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:33 smithi078 ceph-mon[99959]: pgmap v1429: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:33 smithi078 ceph-mon[91136]: pgmap v1429: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:33.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:33.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:33.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:35 smithi146 ceph-mon[103278]: pgmap v1430: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:35 smithi078 ceph-mon[91136]: pgmap v1430: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:35 smithi078 ceph-mon[99959]: pgmap v1430: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:37 smithi146 ceph-mon[103278]: pgmap v1431: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:37 smithi078 ceph-mon[91136]: pgmap v1431: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:37 smithi078 ceph-mon[99959]: pgmap v1431: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:39.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:39 smithi078 ceph-mon[91136]: pgmap v1432: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:39.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:39 smithi078 ceph-mon[99959]: pgmap v1432: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:39 smithi146 ceph-mon[103278]: pgmap v1432: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:40.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:40.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:40.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:40.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:07:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:07:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:41 smithi078 ceph-mon[91136]: pgmap v1433: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:41 smithi078 ceph-mon[99959]: pgmap v1433: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:41 smithi146 ceph-mon[103278]: pgmap v1433: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:43 smithi146 ceph-mon[103278]: pgmap v1434: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:43 smithi078 ceph-mon[91136]: pgmap v1434: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:43 smithi078 ceph-mon[99959]: pgmap v1434: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:43.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:43.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:43.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:45 smithi146 ceph-mon[103278]: pgmap v1435: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[91136]: pgmap v1435: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[99959]: pgmap v1435: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:47 smithi146 ceph-mon[103278]: pgmap v1436: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:47 smithi078 ceph-mon[91136]: pgmap v1436: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:47 smithi078 ceph-mon[99959]: pgmap v1436: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:48.970 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 58m - - 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 58m - - 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (52m) 5s ago 53m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (53m) 0s ago 53m 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (45m) 5s ago 51m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:07:49.497 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (44m) 0s ago 57m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (46m) 5s ago 59m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (44m) 5s ago 59m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (43m) 0s ago 58m 93.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (44m) 5s ago 57m 98.1M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (53m) 5s ago 53m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (53m) 0s ago 53m 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (42m) 5s ago 57m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (41m) 5s ago 56m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (41m) 5s ago 56m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (40m) 5s ago 56m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (40m) 0s ago 55m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (38m) 0s ago 55m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:07:49.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (38m) 0s ago 54m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:07:49.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (37m) 0s ago 54m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:07:49.499 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (46m) 0s ago 53m 64.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:07:49.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (36m) 5s ago 51m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:07:49.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (36m) 0s ago 51m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:07:49.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (36m) 5s ago 51m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:07:49.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (36m) 0s ago 51m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:07:49.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[99959]: pgmap v1437: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:49.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[91136]: pgmap v1437: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:49.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:49 smithi146 ceph-mon[103278]: pgmap v1437: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:07:49.810 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:07:49.811 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:07:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:07:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:07:49.812 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:07:50.080 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:07:50.080 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:07:50.080 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:07:50.080 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:07:50.080 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:07:50.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:07:50.081 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:07:50.394 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:07:50.394 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:50.394 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:07:50.394 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:50.394 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:07:50.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-mon[99959]: from='client.47283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:50.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-mon[99959]: from='client.47289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:50.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1263368686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:50.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-mon[91136]: from='client.47283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:50.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-mon[91136]: from='client.47289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:50.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1263368686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:50.395 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:50.281Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:50.396 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:50.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:50.396 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:50.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:50 smithi146 ceph-mon[103278]: from='client.47283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:50 smithi146 ceph-mon[103278]: from='client.47289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1263368686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:07:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:07:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:07:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:51 smithi146 ceph-mon[103278]: from='client.47295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:51 smithi146 ceph-mon[103278]: pgmap v1438: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:51 smithi146 ceph-mon[103278]: from='client.47307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1447104976' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[91136]: from='client.47295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[91136]: pgmap v1438: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[91136]: from='client.47307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1447104976' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[99959]: from='client.47295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[99959]: pgmap v1438: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[99959]: from='client.47307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:07:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1447104976' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:07:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:07:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:53 smithi146 ceph-mon[103278]: pgmap v1439: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:53 smithi078 ceph-mon[91136]: pgmap v1439: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:53 smithi078 ceph-mon[99959]: pgmap v1439: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:53.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:53.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:07:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:07:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:07:53.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:07:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:55 smithi146 ceph-mon[103278]: pgmap v1440: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:55 smithi078 ceph-mon[91136]: pgmap v1440: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:55 smithi078 ceph-mon[99959]: pgmap v1440: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:07:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:57 smithi146 ceph-mon[103278]: pgmap v1441: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:07:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:57 smithi078 ceph-mon[99959]: pgmap v1441: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:07:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:57 smithi078 ceph-mon[91136]: pgmap v1441: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:07:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:07:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:07:59 smithi146 ceph-mon[103278]: pgmap v1442: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:07:59 smithi078 ceph-mon[99959]: pgmap v1442: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:07:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:07:59 smithi078 ceph-mon[91136]: pgmap v1442: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:00 smithi078 ceph-mon[99959]: pgmap v1443: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:00 smithi078 ceph-mon[91136]: pgmap v1443: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:00.282Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:00.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:00.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:00 smithi146 ceph-mon[103278]: pgmap v1443: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:08:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:08:01.531 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:01.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:02 smithi078 ceph-mon[91136]: pgmap v1444: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:02 smithi078 ceph-mon[99959]: pgmap v1444: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:02 smithi146 ceph-mon[103278]: pgmap v1444: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:03.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:03.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:03.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:04 smithi146 ceph-mon[103278]: pgmap v1445: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[91136]: pgmap v1445: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[99959]: pgmap v1445: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:07.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:06 smithi146 ceph-mon[103278]: pgmap v1446: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:06 smithi078 ceph-mon[91136]: pgmap v1446: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:06 smithi078 ceph-mon[99959]: pgmap v1446: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:09.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:08 smithi146 ceph-mon[103278]: pgmap v1447: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:08 smithi078 ceph-mon[91136]: pgmap v1447: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:08 smithi078 ceph-mon[99959]: pgmap v1447: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:10.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:10.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:10.283Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:10.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:10.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:10.874 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:08:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:08:11.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:10 smithi078 ceph-mon[91136]: pgmap v1448: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:11.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:10 smithi078 ceph-mon[99959]: pgmap v1448: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:10 smithi146 ceph-mon[103278]: pgmap v1448: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:12.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:13.191 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:12 smithi146 ceph-mon[103278]: pgmap v1449: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:12 smithi078 ceph-mon[91136]: pgmap v1449: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:12 smithi078 ceph-mon[99959]: pgmap v1449: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:13.724Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:13.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:13.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:14 smithi146 ceph-mon[103278]: pgmap v1450: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:14 smithi078 ceph-mon[99959]: pgmap v1450: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:14 smithi078 ceph-mon[91136]: pgmap v1450: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:16 smithi146 ceph-mon[103278]: pgmap v1451: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:16 smithi078 ceph-mon[91136]: pgmap v1451: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:16 smithi078 ceph-mon[99959]: pgmap v1451: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:18 smithi146 ceph-mon[103278]: pgmap v1452: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:18 smithi078 ceph-mon[91136]: pgmap v1452: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:18 smithi078 ceph-mon[99959]: pgmap v1452: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:20.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:20.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:20.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:20.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:20.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:08:20.875 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:08:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:08:21.145 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:20 smithi078 ceph-mon[91136]: pgmap v1453: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:21.146 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:20 smithi078 ceph-mon[99959]: pgmap v1453: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:21.241 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:08:21.241 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 58m - - 2024-09-20T21:08:21.241 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 58m - - 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (53m) 16s ago 54m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (53m) 12s ago 53m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (46m) 16s ago 51m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (45m) 12s ago 58m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (47m) 16s ago 59m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (45m) 16s ago 60m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (44m) 12s ago 58m 90.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (44m) 16s ago 58m 96.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (53m) 16s ago 53m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (53m) 12s ago 53m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (43m) 16s ago 57m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (42m) 16s ago 57m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (41m) 16s ago 57m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:08:21.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (41m) 16s ago 56m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (40m) 12s ago 56m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (39m) 12s ago 55m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (38m) 12s ago 55m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (37m) 12s ago 55m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (46m) 12s ago 54m 64.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (37m) 16s ago 52m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (37m) 12s ago 52m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (37m) 16s ago 52m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:08:21.243 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (37m) 12s ago 52m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:08:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:20 smithi146 ceph-mon[103278]: pgmap v1453: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:08:21.571 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:08:21.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:21.571 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:08:21.571 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:08:21.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:08:21.571 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:08:21.845 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:08:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:08:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:08:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:08:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:08:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:08:21.846 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:08:22.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[99959]: from='client.67241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[99959]: from='client.47325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[99959]: from='client.67253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/457968459' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[91136]: from='client.67241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[91136]: from='client.47325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[91136]: from='client.67253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/457968459' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:22.160 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:08:22.160 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:22.160 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:22.160 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:22.160 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:21 smithi146 ceph-mon[103278]: from='client.67241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:21 smithi146 ceph-mon[103278]: from='client.47325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:21 smithi146 ceph-mon[103278]: from='client.67253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/457968459' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:22 smithi146 ceph-mon[103278]: pgmap v1454: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:22 smithi146 ceph-mon[103278]: from='client.67265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1954488453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:22 smithi078 ceph-mon[91136]: pgmap v1454: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:22 smithi078 ceph-mon[91136]: from='client.67265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1954488453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:22 smithi078 ceph-mon[99959]: pgmap v1454: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:22 smithi078 ceph-mon[99959]: from='client.67265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1954488453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:24.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:23.724Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:24.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:23.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:24.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:23.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:24 smithi146 ceph-mon[103278]: pgmap v1455: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[91136]: pgmap v1455: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[99959]: pgmap v1455: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:26 smithi146 ceph-mon[103278]: pgmap v1456: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:26 smithi078 ceph-mon[91136]: pgmap v1456: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:26 smithi078 ceph-mon[99959]: pgmap v1456: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:28 smithi146 ceph-mon[103278]: pgmap v1457: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:28 smithi078 ceph-mon[91136]: pgmap v1457: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:28 smithi078 ceph-mon[99959]: pgmap v1457: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:30.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:30.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:30.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:31.026 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:08:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:08:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:31 smithi078 ceph-mon[91136]: pgmap v1458: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:31 smithi078 ceph-mon[99959]: pgmap v1458: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:31 smithi146 ceph-mon[103278]: pgmap v1458: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:33 smithi078 ceph-mon[91136]: pgmap v1459: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:33 smithi078 ceph-mon[99959]: pgmap v1459: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:33 smithi146 ceph-mon[103278]: pgmap v1459: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:33.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:33.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:33.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:35 smithi078 ceph-mon[91136]: pgmap v1460: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:35 smithi078 ceph-mon[99959]: pgmap v1460: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:35 smithi146 ceph-mon[103278]: pgmap v1460: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:37 smithi146 ceph-mon[103278]: pgmap v1461: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:37 smithi078 ceph-mon[91136]: pgmap v1461: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:37 smithi078 ceph-mon[99959]: pgmap v1461: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:39 smithi146 ceph-mon[103278]: pgmap v1462: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:39 smithi078 ceph-mon[91136]: pgmap v1462: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:39 smithi078 ceph-mon[99959]: pgmap v1462: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:40.285Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:40.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:40.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:40.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:40.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:08:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:08:41.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:41 smithi078 ceph-mon[99959]: pgmap v1463: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:41.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:41 smithi078 ceph-mon[91136]: pgmap v1463: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:41 smithi146 ceph-mon[103278]: pgmap v1463: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:42.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:42.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:43 smithi078 ceph-mon[91136]: pgmap v1464: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:43 smithi078 ceph-mon[99959]: pgmap v1464: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:43.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:43 smithi146 ceph-mon[103278]: pgmap v1464: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:44.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:43.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:43.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:43.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:44.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:44.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:45 smithi146 ceph-mon[103278]: pgmap v1465: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[91136]: pgmap v1465: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[99959]: pgmap v1465: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:47 smithi146 ceph-mon[103278]: pgmap v1466: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:47 smithi078 ceph-mon[91136]: pgmap v1466: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:47 smithi078 ceph-mon[99959]: pgmap v1466: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:49.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:49 smithi146 ceph-mon[103278]: pgmap v1467: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:49 smithi078 ceph-mon[91136]: pgmap v1467: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:49 smithi078 ceph-mon[99959]: pgmap v1467: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:50.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:08:50.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:50.286Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:50.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:50.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:50.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:50.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:08:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:08:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:51 smithi146 ceph-mon[103278]: pgmap v1468: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:51 smithi078 ceph-mon[91136]: pgmap v1468: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:51 smithi078 ceph-mon[99959]: pgmap v1468: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:52.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:52.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:52.447 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:08:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:08:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:08:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 59m - - 2024-09-20T21:08:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 59m - - 2024-09-20T21:08:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (53m) 8s ago 54m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:08:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (54m) 3s ago 54m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:08:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (47m) 8s ago 52m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (45m) 3s ago 58m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (47m) 8s ago 60m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (46m) 8s ago 60m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (44m) 3s ago 59m 92.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (45m) 8s ago 58m 97.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (54m) 8s ago 54m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (54m) 3s ago 54m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (43m) 8s ago 58m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (42m) 8s ago 57m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (42m) 8s ago 57m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:08:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (41m) 8s ago 57m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (41m) 3s ago 56m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (40m) 3s ago 56m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (39m) 3s ago 56m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (38m) 3s ago 55m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (47m) 3s ago 55m 66.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (38m) 8s ago 52m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (37m) 3s ago 52m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (37m) 8s ago 52m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:08:52.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (37m) 3s ago 52m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:08:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:53 smithi078 ceph-mon[91136]: pgmap v1469: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:53 smithi078 ceph-mon[99959]: pgmap v1469: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:08:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:08:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:08:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:53 smithi146 ceph-mon[103278]: pgmap v1469: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:08:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:08:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:08:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:08:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:08:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:08:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:08:53.561 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:08:53.871 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:08:53.871 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:53.871 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:08:53.871 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:53.871 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:08:53.871 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:53.726Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:53.872 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:53.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:08:53.872 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:08:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:08:53.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:08:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:54 smithi146 ceph-mon[103278]: from='client.67277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:54 smithi146 ceph-mon[103278]: from='client.67283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:54 smithi146 ceph-mon[103278]: from='client.47367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1266171450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3222532806' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[99959]: from='client.67277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[99959]: from='client.67283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[99959]: from='client.47367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1266171450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3222532806' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[91136]: from='client.67277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[91136]: from='client.67283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[91136]: from='client.47367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1266171450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:08:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3222532806' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:08:55.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:55 smithi146 ceph-mon[103278]: from='client.67301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:55.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:55 smithi146 ceph-mon[103278]: pgmap v1470: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:55 smithi078 ceph-mon[91136]: from='client.67301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:55 smithi078 ceph-mon[91136]: pgmap v1470: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:55 smithi078 ceph-mon[99959]: from='client.67301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:08:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:55 smithi078 ceph-mon[99959]: pgmap v1470: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:08:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:57 smithi146 ceph-mon[103278]: pgmap v1471: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:57 smithi078 ceph-mon[91136]: pgmap v1471: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:57 smithi078 ceph-mon[99959]: pgmap v1471: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:08:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:08:59 smithi146 ceph-mon[103278]: pgmap v1472: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:08:59 smithi078 ceph-mon[91136]: pgmap v1472: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:08:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:08:59 smithi078 ceph-mon[99959]: pgmap v1472: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:00.287Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:00.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:00.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:09:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:09:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:01 smithi146 ceph-mon[103278]: pgmap v1473: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:01 smithi078 ceph-mon[91136]: pgmap v1473: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:01 smithi078 ceph-mon[99959]: pgmap v1473: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:02.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:02.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:03.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:03 smithi078 ceph-mon[91136]: pgmap v1474: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:03.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:03 smithi078 ceph-mon[99959]: pgmap v1474: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:03 smithi146 ceph-mon[103278]: pgmap v1474: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:04.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:03.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:03.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:03.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[99959]: pgmap v1475: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[91136]: pgmap v1475: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:05 smithi146 ceph-mon[103278]: pgmap v1475: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:07 smithi146 ceph-mon[103278]: pgmap v1476: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:07 smithi078 ceph-mon[99959]: pgmap v1476: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:07 smithi078 ceph-mon[91136]: pgmap v1476: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:09 smithi146 ceph-mon[103278]: pgmap v1477: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:09 smithi078 ceph-mon[91136]: pgmap v1477: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:09 smithi078 ceph-mon[99959]: pgmap v1477: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[91136]: pgmap v1478: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-mon[99959]: pgmap v1478: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:10.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:10.288Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:10.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:10.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:10.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:10.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:10.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:10.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:10 smithi146 ceph-mon[103278]: pgmap v1478: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:09:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:09:11.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:12.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:12 smithi146 ceph-mon[103278]: pgmap v1479: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:12 smithi078 ceph-mon[91136]: pgmap v1479: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:12 smithi078 ceph-mon[99959]: pgmap v1479: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:13.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:13.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:14.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:13.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:13.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:13.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:14 smithi146 ceph-mon[103278]: pgmap v1480: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:14 smithi078 ceph-mon[91136]: pgmap v1480: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:14 smithi078 ceph-mon[99959]: pgmap v1480: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:16 smithi146 ceph-mon[103278]: pgmap v1481: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:16 smithi078 ceph-mon[99959]: pgmap v1481: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:16 smithi078 ceph-mon[91136]: pgmap v1481: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:18 smithi146 ceph-mon[103278]: pgmap v1482: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:18 smithi078 ceph-mon[91136]: pgmap v1482: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:18 smithi078 ceph-mon[99959]: pgmap v1482: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:20.289Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:20.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:20.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:20 smithi078 ceph-mon[91136]: pgmap v1483: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:20 smithi078 ceph-mon[99959]: pgmap v1483: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:09:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:09:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:20 smithi146 ceph-mon[103278]: pgmap v1483: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:23.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:22 smithi078 ceph-mon[91136]: pgmap v1484: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:23.132 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:22 smithi078 ceph-mon[99959]: pgmap v1484: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:22 smithi146 ceph-mon[103278]: pgmap v1484: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:24.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:23.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:23.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:23.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:09:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:09:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 20s ago 60m - - 2024-09-20T21:09:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 59m - - 2024-09-20T21:09:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (54m) 20s ago 55m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:09:24.746 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (55m) 15s ago 55m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (47m) 20s ago 52m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (46m) 15s ago 59m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (48m) 20s ago 60m 548M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (46m) 20s ago 61m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (45m) 15s ago 59m 92.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (45m) 20s ago 59m 97.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (54m) 20s ago 54m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (54m) 15s ago 54m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (44m) 20s ago 58m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (43m) 20s ago 58m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (42m) 20s ago 58m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:09:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (42m) 20s ago 57m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (41m) 15s ago 57m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (40m) 15s ago 56m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (39m) 15s ago 56m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (38m) 15s ago 56m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (47m) 15s ago 55m 66.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (38m) 20s ago 53m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (38m) 15s ago 53m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (38m) 20s ago 53m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:09:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (38m) 15s ago 53m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:09:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:24 smithi078 ceph-mon[91136]: pgmap v1485: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:24 smithi078 ceph-mon[91136]: from='client.47391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:24 smithi078 ceph-mon[99959]: pgmap v1485: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:24 smithi078 ceph-mon[99959]: from='client.47391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:25.111 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:09:25.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:09:25.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:25.112 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:09:25.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:09:25.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:09:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:09:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:24 smithi146 ceph-mon[103278]: pgmap v1485: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:24 smithi146 ceph-mon[103278]: from='client.47391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:09:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:09:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:09:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:09:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:09:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:09:25.386 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:09:25.698 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:09:25.698 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:25.698 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:25.699 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:25.699 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='client.47397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='client.47403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1744562877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1958887316' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='client.47397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='client.47403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1744562877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:26.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1958887316' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='client.47397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='client.47403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1744562877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1958887316' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:26 smithi146 ceph-mon[103278]: from='client.47415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:26 smithi146 ceph-mon[103278]: pgmap v1486: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:26 smithi078 ceph-mon[91136]: from='client.47415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:26 smithi078 ceph-mon[91136]: pgmap v1486: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:26 smithi078 ceph-mon[99959]: from='client.47415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:26 smithi078 ceph-mon[99959]: pgmap v1486: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:29.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:28 smithi146 ceph-mon[103278]: pgmap v1487: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:29.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:28 smithi078 ceph-mon[91136]: pgmap v1487: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:28 smithi078 ceph-mon[99959]: pgmap v1487: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:09:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[91136]: pgmap v1488: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-mon[99959]: pgmap v1488: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:30.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:30.290Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:30.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:30.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:30.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:30.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:30 smithi146 ceph-mon[103278]: pgmap v1488: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:09:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:09:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:31.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:32 smithi146 ceph-mon[103278]: pgmap v1489: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:32 smithi078 ceph-mon[91136]: pgmap v1489: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:32 smithi078 ceph-mon[99959]: pgmap v1489: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:34.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:33.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:33.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:33.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:34 smithi146 ceph-mon[103278]: pgmap v1490: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:34 smithi078 ceph-mon[99959]: pgmap v1490: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:34 smithi078 ceph-mon[91136]: pgmap v1490: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:36 smithi146 ceph-mon[103278]: pgmap v1491: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:36 smithi078 ceph-mon[91136]: pgmap v1491: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:36 smithi078 ceph-mon[99959]: pgmap v1491: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:38 smithi146 ceph-mon[103278]: pgmap v1492: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:38 smithi078 ceph-mon[91136]: pgmap v1492: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:38 smithi078 ceph-mon[99959]: pgmap v1492: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:40.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:40.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:40.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:40 smithi078 ceph-mon[99959]: pgmap v1493: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:40 smithi078 ceph-mon[91136]: pgmap v1493: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:09:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:09:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:40 smithi146 ceph-mon[103278]: pgmap v1493: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:42.348 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:42 smithi146 ceph-mon[103278]: pgmap v1494: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:42 smithi078 ceph-mon[91136]: pgmap v1494: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:42 smithi078 ceph-mon[99959]: pgmap v1494: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:43.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:43.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:43.983 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:43.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:43.983 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:43.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:43.983 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:43.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:45.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:44 smithi078 ceph-mon[91136]: pgmap v1495: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:45.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:44 smithi078 ceph-mon[99959]: pgmap v1495: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:44 smithi146 ceph-mon[103278]: pgmap v1495: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:46.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:47 smithi078 ceph-mon[91136]: pgmap v1496: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:47 smithi078 ceph-mon[99959]: pgmap v1496: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:47 smithi146 ceph-mon[103278]: pgmap v1496: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:49 smithi146 ceph-mon[103278]: pgmap v1497: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:49 smithi078 ceph-mon[99959]: pgmap v1497: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:49 smithi078 ceph-mon[91136]: pgmap v1497: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[91136]: pgmap v1498: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-mon[99959]: pgmap v1498: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:50.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:50.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:50.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:50.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:50.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:50.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:09:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:50 smithi146 ceph-mon[103278]: pgmap v1498: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:09:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:09:51.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:09:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:52 smithi146 ceph-mon[103278]: pgmap v1499: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:52 smithi078 ceph-mon[99959]: pgmap v1499: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:52 smithi078 ceph-mon[91136]: pgmap v1499: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:54.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:53.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:53.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:09:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:09:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:09:53.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:09:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:54 smithi146 ceph-mon[103278]: pgmap v1500: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:54 smithi078 ceph-mon[91136]: pgmap v1500: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:54 smithi078 ceph-mon[99959]: pgmap v1500: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:09:55.998 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:09:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:09:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 60m - - 2024-09-20T21:09:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 60m - - 2024-09-20T21:09:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (54m) 11s ago 55m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:09:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (55m) 6s ago 55m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:09:56.522 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (48m) 11s ago 53m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (46m) 6s ago 59m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (48m) 11s ago 61m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (47m) 11s ago 61m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (45m) 6s ago 60m 93.5M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (46m) 11s ago 60m 98.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (55m) 11s ago 55m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (55m) 6s ago 55m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (44m) 11s ago 59m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (43m) 11s ago 59m 66.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (43m) 11s ago 58m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (42m) 11s ago 58m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (42m) 6s ago 57m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (41m) 6s ago 57m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:09:56.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (40m) 6s ago 57m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:09:56.524 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (39m) 6s ago 56m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:09:56.524 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (48m) 6s ago 56m 64.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:09:56.524 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (39m) 11s ago 53m 87.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:09:56.524 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (38m) 6s ago 53m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:09:56.524 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (39m) 11s ago 53m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:09:56.524 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (38m) 6s ago 53m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:09:56.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:09:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:09:56.868 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:09:56.868 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:09:56.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:09:56.868 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:09:57.142 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:09:57.143 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:09:57.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:56 smithi078 ceph-mon[99959]: pgmap v1501: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:57.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:56 smithi078 ceph-mon[99959]: from='client.67346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:57.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:56 smithi078 ceph-mon[99959]: from='client.67352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:57.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:56 smithi078 ceph-mon[91136]: pgmap v1501: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:57.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:56 smithi078 ceph-mon[91136]: from='client.67346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:57.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:56 smithi078 ceph-mon[91136]: from='client.67352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:56 smithi146 ceph-mon[103278]: pgmap v1501: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:56 smithi146 ceph-mon[103278]: from='client.67346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:56 smithi146 ceph-mon[103278]: from='client.67352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:57.455 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:09:57.455 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:57.455 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:09:57.455 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:57.455 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:09:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:57 smithi146 ceph-mon[103278]: from='client.47439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3374808272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:57 smithi146 ceph-mon[103278]: from='client.67370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3461707699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[91136]: from='client.47439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3374808272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[91136]: from='client.67370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3461707699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[99959]: from='client.47439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3374808272' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[99959]: from='client.67370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:09:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3461707699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:09:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:58 smithi146 ceph-mon[103278]: pgmap v1502: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:58 smithi078 ceph-mon[91136]: pgmap v1502: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:09:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:58 smithi078 ceph-mon[99959]: pgmap v1502: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:09:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T21:10:00.000+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:00.292Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:00.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:00.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:00.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:10:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:10:01.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-mon[91136]: pgmap v1503: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:01.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:01.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-mon[99959]: pgmap v1503: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:01.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:00 smithi146 ceph-mon[103278]: pgmap v1503: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:02 smithi146 ceph-mon[103278]: pgmap v1504: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:02 smithi078 ceph-mon[99959]: pgmap v1504: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:02 smithi078 ceph-mon[91136]: pgmap v1504: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:04.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:03.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:03.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:03.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:04 smithi078 ceph-mon[91136]: pgmap v1505: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:04 smithi078 ceph-mon[99959]: pgmap v1505: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:04 smithi146 ceph-mon[103278]: pgmap v1505: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:06.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:07 smithi078 ceph-mon[91136]: pgmap v1506: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:07 smithi078 ceph-mon[99959]: pgmap v1506: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:07 smithi146 ceph-mon[103278]: pgmap v1506: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:09 smithi146 ceph-mon[103278]: pgmap v1507: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:09 smithi078 ceph-mon[91136]: pgmap v1507: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:09 smithi078 ceph-mon[99959]: pgmap v1507: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:10.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:10.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:10.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[99959]: pgmap v1508: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-mon[91136]: pgmap v1508: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:10.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:10:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:10:10.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:10.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:10 smithi146 ceph-mon[103278]: pgmap v1508: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:12 smithi146 ceph-mon[103278]: pgmap v1509: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:12 smithi078 ceph-mon[91136]: pgmap v1509: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:12 smithi078 ceph-mon[99959]: pgmap v1509: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:13.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:13.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:13.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:14 smithi146 ceph-mon[103278]: pgmap v1510: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:14 smithi078 ceph-mon[91136]: pgmap v1510: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:14 smithi078 ceph-mon[99959]: pgmap v1510: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:16 smithi146 ceph-mon[103278]: pgmap v1511: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:16 smithi078 ceph-mon[91136]: pgmap v1511: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:16 smithi078 ceph-mon[99959]: pgmap v1511: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:18 smithi146 ceph-mon[103278]: pgmap v1512: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:18 smithi078 ceph-mon[91136]: pgmap v1512: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:18 smithi078 ceph-mon[99959]: pgmap v1512: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:20.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:20.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:20.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:20.869 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:10:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:10:21.131 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:20 smithi078 ceph-mon[99959]: pgmap v1513: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:21.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:20 smithi078 ceph-mon[91136]: pgmap v1513: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:20 smithi146 ceph-mon[103278]: pgmap v1513: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:22 smithi146 ceph-mon[103278]: pgmap v1514: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:22 smithi078 ceph-mon[91136]: pgmap v1514: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:22 smithi078 ceph-mon[99959]: pgmap v1514: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:23.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:23.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:23.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:25.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:24 smithi078 ceph-mon[91136]: pgmap v1515: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:25.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:24 smithi078 ceph-mon[99959]: pgmap v1515: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:24 smithi146 ceph-mon[103278]: pgmap v1515: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:26.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.402 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:27 smithi078 ceph-mon[91136]: pgmap v1516: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:27 smithi078 ceph-mon[99959]: pgmap v1516: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:27 smithi146 ceph-mon[103278]: pgmap v1516: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:10:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:10:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 61m - - 2024-09-20T21:10:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 60m - - 2024-09-20T21:10:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (55m) 3s ago 56m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:10:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (56m) 18s ago 56m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:10:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (48m) 3s ago 54m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:10:28.336 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (47m) 18s ago 60m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:10:28.336 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (49m) 3s ago 61m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:10:28.336 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (47m) 3s ago 62m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (46m) 18s ago 60m 92.0M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (46m) 3s ago 60m 97.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (55m) 3s ago 55m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (55m) 18s ago 55m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (45m) 3s ago 59m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (44m) 3s ago 59m 66.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (43m) 3s ago 59m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (43m) 3s ago 58m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (42m) 18s ago 58m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:10:28.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (41m) 18s ago 58m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (40m) 18s ago 57m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (39m) 18s ago 57m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (48m) 18s ago 56m 64.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (39m) 3s ago 54m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (39m) 18s ago 54m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (39m) 3s ago 54m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:10:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (39m) 18s ago 54m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:10:28.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:10:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:10:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:10:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:10:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:10:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:10:28.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:10:28.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[99959]: from='client.67382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:28.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[99959]: pgmap v1517: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:28.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[99959]: from='client.67388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:28.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:28.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[91136]: from='client.67382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:28.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[91136]: pgmap v1517: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:28.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[91136]: from='client.67388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:28.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:28 smithi146 ceph-mon[103278]: from='client.67382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:28 smithi146 ceph-mon[103278]: pgmap v1517: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:28 smithi146 ceph-mon[103278]: from='client.67388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:28.919 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:10:28.919 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:10:28.919 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:10:28.920 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:10:29.224 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:10:29.224 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:10:29.224 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:10:29.224 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:10:29.224 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:10:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[99959]: from='client.47475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3701853078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[99959]: from='client.67406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/569792962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[91136]: from='client.47475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3701853078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[91136]: from='client.67406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/569792962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:29 smithi146 ceph-mon[103278]: from='client.47475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3701853078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:10:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:29 smithi146 ceph-mon[103278]: from='client.67406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:10:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/569792962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:10:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:30.294Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:30.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:30.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:30.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:30.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:30.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:30.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:30 smithi146 ceph-mon[103278]: pgmap v1518: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:10:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:10:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[99959]: pgmap v1518: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:31.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:30 smithi078 ceph-mon[91136]: pgmap v1518: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:32.095 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:33.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:32 smithi146 ceph-mon[103278]: pgmap v1519: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:32 smithi078 ceph-mon[91136]: pgmap v1519: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:32 smithi078 ceph-mon[99959]: pgmap v1519: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:33.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:33.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:33.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:34 smithi146 ceph-mon[103278]: pgmap v1520: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:34 smithi078 ceph-mon[99959]: pgmap v1520: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:34 smithi078 ceph-mon[91136]: pgmap v1520: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:36 smithi146 ceph-mon[103278]: pgmap v1521: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:36 smithi078 ceph-mon[99959]: pgmap v1521: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:36 smithi078 ceph-mon[91136]: pgmap v1521: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:38 smithi146 ceph-mon[103278]: pgmap v1522: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:38 smithi078 ceph-mon[91136]: pgmap v1522: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:38 smithi078 ceph-mon[99959]: pgmap v1522: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:40.295Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:40.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:40.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:40.881 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:10:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:10:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:40 smithi146 ceph-mon[103278]: pgmap v1523: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:41.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:40 smithi078 ceph-mon[99959]: pgmap v1523: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:41.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:40 smithi078 ceph-mon[91136]: pgmap v1523: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:42.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:42 smithi146 ceph-mon[103278]: pgmap v1524: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:42 smithi078 ceph-mon[91136]: pgmap v1524: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:42 smithi078 ceph-mon[99959]: pgmap v1524: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:43.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:43.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:43.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:44 smithi146 ceph-mon[103278]: pgmap v1525: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:44 smithi078 ceph-mon[99959]: pgmap v1525: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:44 smithi078 ceph-mon[91136]: pgmap v1525: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:46.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[91136]: pgmap v1526: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:46.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:46 smithi078 ceph-mon[99959]: pgmap v1526: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:46 smithi146 ceph-mon[103278]: pgmap v1526: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:48 smithi146 ceph-mon[103278]: pgmap v1527: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:48 smithi078 ceph-mon[91136]: pgmap v1527: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:48 smithi078 ceph-mon[99959]: pgmap v1527: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:50.296Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:50.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:50.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:50.915 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:10:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:10:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:50 smithi146 ceph-mon[103278]: pgmap v1528: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[91136]: pgmap v1528: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[99959]: pgmap v1528: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:10:51.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:51.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:10:52.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:10:53.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:52 smithi146 ceph-mon[103278]: pgmap v1529: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:52 smithi078 ceph-mon[91136]: pgmap v1529: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:52 smithi078 ceph-mon[99959]: pgmap v1529: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:10:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:53.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:53.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:10:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:10:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:10:53.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:10:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:54 smithi146 ceph-mon[103278]: pgmap v1530: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:10:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:54 smithi078 ceph-mon[91136]: pgmap v1530: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:10:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:54 smithi078 ceph-mon[99959]: pgmap v1530: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:10:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:56 smithi146 ceph-mon[103278]: pgmap v1531: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:10:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:56 smithi078 ceph-mon[91136]: pgmap v1531: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:10:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:56 smithi078 ceph-mon[99959]: pgmap v1531: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:10:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:58 smithi146 ceph-mon[103278]: pgmap v1532: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:10:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:10:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:58 smithi078 ceph-mon[99959]: pgmap v1532: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:10:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:10:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:58 smithi078 ceph-mon[91136]: pgmap v1532: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:10:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:10:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:10:59.516 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:11:00.045 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:11:00.045 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 61m - - 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 61m - - 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (55m) 14s ago 56m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (56m) 10s ago 56m 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (49m) 14s ago 54m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (47m) 10s ago 60m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (49m) 14s ago 62m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (48m) 14s ago 62m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (46m) 10s ago 61m 93.4M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (47m) 14s ago 61m 98.3M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (56m) 14s ago 56m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (56m) 10s ago 56m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:11:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (45m) 14s ago 60m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (44m) 14s ago 60m 66.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (44m) 14s ago 59m 59.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (43m) 14s ago 59m 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (43m) 10s ago 58m 69.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (42m) 10s ago 58m 59.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (41m) 10s ago 58m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (40m) 10s ago 57m 58.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (49m) 10s ago 57m 66.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (40m) 14s ago 54m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (39m) 10s ago 55m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (40m) 14s ago 54m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:11:00.047 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (39m) 10s ago 54m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:11:00.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:11:00.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:00.359 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:11:00.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:11:00.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:11:00.359 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:11:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:00.297Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:00.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:00.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:11:00.627 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:11:00.628 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:11:00.628 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:11:00.628 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:11:00.628 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:11:00.628 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:11:00.932 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:11:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:11:00.937 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:11:00.937 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:00.938 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:00.938 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:00.938 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:00 smithi146 ceph-mon[103278]: from='client.47499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:00 smithi146 ceph-mon[103278]: from='client.67424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:00 smithi146 ceph-mon[103278]: pgmap v1533: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-20T21:11:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:00 smithi146 ceph-mon[103278]: from='client.47511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1214238382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[91136]: from='client.47499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[91136]: from='client.67424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[91136]: pgmap v1533: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[91136]: from='client.47511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1214238382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[99959]: from='client.47499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[99959]: from='client.67424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[99959]: pgmap v1533: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[99959]: from='client.47511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1214238382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:01 smithi146 ceph-mon[103278]: from='client.67442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/351288262' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:01 smithi078 ceph-mon[99959]: from='client.67442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/351288262' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:01 smithi078 ceph-mon[91136]: from='client.67442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/351288262' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:02 smithi146 ceph-mon[103278]: pgmap v1534: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T21:11:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:02 smithi078 ceph-mon[99959]: pgmap v1534: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T21:11:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:02 smithi078 ceph-mon[91136]: pgmap v1534: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T21:11:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:03.736Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:03.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:03.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:04 smithi146 ceph-mon[103278]: pgmap v1535: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:11:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:04 smithi078 ceph-mon[91136]: pgmap v1535: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:11:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:04 smithi078 ceph-mon[99959]: pgmap v1535: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:11:06.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:06.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:06.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:06.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[91136]: pgmap v1536: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T21:11:06.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:06.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:06.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:06.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:06 smithi078 ceph-mon[99959]: pgmap v1536: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T21:11:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:06 smithi146 ceph-mon[103278]: pgmap v1536: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T21:11:08.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:08 smithi078 ceph-mon[91136]: pgmap v1537: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:11:08.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:08 smithi078 ceph-mon[99959]: pgmap v1537: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:11:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:08 smithi146 ceph-mon[103278]: pgmap v1537: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:11:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:10.298Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:10.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:10.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:11:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[91136]: pgmap v1538: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[99959]: pgmap v1538: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.299 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:11 smithi146 ceph-mon[103278]: pgmap v1538: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:11:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:13 smithi146 ceph-mon[103278]: pgmap v1539: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T21:11:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-mon[99959]: pgmap v1539: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T21:11:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-mon[91136]: pgmap v1539: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T21:11:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:13.736Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:13.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:13.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:14.453 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:15.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:15 smithi146 ceph-mon[103278]: pgmap v1540: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T21:11:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:15 smithi078 ceph-mon[91136]: pgmap v1540: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T21:11:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:15 smithi078 ceph-mon[99959]: pgmap v1540: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T21:11:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:17 smithi146 ceph-mon[103278]: pgmap v1541: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:11:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:17 smithi078 ceph-mon[91136]: pgmap v1541: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:11:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:17 smithi078 ceph-mon[99959]: pgmap v1541: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:11:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:19 smithi146 ceph-mon[103278]: pgmap v1542: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T21:11:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:19 smithi078 ceph-mon[91136]: pgmap v1542: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T21:11:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:19 smithi078 ceph-mon[99959]: pgmap v1542: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T21:11:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:20.299Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:20.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:20.300Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:11:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:11:21.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:21 smithi078 ceph-mon[99959]: pgmap v1543: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:11:21.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:21 smithi078 ceph-mon[91136]: pgmap v1543: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:11:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:21 smithi146 ceph-mon[103278]: pgmap v1543: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:11:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:23 smithi146 ceph-mon[103278]: pgmap v1544: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:11:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-mon[91136]: pgmap v1544: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:11:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-mon[99959]: pgmap v1544: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:11:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:23.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:23.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:23.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:25.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:25 smithi146 ceph-mon[103278]: pgmap v1545: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:11:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:25 smithi078 ceph-mon[91136]: pgmap v1545: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:11:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:25 smithi078 ceph-mon[99959]: pgmap v1545: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:11:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:26 smithi146 ceph-mon[103278]: pgmap v1546: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[91136]: pgmap v1546: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:26 smithi078 ceph-mon[99959]: pgmap v1546: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:28 smithi146 ceph-mon[103278]: pgmap v1547: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:28 smithi078 ceph-mon[91136]: pgmap v1547: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:28 smithi078 ceph-mon[99959]: pgmap v1547: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:30.300Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:30.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:30.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:11:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:11:31.232 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:11:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:31 smithi146 ceph-mon[103278]: pgmap v1548: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[99959]: pgmap v1548: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[91136]: pgmap v1548: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:31.764 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:11:31.764 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 62m - - 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 61m - - 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (56m) 6s ago 57m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (57m) 1s ago 57m 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (49m) 6s ago 55m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (48m) 1s ago 61m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (50m) 6s ago 62m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (48m) 6s ago 63m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (47m) 1s ago 61m 95.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (47m) 6s ago 61m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (56m) 6s ago 56m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (56m) 1s ago 56m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:11:31.765 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (46m) 6s ago 61m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (45m) 6s ago 60m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (44m) 6s ago 60m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (44m) 6s ago 59m 66.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (43m) 1s ago 59m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (42m) 1s ago 59m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (41m) 1s ago 58m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (40m) 1s ago 58m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (49m) 1s ago 57m 66.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (40m) 6s ago 55m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (40m) 1s ago 55m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (40m) 6s ago 55m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:11:31.766 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (40m) 1s ago 55m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:11:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:11:32.076 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:11:32.348 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:11:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:11:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:11:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:11:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:11:32.349 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:11:32.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:11:32.349 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:32 smithi078 ceph-mon[99959]: from='client.47535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:32.349 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2494417897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:32.349 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:32 smithi078 ceph-mon[91136]: from='client.47535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:32.350 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2494417897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:32 smithi146 ceph-mon[103278]: from='client.47535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2494417897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:11:32.656 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:11:32.656 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:11:32.656 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:33 smithi146 ceph-mon[103278]: from='client.47541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:33 smithi146 ceph-mon[103278]: from='client.47547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:33 smithi146 ceph-mon[103278]: pgmap v1549: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1743656987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[99959]: from='client.47541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[99959]: from='client.47547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[99959]: pgmap v1549: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1743656987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[91136]: from='client.47541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[91136]: from='client.47547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[91136]: pgmap v1549: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1743656987' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:11:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:33.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:33.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:33.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:34.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:34 smithi146 ceph-mon[103278]: from='client.67478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:34 smithi078 ceph-mon[91136]: from='client.67478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:34 smithi078 ceph-mon[99959]: from='client.67478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:11:35.391 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:35 smithi146 ceph-mon[103278]: pgmap v1550: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:35 smithi078 ceph-mon[91136]: pgmap v1550: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:35 smithi078 ceph-mon[99959]: pgmap v1550: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:37 smithi146 ceph-mon[103278]: pgmap v1551: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:37 smithi078 ceph-mon[91136]: pgmap v1551: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:37 smithi078 ceph-mon[99959]: pgmap v1551: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:39 smithi146 ceph-mon[103278]: pgmap v1552: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:39 smithi078 ceph-mon[91136]: pgmap v1552: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:39 smithi078 ceph-mon[99959]: pgmap v1552: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:40.301Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:40.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:40.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:40.894 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:11:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:11:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:41 smithi146 ceph-mon[103278]: pgmap v1553: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:41 smithi078 ceph-mon[99959]: pgmap v1553: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:41 smithi078 ceph-mon[91136]: pgmap v1553: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:43 smithi146 ceph-mon[103278]: pgmap v1554: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-mon[91136]: pgmap v1554: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-mon[99959]: pgmap v1554: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:43.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:43.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:43.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:45 smithi146 ceph-mon[103278]: pgmap v1555: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:45 smithi078 ceph-mon[91136]: pgmap v1555: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:45 smithi078 ceph-mon[99959]: pgmap v1555: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:46 smithi146 ceph-mon[103278]: pgmap v1556: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[99959]: pgmap v1556: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:46 smithi078 ceph-mon[91136]: pgmap v1556: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:48 smithi078 ceph-mon[91136]: pgmap v1557: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:48 smithi078 ceph-mon[99959]: pgmap v1557: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:48 smithi146 ceph-mon[103278]: pgmap v1557: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:50.302Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:50.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:50.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:50.896 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:11:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:11:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:50 smithi146 ceph-mon[103278]: pgmap v1558: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[91136]: pgmap v1558: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[99959]: pgmap v1558: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:11:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:52 smithi146 ceph-mon[103278]: pgmap v1559: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:52 smithi078 ceph-mon[91136]: pgmap v1559: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:11:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:52 smithi078 ceph-mon[99959]: pgmap v1559: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:53.739Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:53.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:11:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:11:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:11:53.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:11:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:54 smithi146 ceph-mon[103278]: pgmap v1560: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:54 smithi078 ceph-mon[91136]: pgmap v1560: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:54 smithi078 ceph-mon[99959]: pgmap v1560: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:11:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:56 smithi146 ceph-mon[103278]: pgmap v1561: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:56 smithi078 ceph-mon[91136]: pgmap v1561: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:56 smithi078 ceph-mon[99959]: pgmap v1561: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:58 smithi146 ceph-mon[103278]: pgmap v1562: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:11:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:58 smithi078 ceph-mon[91136]: pgmap v1562: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:11:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:11:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:58 smithi078 ceph-mon[99959]: pgmap v1562: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:11:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:11:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:00.303Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:00.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:00.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:00.913 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:12:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:12:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:00 smithi146 ceph-mon[103278]: pgmap v1563: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:00 smithi078 ceph-mon[91136]: pgmap v1563: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:00 smithi078 ceph-mon[99959]: pgmap v1563: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:02.953 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:12:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:02 smithi146 ceph-mon[103278]: pgmap v1564: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:02 smithi078 ceph-mon[91136]: pgmap v1564: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:02 smithi078 ceph-mon[99959]: pgmap v1564: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:03.482 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 62m - - 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 62m - - 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (56m) 17s ago 57m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (57m) 13s ago 57m 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (50m) 17s ago 55m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (48m) 13s ago 61m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (50m) 17s ago 63m 549M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (49m) 17s ago 63m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (47m) 13s ago 62m 95.8M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (48m) 17s ago 62m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (57m) 17s ago 57m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (57m) 13s ago 57m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (46m) 17s ago 61m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:12:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (45m) 17s ago 61m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (45m) 17s ago 60m 61.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (44m) 17s ago 60m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (44m) 13s ago 59m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (43m) 13s ago 59m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (42m) 13s ago 59m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (41m) 13s ago 58m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (50m) 13s ago 58m 67.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (41m) 17s ago 55m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (41m) 13s ago 56m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (41m) 17s ago 55m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:12:03.484 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (40m) 13s ago 55m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:12:03.792 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:12:03.792 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:12:03.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:12:03.793 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:12:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-mon[91136]: from='client.67490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-mon[91136]: from='client.67496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/75458432' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-mon[99959]: from='client.67490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-mon[99959]: from='client.67496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/75458432' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:03.739Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:03.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:04.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:03.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:12:04.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:12:04.067 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:12:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:03 smithi146 ceph-mon[103278]: from='client.67490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:03 smithi146 ceph-mon[103278]: from='client.67496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/75458432' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:04.374 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:12:04.375 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:12:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:12:04.375 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:12:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:12:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:04 smithi146 ceph-mon[103278]: from='client.67502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:04 smithi146 ceph-mon[103278]: pgmap v1565: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:04 smithi146 ceph-mon[103278]: from='client.47595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4209553173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[99959]: from='client.67502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[99959]: pgmap v1565: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[99959]: from='client.47595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4209553173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[91136]: from='client.67502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[91136]: pgmap v1565: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[91136]: from='client.47595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4209553173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:06 smithi146 ceph-mon[103278]: pgmap v1566: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[91136]: pgmap v1566: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[99959]: pgmap v1566: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:08 smithi146 ceph-mon[103278]: pgmap v1567: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:09.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:08 smithi078 ceph-mon[99959]: pgmap v1567: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:09.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:08 smithi078 ceph-mon[91136]: pgmap v1567: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:10.304Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:10.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:10.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:10.857 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:12:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:12:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:10 smithi146 ceph-mon[103278]: pgmap v1568: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[99959]: pgmap v1568: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:11.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[91136]: pgmap v1568: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:11.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:11.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:12 smithi146 ceph-mon[103278]: pgmap v1569: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:12 smithi078 ceph-mon[99959]: pgmap v1569: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:12 smithi078 ceph-mon[91136]: pgmap v1569: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:13.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:13.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:13.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:14.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:14 smithi146 ceph-mon[103278]: pgmap v1570: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:14 smithi078 ceph-mon[91136]: pgmap v1570: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:14 smithi078 ceph-mon[99959]: pgmap v1570: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:16 smithi146 ceph-mon[103278]: pgmap v1571: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:16 smithi078 ceph-mon[91136]: pgmap v1571: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:16 smithi078 ceph-mon[99959]: pgmap v1571: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:18 smithi146 ceph-mon[103278]: pgmap v1572: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:18 smithi078 ceph-mon[91136]: pgmap v1572: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:18 smithi078 ceph-mon[99959]: pgmap v1572: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:20.305Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:20.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:20.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:20.957 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:12:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:12:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:20 smithi146 ceph-mon[103278]: pgmap v1573: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:20 smithi078 ceph-mon[91136]: pgmap v1573: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:20 smithi078 ceph-mon[99959]: pgmap v1573: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:22 smithi146 ceph-mon[103278]: pgmap v1574: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:22 smithi078 ceph-mon[91136]: pgmap v1574: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:22 smithi078 ceph-mon[99959]: pgmap v1574: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:23.741Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:23.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:23.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:24 smithi146 ceph-mon[103278]: pgmap v1575: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:24 smithi078 ceph-mon[91136]: pgmap v1575: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:24 smithi078 ceph-mon[99959]: pgmap v1575: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:27 smithi146 ceph-mon[103278]: pgmap v1576: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[91136]: pgmap v1576: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[99959]: pgmap v1576: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:29 smithi078 ceph-mon[99959]: pgmap v1577: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:29 smithi078 ceph-mon[91136]: pgmap v1577: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:29 smithi146 ceph-mon[103278]: pgmap v1577: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:30.305Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:30.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:30.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:30.917 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:12:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:12:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[99959]: pgmap v1578: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[91136]: pgmap v1578: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:31 smithi146 ceph-mon[103278]: pgmap v1578: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:33 smithi146 ceph-mon[103278]: pgmap v1579: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-mon[91136]: pgmap v1579: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-mon[99959]: pgmap v1579: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:33.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:33.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:33.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:34.667 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:12:35.193 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:12:35.193 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 63m - - 2024-09-20T21:12:35.193 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 62m - - 2024-09-20T21:12:35.193 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (57m) 9s ago 58m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (58m) 4s ago 58m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (50m) 9s ago 56m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (49m) 4s ago 62m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (51m) 9s ago 63m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (49m) 9s ago 64m 146M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (48m) 4s ago 62m 97.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (48m) 9s ago 62m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (57m) 9s ago 57m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (57m) 4s ago 57m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (47m) 9s ago 62m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (46m) 9s ago 61m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (45m) 9s ago 61m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (45m) 9s ago 60m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:12:35.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (44m) 4s ago 60m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (43m) 4s ago 60m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (42m) 4s ago 59m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (42m) 4s ago 59m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (50m) 4s ago 58m 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (41m) 9s ago 56m 87.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (41m) 4s ago 56m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (41m) 9s ago 56m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:12:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (41m) 4s ago 56m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:12:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:35 smithi146 ceph-mon[103278]: pgmap v1580: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:35.505 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:12:35.506 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:12:35.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:35 smithi078 ceph-mon[91136]: pgmap v1580: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:35.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:35 smithi078 ceph-mon[99959]: pgmap v1580: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:35.774 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:12:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:12:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:12:35.775 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:12:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:12:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:12:36.086 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:12:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:12:36.087 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:12:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:36 smithi146 ceph-mon[103278]: from='client.47607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:36 smithi146 ceph-mon[103278]: from='client.67532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:36 smithi146 ceph-mon[103278]: from='client.47619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1784762277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[99959]: from='client.47607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[99959]: from='client.67532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[99959]: from='client.47619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1784762277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[91136]: from='client.47607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[91136]: from='client.67532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[91136]: from='client.47619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:36.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1784762277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:12:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:37 smithi146 ceph-mon[103278]: from='client.47631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:37 smithi146 ceph-mon[103278]: pgmap v1581: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2018282713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:37 smithi078 ceph-mon[91136]: from='client.47631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:37 smithi078 ceph-mon[91136]: pgmap v1581: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2018282713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:37 smithi078 ceph-mon[99959]: from='client.47631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:12:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:37 smithi078 ceph-mon[99959]: pgmap v1581: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2018282713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:12:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:39 smithi146 ceph-mon[103278]: pgmap v1582: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:39 smithi078 ceph-mon[91136]: pgmap v1582: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:39 smithi078 ceph-mon[99959]: pgmap v1582: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:40.306Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:40.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:40.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:12:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:12:41.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:41 smithi078 ceph-mon[91136]: pgmap v1583: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:41.366 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:41 smithi078 ceph-mon[99959]: pgmap v1583: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:41 smithi146 ceph-mon[103278]: pgmap v1583: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:43 smithi146 ceph-mon[103278]: pgmap v1584: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-mon[91136]: pgmap v1584: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-mon[99959]: pgmap v1584: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:43.743Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:43.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:43.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:45 smithi146 ceph-mon[103278]: pgmap v1585: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:45 smithi078 ceph-mon[91136]: pgmap v1585: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:45 smithi078 ceph-mon[99959]: pgmap v1585: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:47 smithi146 ceph-mon[103278]: pgmap v1586: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[91136]: pgmap v1586: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[99959]: pgmap v1586: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:47.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:49 smithi146 ceph-mon[103278]: pgmap v1587: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:49 smithi078 ceph-mon[91136]: pgmap v1587: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:49 smithi078 ceph-mon[99959]: pgmap v1587: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:50.430 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:50.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:50.430 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:50.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:50.430 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:50.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:12:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:12:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:51 smithi146 ceph-mon[103278]: pgmap v1588: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[91136]: pgmap v1588: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[99959]: pgmap v1588: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:12:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:53 smithi146 ceph-mon[103278]: pgmap v1589: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-mon[91136]: pgmap v1589: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:12:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-mon[99959]: pgmap v1589: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:53.743Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:53.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:12:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:12:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:12:53.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:12:55.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:55 smithi146 ceph-mon[103278]: pgmap v1590: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:55 smithi078 ceph-mon[91136]: pgmap v1590: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:55 smithi078 ceph-mon[99959]: pgmap v1590: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:12:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:57 smithi146 ceph-mon[103278]: pgmap v1591: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:57 smithi078 ceph-mon[91136]: pgmap v1591: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:57 smithi078 ceph-mon[99959]: pgmap v1591: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:12:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:12:59 smithi146 ceph-mon[103278]: pgmap v1592: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:12:59 smithi078 ceph-mon[91136]: pgmap v1592: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:12:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:12:59 smithi078 ceph-mon[99959]: pgmap v1592: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:00.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:00.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:00.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:13:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:13:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:01 smithi146 ceph-mon[103278]: pgmap v1593: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:01 smithi078 ceph-mon[99959]: pgmap v1593: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:01 smithi078 ceph-mon[91136]: pgmap v1593: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:03 smithi146 ceph-mon[103278]: pgmap v1594: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-mon[91136]: pgmap v1594: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-mon[99959]: pgmap v1594: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:03.744Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:03.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:03.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:05 smithi146 ceph-mon[103278]: pgmap v1595: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:05 smithi078 ceph-mon[91136]: pgmap v1595: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:05 smithi078 ceph-mon[99959]: pgmap v1595: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:06.392 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 63m - - 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 63m - - 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (57m) 0s ago 58m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (58m) 16s ago 58m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (51m) 0s ago 56m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (50m) 16s ago 62m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (51m) 0s ago 64m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (50m) 0s ago 64m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (48m) 16s ago 63m 97.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:13:06.919 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (49m) 0s ago 63m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (58m) 0s ago 58m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (58m) 16s ago 58m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (47m) 0s ago 62m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (46m) 0s ago 62m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (46m) 0s ago 61m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (45m) 0s ago 61m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (45m) 16s ago 61m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (44m) 16s ago 60m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (43m) 16s ago 60m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (42m) 16s ago 59m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (51m) 16s ago 59m 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (42m) 0s ago 57m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (42m) 16s ago 57m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (42m) 0s ago 56m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:13:06.920 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (41m) 16s ago 56m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:07.230 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:13:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:13:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:13:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:13:07.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:13:07.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:13:07.501 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[99959]: pgmap v1596: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[91136]: pgmap v1596: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:07 smithi146 ceph-mon[103278]: pgmap v1596: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:07.819 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:13:07.819 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:07.819 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:07.819 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:07.819 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:08 smithi146 ceph-mon[103278]: from='client.47643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:08 smithi146 ceph-mon[103278]: from='client.47649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:08 smithi146 ceph-mon[103278]: from='client.47655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2427240084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/384976876' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[99959]: from='client.47643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[99959]: from='client.47649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[99959]: from='client.47655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2427240084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/384976876' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[91136]: from='client.47643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[91136]: from='client.47649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[91136]: from='client.47655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2427240084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/384976876' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:09 smithi146 ceph-mon[103278]: from='client.47667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:09 smithi146 ceph-mon[103278]: pgmap v1597: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:09 smithi078 ceph-mon[99959]: from='client.47667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:09 smithi078 ceph-mon[99959]: pgmap v1597: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:09 smithi078 ceph-mon[91136]: from='client.47667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:09 smithi078 ceph-mon[91136]: pgmap v1597: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:10.308Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:10.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:10.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:10.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:13:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:13:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:11 smithi146 ceph-mon[103278]: pgmap v1598: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[91136]: pgmap v1598: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[99959]: pgmap v1598: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:13 smithi146 ceph-mon[103278]: pgmap v1599: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-mon[91136]: pgmap v1599: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-mon[99959]: pgmap v1599: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:13.745Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:13.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:13.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:15 smithi146 ceph-mon[103278]: pgmap v1600: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:15 smithi078 ceph-mon[91136]: pgmap v1600: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:15 smithi078 ceph-mon[99959]: pgmap v1600: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:17.408 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:17 smithi146 ceph-mon[103278]: pgmap v1601: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:17 smithi078 ceph-mon[91136]: pgmap v1601: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:17 smithi078 ceph-mon[99959]: pgmap v1601: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:19 smithi146 ceph-mon[103278]: pgmap v1602: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:19 smithi078 ceph-mon[99959]: pgmap v1602: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:19 smithi078 ceph-mon[91136]: pgmap v1602: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:20.309Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:20.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:20.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:13:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:13:21.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:21 smithi078 ceph-mon[91136]: pgmap v1603: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:21.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:21 smithi078 ceph-mon[99959]: pgmap v1603: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:21 smithi146 ceph-mon[103278]: pgmap v1603: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:23 smithi146 ceph-mon[103278]: pgmap v1604: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-mon[99959]: pgmap v1604: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-mon[91136]: pgmap v1604: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:23.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:23.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:23.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:25 smithi146 ceph-mon[103278]: pgmap v1605: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:25 smithi078 ceph-mon[99959]: pgmap v1605: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:25 smithi078 ceph-mon[91136]: pgmap v1605: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[99959]: pgmap v1606: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[91136]: pgmap v1606: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:27.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:27 smithi146 ceph-mon[103278]: pgmap v1606: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:29 smithi078 ceph-mon[91136]: pgmap v1607: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:29 smithi078 ceph-mon[99959]: pgmap v1607: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:29 smithi146 ceph-mon[103278]: pgmap v1607: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:30.310Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:30.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:30.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:13:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:13:31.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[91136]: pgmap v1608: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:31.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[99959]: pgmap v1608: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:31.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:31 smithi146 ceph-mon[103278]: pgmap v1608: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-mon[91136]: pgmap v1609: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-mon[99959]: pgmap v1609: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:33 smithi146 ceph-mon[103278]: pgmap v1609: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:33.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:33.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:33.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:33.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:35 smithi078 ceph-mon[91136]: pgmap v1610: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:35 smithi078 ceph-mon[99959]: pgmap v1610: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:35 smithi146 ceph-mon[103278]: pgmap v1610: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:37.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:37 smithi146 ceph-mon[103278]: pgmap v1611: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:37 smithi078 ceph-mon[91136]: pgmap v1611: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:37 smithi078 ceph-mon[99959]: pgmap v1611: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:38.113 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:13:38.639 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:13:38.639 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 64m - - 2024-09-20T21:13:38.639 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 64m - - 2024-09-20T21:13:38.639 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (58m) 12s ago 59m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:13:38.640 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (59m) 7s ago 59m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:13:38.640 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (51m) 12s ago 57m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:13:38.640 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (50m) 7s ago 63m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:13:38.640 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (52m) 12s ago 65m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:13:38.640 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (50m) 12s ago 65m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:13:38.640 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (49m) 7s ago 63m 99.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:13:38.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (49m) 12s ago 63m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:13:38.641 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (58m) 12s ago 58m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:13:38.641 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (58m) 7s ago 58m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:13:38.641 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (48m) 12s ago 63m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:13:38.641 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (47m) 12s ago 62m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:13:38.641 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (46m) 12s ago 62m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (46m) 12s ago 61m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (45m) 7s ago 61m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (44m) 7s ago 61m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (44m) 7s ago 60m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (43m) 7s ago 60m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (51m) 7s ago 59m 68.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:13:38.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (42m) 12s ago 57m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:13:38.643 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (42m) 7s ago 57m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:13:38.643 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (42m) 12s ago 57m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:13:38.643 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (42m) 7s ago 57m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:13:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:13:38.952 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:13:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:13:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:13:39.218 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:13:39.526 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:13:39.526 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:13:39.526 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:13:39.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:39 smithi078 ceph-mon[99959]: pgmap v1612: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:39.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:39 smithi078 ceph-mon[99959]: from='client.67598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:39.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1233699207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:39.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:39 smithi078 ceph-mon[91136]: pgmap v1612: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:39.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:39 smithi078 ceph-mon[91136]: from='client.67598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:39.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1233699207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:39 smithi146 ceph-mon[103278]: pgmap v1612: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:39 smithi146 ceph-mon[103278]: from='client.67598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1233699207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:13:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[99959]: from='client.67604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[99959]: from='client.67610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[99959]: from='client.67622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2572181040' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[91136]: from='client.67604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[91136]: from='client.67610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[91136]: from='client.67622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2572181040' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:40.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:40.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:40.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:40 smithi146 ceph-mon[103278]: from='client.67604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:40 smithi146 ceph-mon[103278]: from='client.67610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:40 smithi146 ceph-mon[103278]: from='client.67622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:13:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2572181040' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:13:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:13:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:13:41.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:41 smithi078 ceph-mon[91136]: pgmap v1613: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:41.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:41 smithi078 ceph-mon[99959]: pgmap v1613: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:41 smithi146 ceph-mon[103278]: pgmap v1613: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:43.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:43.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:43 smithi146 ceph-mon[103278]: pgmap v1614: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:43.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:43.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:43.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-mon[99959]: pgmap v1614: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:43.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:43.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:43.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-mon[91136]: pgmap v1614: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:43.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:43.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:43.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:43.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:45 smithi146 ceph-mon[103278]: pgmap v1615: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:45 smithi078 ceph-mon[91136]: pgmap v1615: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:45 smithi078 ceph-mon[99959]: pgmap v1615: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:47 smithi146 ceph-mon[103278]: pgmap v1616: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[91136]: pgmap v1616: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[99959]: pgmap v1616: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:49 smithi146 ceph-mon[103278]: pgmap v1617: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:49 smithi078 ceph-mon[91136]: pgmap v1617: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:49 smithi078 ceph-mon[99959]: pgmap v1617: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:50.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:50.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:50.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:13:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:13:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:51 smithi146 ceph-mon[103278]: pgmap v1618: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[91136]: pgmap v1618: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[99959]: pgmap v1618: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:13:52.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:52.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:52 smithi078 ceph-mon[99959]: pgmap v1619: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:52.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:52.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:52 smithi078 ceph-mon[91136]: pgmap v1619: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:13:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:52 smithi146 ceph-mon[103278]: pgmap v1619: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:53.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:53.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:13:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:13:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:13:53.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:13:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:54 smithi078 ceph-mon[91136]: pgmap v1620: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:54 smithi078 ceph-mon[99959]: pgmap v1620: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:54 smithi146 ceph-mon[103278]: pgmap v1620: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:13:57.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:56 smithi146 ceph-mon[103278]: pgmap v1621: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:56 smithi078 ceph-mon[91136]: pgmap v1621: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:56 smithi078 ceph-mon[99959]: pgmap v1621: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:59.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:58 smithi146 ceph-mon[103278]: pgmap v1622: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:59.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:58 smithi078 ceph-mon[91136]: pgmap v1622: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:13:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:58 smithi078 ceph-mon[99959]: pgmap v1622: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:13:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:00.312Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:00.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:00.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:00.944 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:14:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:14:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:00 smithi146 ceph-mon[103278]: pgmap v1623: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:00 smithi078 ceph-mon[91136]: pgmap v1623: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:00 smithi078 ceph-mon[99959]: pgmap v1623: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:02 smithi146 ceph-mon[103278]: pgmap v1624: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:02 smithi078 ceph-mon[91136]: pgmap v1624: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:02 smithi078 ceph-mon[99959]: pgmap v1624: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:03.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:03.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:03.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:04 smithi078 ceph-mon[91136]: pgmap v1625: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:04 smithi078 ceph-mon[99959]: pgmap v1625: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:04 smithi146 ceph-mon[103278]: pgmap v1625: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[91136]: pgmap v1626: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[99959]: pgmap v1626: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:06 smithi146 ceph-mon[103278]: pgmap v1626: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:08 smithi146 ceph-mon[103278]: pgmap v1627: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:08 smithi078 ceph-mon[91136]: pgmap v1627: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:08 smithi078 ceph-mon[99959]: pgmap v1627: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:09.824 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:14:10.363 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 64m - - 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 64m - - 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (58m) 3s ago 59m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (59m) 19s ago 59m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (52m) 3s ago 57m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (51m) 19s ago 63m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (53m) 3s ago 65m 550M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (51m) 3s ago 65m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (49m) 19s ago 64m 99.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (50m) 3s ago 64m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (59m) 3s ago 59m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:14:10.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (59m) 19s ago 59m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (49m) 3s ago 63m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (48m) 3s ago 63m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (47m) 3s ago 62m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (46m) 3s ago 62m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (46m) 19s ago 62m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (45m) 19s ago 61m 60.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (44m) 19s ago 61m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (43m) 19s ago 60m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (52m) 19s ago 60m 64.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (43m) 3s ago 58m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (43m) 19s ago 58m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (43m) 3s ago 58m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:14:10.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (43m) 19s ago 58m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:14:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:10.313Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:10.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:10.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:10.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:10.686 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:14:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:14:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:14:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:10.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:14:10.687 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:14:10.959 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:14:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:14:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:14:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:14:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:14:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:14:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:14:10.970 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:14:11.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:10 smithi146 ceph-mon[103278]: from='client.67634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:11.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:10 smithi146 ceph-mon[103278]: pgmap v1628: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:11.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:10 smithi146 ceph-mon[103278]: from='client.67640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:11.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2300123460' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[91136]: from='client.67634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[91136]: pgmap v1628: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[91136]: from='client.67640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2300123460' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[99959]: from='client.67634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[99959]: pgmap v1628: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[99959]: from='client.67640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2300123460' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:11.291 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:14:11.292 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:11.292 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:12 smithi146 ceph-mon[103278]: from='client.47727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:12 smithi146 ceph-mon[103278]: from='client.67658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1686528305' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[91136]: from='client.47727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[91136]: from='client.67658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1686528305' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[99959]: from='client.47727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[99959]: from='client.67658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:12.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1686528305' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:13 smithi146 ceph-mon[103278]: pgmap v1629: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-mon[91136]: pgmap v1629: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-mon[99959]: pgmap v1629: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:13.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:13.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:13.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:13.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:13.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:13.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:15.457 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:15 smithi078 ceph-mon[99959]: pgmap v1630: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:15.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:15 smithi078 ceph-mon[91136]: pgmap v1630: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:15 smithi146 ceph-mon[103278]: pgmap v1630: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:17 smithi146 ceph-mon[103278]: pgmap v1631: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:17 smithi078 ceph-mon[91136]: pgmap v1631: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:17 smithi078 ceph-mon[99959]: pgmap v1631: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:19 smithi146 ceph-mon[103278]: pgmap v1632: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:19 smithi078 ceph-mon[91136]: pgmap v1632: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:19 smithi078 ceph-mon[99959]: pgmap v1632: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:20.314Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:20.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:20.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:20.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:14:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:14:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:21 smithi146 ceph-mon[103278]: pgmap v1633: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:21 smithi078 ceph-mon[91136]: pgmap v1633: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:21 smithi078 ceph-mon[99959]: pgmap v1633: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:23 smithi146 ceph-mon[103278]: pgmap v1634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-mon[91136]: pgmap v1634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-mon[99959]: pgmap v1634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:23.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:23.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:23.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:25 smithi146 ceph-mon[103278]: pgmap v1635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:25 smithi078 ceph-mon[99959]: pgmap v1635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:25 smithi078 ceph-mon[91136]: pgmap v1635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 21:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T21:14:25.850585191Z level=info msg="Completed cleanup jobs" duration=15.459136ms 2024-09-20T21:14:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:27 smithi146 ceph-mon[103278]: pgmap v1636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[99959]: pgmap v1636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[91136]: pgmap v1636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:29 smithi146 ceph-mon[103278]: pgmap v1637: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:29 smithi078 ceph-mon[91136]: pgmap v1637: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:29 smithi078 ceph-mon[99959]: pgmap v1637: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:30.315Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:30.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:30.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:14:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:14:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:31 smithi146 ceph-mon[103278]: pgmap v1638: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:31 smithi078 ceph-mon[91136]: pgmap v1638: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:31 smithi078 ceph-mon[99959]: pgmap v1638: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:32 smithi146 ceph-mon[103278]: pgmap v1639: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[99959]: pgmap v1639: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:32 smithi078 ceph-mon[91136]: pgmap v1639: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:33.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:33.752Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:33.875 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:33.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:33.875 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:33.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:35.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:34 smithi078 ceph-mon[91136]: pgmap v1640: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:35.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:34 smithi078 ceph-mon[99959]: pgmap v1640: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:34 smithi146 ceph-mon[103278]: pgmap v1640: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:36 smithi078 ceph-mon[91136]: pgmap v1641: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:36 smithi078 ceph-mon[99959]: pgmap v1641: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:36 smithi146 ceph-mon[103278]: pgmap v1641: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:39.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:38 smithi146 ceph-mon[103278]: pgmap v1642: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:38 smithi078 ceph-mon[91136]: pgmap v1642: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:38 smithi078 ceph-mon[99959]: pgmap v1642: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:40.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:40.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:40.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:40.959 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:14:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:14:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:40 smithi146 ceph-mon[103278]: pgmap v1643: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:40 smithi078 ceph-mon[91136]: pgmap v1643: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:40 smithi078 ceph-mon[99959]: pgmap v1643: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:41.588 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 65m - - 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 65m - - 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (59m) 15s ago 60m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (60m) 10s ago 60m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (52m) 15s ago 58m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (51m) 10s ago 64m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (53m) 15s ago 66m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (51m) 15s ago 66m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (50m) 10s ago 64m 97.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (50m) 15s ago 64m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:14:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (60m) 15s ago 60m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (59m) 10s ago 59m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (49m) 15s ago 64m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (48m) 15s ago 63m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (47m) 15s ago 63m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (47m) 15s ago 63m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (46m) 10s ago 62m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (45m) 10s ago 62m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (45m) 10s ago 61m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (44m) 10s ago 61m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (53m) 10s ago 60m 64.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (43m) 15s ago 58m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (43m) 10s ago 58m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (43m) 15s ago 58m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:14:42.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (43m) 10s ago 58m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:42.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:14:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:14:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:14:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:14:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:14:42.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:14:42.438 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:14:42.707 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:14:42.707 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:14:42.707 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:14:42.707 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:14:42.707 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:14:42.707 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:14:42.708 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:14:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:14:43.023 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:14:43.023 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:14:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:42 smithi146 ceph-mon[103278]: from='client.47751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:42 smithi146 ceph-mon[103278]: from='client.47757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:42 smithi146 ceph-mon[103278]: pgmap v1644: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:42 smithi146 ceph-mon[103278]: from='client.47763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1545071242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[91136]: from='client.47751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[91136]: from='client.47757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[91136]: pgmap v1644: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[91136]: from='client.47763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1545071242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[99959]: from='client.47751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[99959]: from='client.47757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[99959]: pgmap v1644: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[99959]: from='client.47763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1545071242' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:14:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-mon[91136]: from='client.47775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4191664267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-mon[99959]: from='client.47775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4191664267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:43.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:43.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:43.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:43 smithi146 ceph-mon[103278]: from='client.47775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:14:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4191664267' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:14:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:44 smithi146 ceph-mon[103278]: pgmap v1645: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:44 smithi078 ceph-mon[91136]: pgmap v1645: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:44 smithi078 ceph-mon[99959]: pgmap v1645: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:46 smithi146 ceph-mon[103278]: pgmap v1646: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:46 smithi078 ceph-mon[91136]: pgmap v1646: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:46 smithi078 ceph-mon[99959]: pgmap v1646: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:48 smithi146 ceph-mon[103278]: pgmap v1647: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:48 smithi078 ceph-mon[99959]: pgmap v1647: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:48 smithi078 ceph-mon[91136]: pgmap v1647: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:50.317Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:50.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:50.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:50.993 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:14:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:14:51.259 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:50 smithi146 ceph-mon[103278]: pgmap v1648: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:50 smithi078 ceph-mon[91136]: pgmap v1648: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:50 smithi078 ceph-mon[99959]: pgmap v1648: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:52 smithi146 ceph-mon[103278]: pgmap v1649: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[91136]: pgmap v1649: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:52 smithi078 ceph-mon[99959]: pgmap v1649: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:53.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:53.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:14:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:14:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:14:53.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:14:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:54 smithi146 ceph-mon[103278]: pgmap v1650: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:54 smithi078 ceph-mon[91136]: pgmap v1650: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:54 smithi078 ceph-mon[99959]: pgmap v1650: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:14:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:56 smithi146 ceph-mon[103278]: pgmap v1651: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:56 smithi078 ceph-mon[91136]: pgmap v1651: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:56 smithi078 ceph-mon[99959]: pgmap v1651: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:59.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:58 smithi146 ceph-mon[103278]: pgmap v1652: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:59.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:58 smithi078 ceph-mon[91136]: pgmap v1652: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:58 smithi078 ceph-mon[99959]: pgmap v1652: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:00.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:00.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:00.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:00.965 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:15:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:15:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:00 smithi146 ceph-mon[103278]: pgmap v1653: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:00 smithi078 ceph-mon[91136]: pgmap v1653: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:00 smithi078 ceph-mon[99959]: pgmap v1653: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:02 smithi146 ceph-mon[103278]: pgmap v1654: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:02 smithi078 ceph-mon[91136]: pgmap v1654: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:02 smithi078 ceph-mon[99959]: pgmap v1654: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:03.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:03.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:03.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:04 smithi146 ceph-mon[103278]: pgmap v1655: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:04 smithi078 ceph-mon[91136]: pgmap v1655: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:04 smithi078 ceph-mon[99959]: pgmap v1655: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:06 smithi146 ceph-mon[103278]: pgmap v1656: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:06 smithi078 ceph-mon[99959]: pgmap v1656: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:06 smithi078 ceph-mon[91136]: pgmap v1656: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:09 smithi146 ceph-mon[103278]: pgmap v1657: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:09 smithi078 ceph-mon[91136]: pgmap v1657: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:09 smithi078 ceph-mon[99959]: pgmap v1657: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:10.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:10.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:10.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:10.905 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:15:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:15:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:11 smithi078 ceph-mon[91136]: pgmap v1658: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:11 smithi078 ceph-mon[99959]: pgmap v1658: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:11 smithi146 ceph-mon[103278]: pgmap v1658: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:12 smithi146 ceph-mon[103278]: pgmap v1659: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[99959]: pgmap v1659: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:12 smithi078 ceph-mon[91136]: pgmap v1659: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:13.321 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 65m - - 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 65m - - 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (59m) 6s ago 61m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (60m) 2s ago 60m 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (53m) 6s ago 58m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (52m) 2s ago 65m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (54m) 6s ago 66m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (52m) 6s ago 66m 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (51m) 2s ago 65m 99.2M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:15:13.853 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (51m) 6s ago 65m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (60m) 6s ago 60m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (60m) 2s ago 60m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (50m) 6s ago 64m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (49m) 6s ago 64m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (48m) 6s ago 63m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (47m) 6s ago 63m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (47m) 2s ago 63m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (46m) 2s ago 62m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (45m) 2s ago 62m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (44m) 2s ago 61m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (53m) 2s ago 61m 66.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (44m) 6s ago 59m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (44m) 2s ago 59m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:15:13.854 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (44m) 6s ago 59m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:15:13.855 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (44m) 2s ago 59m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:15:13.855 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:13.855 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-mon[99959]: from='client.47787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:13.855 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:13.855 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-mon[91136]: from='client.47787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:13.855 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:13.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:13.856 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:13.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:13.856 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:13.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:13 smithi146 ceph-mon[103278]: from='client.47787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:15:14.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:15:14.165 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:15:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:15:14.435 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:15:14.435 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:15:14.435 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:15:14.435 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:15:14.745 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:15:14.745 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:14.746 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:14.746 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:14.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[99959]: from='client.67712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[99959]: from='client.67718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[99959]: pgmap v1660: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:14.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3930013331' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:14.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[91136]: from='client.67712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[91136]: from='client.67718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[91136]: pgmap v1660: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:14.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3930013331' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:14.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:14 smithi146 ceph-mon[103278]: from='client.67712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:14 smithi146 ceph-mon[103278]: from='client.67718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:14.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:14 smithi146 ceph-mon[103278]: pgmap v1660: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:14.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3930013331' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:15 smithi146 ceph-mon[103278]: from='client.67730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2292662546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:15.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:15 smithi078 ceph-mon[91136]: from='client.67730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:15.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2292662546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:15.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:15 smithi078 ceph-mon[99959]: from='client.67730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:15.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2292662546' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:16.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:16 smithi078 ceph-mon[99959]: pgmap v1661: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:16.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:16 smithi078 ceph-mon[91136]: pgmap v1661: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:16 smithi146 ceph-mon[103278]: pgmap v1661: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:18 smithi146 ceph-mon[103278]: pgmap v1662: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:18 smithi078 ceph-mon[91136]: pgmap v1662: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:18 smithi078 ceph-mon[99959]: pgmap v1662: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:20.320Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:20.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:20.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:20.968 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:15:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:15:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:20 smithi146 ceph-mon[103278]: pgmap v1663: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:20 smithi078 ceph-mon[91136]: pgmap v1663: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:20 smithi078 ceph-mon[99959]: pgmap v1663: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:22 smithi146 ceph-mon[103278]: pgmap v1664: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:22 smithi078 ceph-mon[91136]: pgmap v1664: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:22 smithi078 ceph-mon[99959]: pgmap v1664: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:23.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:23.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:23.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:24 smithi146 ceph-mon[103278]: pgmap v1665: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:24 smithi078 ceph-mon[91136]: pgmap v1665: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:24 smithi078 ceph-mon[99959]: pgmap v1665: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:27.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:26 smithi078 ceph-mon[99959]: pgmap v1666: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:27.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:26 smithi078 ceph-mon[91136]: pgmap v1666: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:26 smithi146 ceph-mon[103278]: pgmap v1666: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:29 smithi146 ceph-mon[103278]: pgmap v1667: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:29 smithi078 ceph-mon[91136]: pgmap v1667: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:29 smithi078 ceph-mon[99959]: pgmap v1667: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:30.320Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:30.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:30.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:15:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:15:31.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:31 smithi078 ceph-mon[91136]: pgmap v1668: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:31.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:31 smithi078 ceph-mon[99959]: pgmap v1668: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:31 smithi146 ceph-mon[103278]: pgmap v1668: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:32 smithi146 ceph-mon[103278]: pgmap v1669: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[91136]: pgmap v1669: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:33.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:32 smithi078 ceph-mon[99959]: pgmap v1669: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:33.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:33.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:33.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:34 smithi146 ceph-mon[103278]: pgmap v1670: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:34 smithi078 ceph-mon[99959]: pgmap v1670: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:34 smithi078 ceph-mon[91136]: pgmap v1670: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:36 smithi146 ceph-mon[103278]: pgmap v1671: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:36 smithi078 ceph-mon[91136]: pgmap v1671: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:36 smithi078 ceph-mon[99959]: pgmap v1671: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:38 smithi146 ceph-mon[103278]: pgmap v1672: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:38 smithi078 ceph-mon[91136]: pgmap v1672: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:38 smithi078 ceph-mon[99959]: pgmap v1672: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:40.321Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:40.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:40.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:40.979 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:15:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:15:41.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:40 smithi146 ceph-mon[103278]: pgmap v1673: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:40 smithi078 ceph-mon[91136]: pgmap v1673: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:40 smithi078 ceph-mon[99959]: pgmap v1673: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:42 smithi146 ceph-mon[103278]: pgmap v1674: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:42 smithi078 ceph-mon[91136]: pgmap v1674: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:42 smithi078 ceph-mon[99959]: pgmap v1674: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:43.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:43.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:43.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:45.043 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:15:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:44 smithi146 ceph-mon[103278]: pgmap v1675: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:44 smithi078 ceph-mon[99959]: pgmap v1675: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:44 smithi078 ceph-mon[91136]: pgmap v1675: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:45.567 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:15:45.567 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 66m - - 2024-09-20T21:15:45.567 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 66m - - 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (60m) 18s ago 61m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (61m) 13s ago 61m 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (53m) 18s ago 59m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (52m) 13s ago 65m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (54m) 18s ago 67m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (52m) 18s ago 67m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (51m) 13s ago 65m 99.9M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (51m) 18s ago 65m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (61m) 18s ago 61m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (61m) 13s ago 61m 15.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (50m) 18s ago 65m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (49m) 18s ago 64m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (48m) 18s ago 64m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (48m) 18s ago 64m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:15:45.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (47m) 13s ago 63m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (46m) 13s ago 63m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (46m) 13s ago 62m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (45m) 13s ago 62m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (54m) 13s ago 61m 66.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (44m) 18s ago 59m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (44m) 13s ago 59m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (44m) 18s ago 59m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:15:45.569 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (44m) 13s ago 59m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:15:45.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:15:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:15:46.149 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:15:46.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:45 smithi078 ceph-mon[99959]: from='client.67742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:46.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:45 smithi078 ceph-mon[99959]: from='client.47829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:46.150 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3027959155' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:46.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:45 smithi078 ceph-mon[91136]: from='client.67742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:46.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:45 smithi078 ceph-mon[91136]: from='client.47829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:46.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3027959155' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:45 smithi146 ceph-mon[103278]: from='client.67742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:45 smithi146 ceph-mon[103278]: from='client.47829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3027959155' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:15:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:15:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:15:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:15:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:46 smithi146 ceph-mon[103278]: from='client.47835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:46 smithi146 ceph-mon[103278]: pgmap v1676: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:46 smithi146 ceph-mon[103278]: from='client.67766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1398076942' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[91136]: from='client.47835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[91136]: pgmap v1676: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[91136]: from='client.67766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1398076942' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[99959]: from='client.47835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[99959]: pgmap v1676: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[99959]: from='client.67766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:15:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1398076942' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:15:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:48 smithi146 ceph-mon[103278]: pgmap v1677: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[99959]: pgmap v1677: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:48 smithi078 ceph-mon[91136]: pgmap v1677: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:50.322Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:50.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:50.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:50.977 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:15:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:15:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:50 smithi146 ceph-mon[103278]: pgmap v1678: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:50 smithi078 ceph-mon[91136]: pgmap v1678: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:50 smithi078 ceph-mon[99959]: pgmap v1678: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:52 smithi146 ceph-mon[103278]: pgmap v1679: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[99959]: pgmap v1679: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:15:53.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:52 smithi078 ceph-mon[91136]: pgmap v1679: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:53.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:53.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:15:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:15:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:15:53.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:15:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:54 smithi146 ceph-mon[103278]: pgmap v1680: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:54 smithi078 ceph-mon[91136]: pgmap v1680: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:54 smithi078 ceph-mon[99959]: pgmap v1680: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:15:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:56 smithi146 ceph-mon[103278]: pgmap v1681: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:56 smithi078 ceph-mon[91136]: pgmap v1681: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:56 smithi078 ceph-mon[99959]: pgmap v1681: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:58 smithi146 ceph-mon[103278]: pgmap v1682: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:15:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:59.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:58 smithi078 ceph-mon[91136]: pgmap v1682: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:59.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:15:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:15:59.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:58 smithi078 ceph-mon[99959]: pgmap v1682: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:15:59.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:15:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:00.322Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:00.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:00.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:00.986 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:16:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:16:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:00 smithi146 ceph-mon[103278]: pgmap v1683: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:00 smithi078 ceph-mon[91136]: pgmap v1683: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:00 smithi078 ceph-mon[99959]: pgmap v1683: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:02 smithi146 ceph-mon[103278]: pgmap v1684: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:02 smithi078 ceph-mon[91136]: pgmap v1684: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:02 smithi078 ceph-mon[99959]: pgmap v1684: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:03.757Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:03.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:03.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:04 smithi146 ceph-mon[103278]: pgmap v1685: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:04 smithi078 ceph-mon[91136]: pgmap v1685: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:04 smithi078 ceph-mon[99959]: pgmap v1685: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:06 smithi146 ceph-mon[103278]: pgmap v1686: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:06 smithi078 ceph-mon[91136]: pgmap v1686: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:06 smithi078 ceph-mon[99959]: pgmap v1686: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:08 smithi146 ceph-mon[103278]: pgmap v1687: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[99959]: pgmap v1687: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:08 smithi078 ceph-mon[91136]: pgmap v1687: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:10.322Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:10.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:10.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:10.983 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:16:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:16:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:10 smithi146 ceph-mon[103278]: pgmap v1688: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:10 smithi078 ceph-mon[91136]: pgmap v1688: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:10 smithi078 ceph-mon[99959]: pgmap v1688: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:12 smithi146 ceph-mon[103278]: pgmap v1689: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[91136]: pgmap v1689: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[99959]: pgmap v1689: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:13.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:13.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:13.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:13.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:14 smithi146 ceph-mon[103278]: pgmap v1690: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:14 smithi078 ceph-mon[91136]: pgmap v1690: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:14 smithi078 ceph-mon[99959]: pgmap v1690: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:16.755 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:16:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:16 smithi078 ceph-mon[99959]: pgmap v1691: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:16 smithi078 ceph-mon[91136]: pgmap v1691: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:16 smithi146 ceph-mon[103278]: pgmap v1691: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 66m - - 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 66m - - 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (61m) 9s ago 62m 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (61m) 5s ago 61m 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (54m) 9s ago 59m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (53m) 5s ago 66m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (55m) 9s ago 67m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (53m) 9s ago 67m 152M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:16:17.281 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (52m) 5s ago 66m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (52m) 9s ago 66m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (61m) 9s ago 61m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (61m) 5s ago 61m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (51m) 9s ago 65m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (50m) 9s ago 65m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (49m) 9s ago 64m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (49m) 9s ago 64m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (48m) 5s ago 64m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (47m) 5s ago 63m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (46m) 5s ago 63m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (45m) 5s ago 63m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (54m) 5s ago 62m 68.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (45m) 9s ago 60m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (45m) 5s ago 60m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (45m) 9s ago 60m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:16:17.283 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (45m) 5s ago 60m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:16:17.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:16:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:16:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:16:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:16:17.601 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:16:17.875 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:16:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:16:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:16:17.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:16:18.202 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:16:18.202 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:18.202 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:18.202 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:18.202 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[91136]: from='client.47859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[91136]: from='client.47865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[91136]: from='client.67790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1192430298' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[99959]: from='client.47859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[99959]: from='client.47865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[99959]: from='client.67790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1192430298' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:17 smithi146 ceph-mon[103278]: from='client.47859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:17 smithi146 ceph-mon[103278]: from='client.47865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:17 smithi146 ceph-mon[103278]: from='client.67790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1192430298' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:19.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:18 smithi078 ceph-mon[91136]: from='client.47883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:19.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:18 smithi078 ceph-mon[91136]: pgmap v1692: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:19.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3107144603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:19.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:18 smithi078 ceph-mon[99959]: from='client.47883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:19.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:18 smithi078 ceph-mon[99959]: pgmap v1692: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:19.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3107144603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:18 smithi146 ceph-mon[103278]: from='client.47883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:18 smithi146 ceph-mon[103278]: pgmap v1692: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3107144603' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:20.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:20.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:20.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:20.993 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:16:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:16:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:20 smithi146 ceph-mon[103278]: pgmap v1693: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:20 smithi078 ceph-mon[91136]: pgmap v1693: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:20 smithi078 ceph-mon[99959]: pgmap v1693: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:22 smithi146 ceph-mon[103278]: pgmap v1694: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:22 smithi078 ceph-mon[99959]: pgmap v1694: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:22 smithi078 ceph-mon[91136]: pgmap v1694: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:23.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:23.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:23.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:24 smithi146 ceph-mon[103278]: pgmap v1695: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:24 smithi078 ceph-mon[91136]: pgmap v1695: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:24 smithi078 ceph-mon[99959]: pgmap v1695: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:27 smithi146 ceph-mon[103278]: pgmap v1696: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:27 smithi078 ceph-mon[99959]: pgmap v1696: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:27 smithi078 ceph-mon[91136]: pgmap v1696: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:28 smithi146 ceph-mon[103278]: pgmap v1697: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[91136]: pgmap v1697: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[99959]: pgmap v1697: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:30.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:30.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:30.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:30.990 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:16:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:16:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:30 smithi146 ceph-mon[103278]: pgmap v1698: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:30 smithi078 ceph-mon[99959]: pgmap v1698: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:30 smithi078 ceph-mon[91136]: pgmap v1698: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:33 smithi146 ceph-mon[103278]: pgmap v1699: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[91136]: pgmap v1699: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[99959]: pgmap v1699: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:33.759Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:33.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:33.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:35 smithi146 ceph-mon[103278]: pgmap v1700: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:35 smithi078 ceph-mon[91136]: pgmap v1700: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:35 smithi078 ceph-mon[99959]: pgmap v1700: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:37 smithi146 ceph-mon[103278]: pgmap v1701: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:37 smithi078 ceph-mon[91136]: pgmap v1701: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:37 smithi078 ceph-mon[99959]: pgmap v1701: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:39.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:39 smithi078 ceph-mon[91136]: pgmap v1702: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:39.336 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:39 smithi078 ceph-mon[99959]: pgmap v1702: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:39 smithi146 ceph-mon[103278]: pgmap v1702: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:40.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:40.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:40.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:40.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:40.952 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:16:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:16:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:41 smithi078 ceph-mon[91136]: pgmap v1703: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:41 smithi078 ceph-mon[99959]: pgmap v1703: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:41 smithi146 ceph-mon[103278]: pgmap v1703: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:43 smithi146 ceph-mon[103278]: pgmap v1704: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:43.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-mon[91136]: pgmap v1704: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-mon[99959]: pgmap v1704: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:43.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:43.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:43.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:45 smithi146 ceph-mon[103278]: pgmap v1705: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:45 smithi078 ceph-mon[99959]: pgmap v1705: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:45 smithi078 ceph-mon[91136]: pgmap v1705: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:47 smithi146 ceph-mon[103278]: pgmap v1706: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:47 smithi078 ceph-mon[91136]: pgmap v1706: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:47 smithi078 ceph-mon[99959]: pgmap v1706: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:48.492 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:16:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[91136]: pgmap v1707: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:48 smithi078 ceph-mon[99959]: pgmap v1707: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:48 smithi146 ceph-mon[103278]: pgmap v1707: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 67m - - 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 67m - - 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (61m) 1s ago 62m 30.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (62m) 16s ago 62m 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (54m) 1s ago 60m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (53m) 16s ago 66m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:16:49.022 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (55m) 1s ago 68m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (53m) 1s ago 68m 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (52m) 16s ago 67m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (53m) 1s ago 66m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (62m) 1s ago 62m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (62m) 16s ago 62m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (51m) 1s ago 66m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (50m) 1s ago 65m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (50m) 1s ago 65m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (49m) 1s ago 65m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (49m) 16s ago 64m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (47m) 16s ago 64m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (47m) 16s ago 63m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (46m) 16s ago 63m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (55m) 16s ago 62m 68.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (45m) 1s ago 60m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:16:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (45m) 16s ago 60m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:16:49.024 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (45m) 1s ago 60m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:16:49.024 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (45m) 16s ago 60m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:16:49.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:16:49.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:16:49.335 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:16:49.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:16:49.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:16:49.335 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:16:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:16:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:16:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:16:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:16:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:16:49.603 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:16:49.909 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:16:49.909 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:16:49.909 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:16:49.909 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[91136]: from='client.47895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[91136]: from='client.67820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[91136]: from='client.67826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/975696531' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[99959]: from='client.47895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[99959]: from='client.67820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[99959]: from='client.67826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:49.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/975696531' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:50.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:49 smithi146 ceph-mon[103278]: from='client.47895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:50.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:49 smithi146 ceph-mon[103278]: from='client.67820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:50.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:49 smithi146 ceph-mon[103278]: from='client.67826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:50.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/975696531' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:16:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:50.325Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:50.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:50.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:50 smithi146 ceph-mon[103278]: from='client.67838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3531903223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:50 smithi146 ceph-mon[103278]: pgmap v1708: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-mon[99959]: from='client.67838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3531903223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-mon[99959]: pgmap v1708: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-mon[91136]: from='client.67838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3531903223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-mon[91136]: pgmap v1708: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:51.038 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:16:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:16:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:53 smithi146 ceph-mon[103278]: pgmap v1709: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[91136]: pgmap v1709: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[99959]: pgmap v1709: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:16:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:53.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:53.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:16:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:16:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:16:53.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:16:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:55 smithi146 ceph-mon[103278]: pgmap v1710: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:55 smithi078 ceph-mon[99959]: pgmap v1710: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:55 smithi078 ceph-mon[91136]: pgmap v1710: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:16:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:57 smithi146 ceph-mon[103278]: pgmap v1711: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:57 smithi078 ceph-mon[91136]: pgmap v1711: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:57 smithi078 ceph-mon[99959]: pgmap v1711: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:16:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:16:59 smithi146 ceph-mon[103278]: pgmap v1712: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:16:59 smithi078 ceph-mon[99959]: pgmap v1712: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:16:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:16:59 smithi078 ceph-mon[91136]: pgmap v1712: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:00.326Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:00.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:00.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:17:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:17:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:01 smithi146 ceph-mon[103278]: pgmap v1713: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:01 smithi078 ceph-mon[91136]: pgmap v1713: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:01 smithi078 ceph-mon[99959]: pgmap v1713: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:03.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:03.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:03 smithi146 ceph-mon[103278]: pgmap v1714: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-mon[91136]: pgmap v1714: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-mon[99959]: pgmap v1714: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:03.761Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:03.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:03.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:05 smithi146 ceph-mon[103278]: pgmap v1715: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:05 smithi078 ceph-mon[91136]: pgmap v1715: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:05 smithi078 ceph-mon[99959]: pgmap v1715: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:07 smithi146 ceph-mon[103278]: pgmap v1716: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:07 smithi078 ceph-mon[91136]: pgmap v1716: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:07 smithi078 ceph-mon[99959]: pgmap v1716: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:08 smithi146 ceph-mon[103278]: pgmap v1717: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[91136]: pgmap v1717: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:08 smithi078 ceph-mon[99959]: pgmap v1717: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:10.327Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:10.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:10.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:11.004 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:17:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:17:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:11 smithi146 ceph-mon[103278]: pgmap v1718: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:11 smithi078 ceph-mon[99959]: pgmap v1718: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:11 smithi078 ceph-mon[91136]: pgmap v1718: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:13 smithi146 ceph-mon[103278]: pgmap v1719: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[99959]: pgmap v1719: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[91136]: pgmap v1719: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:13.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:13.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:13.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:15 smithi146 ceph-mon[103278]: pgmap v1720: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:15 smithi078 ceph-mon[91136]: pgmap v1720: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:15 smithi078 ceph-mon[99959]: pgmap v1720: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:17 smithi078 ceph-mon[91136]: pgmap v1721: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:17 smithi078 ceph-mon[99959]: pgmap v1721: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:17 smithi146 ceph-mon[103278]: pgmap v1721: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:19 smithi078 ceph-mon[91136]: pgmap v1722: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:19.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:19 smithi078 ceph-mon[99959]: pgmap v1722: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:19 smithi146 ceph-mon[103278]: pgmap v1722: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:20.204 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:17:20.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:20.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:20.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:20.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:20.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:20.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:20.756 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:17:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:17:20.756 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:17:20.756 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 67m - - 2024-09-20T21:17:20.756 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 67m - - 2024-09-20T21:17:20.756 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (62m) 12s ago 63m 30.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:17:20.756 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (62m) 8s ago 62m 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:17:20.756 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (55m) 12s ago 60m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (54m) 8s ago 67m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (56m) 12s ago 68m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (54m) 12s ago 69m 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (53m) 8s ago 67m 100M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (53m) 12s ago 67m 105M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (62m) 12s ago 62m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (62m) 8s ago 62m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (52m) 12s ago 66m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (51m) 12s ago 66m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (50m) 12s ago 66m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (50m) 12s ago 65m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (49m) 8s ago 65m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (48m) 8s ago 64m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:17:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (47m) 8s ago 64m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:17:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (46m) 8s ago 64m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:17:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (55m) 8s ago 63m 69.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:17:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (46m) 12s ago 61m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:17:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (46m) 8s ago 61m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:17:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (46m) 12s ago 61m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:17:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (46m) 8s ago 61m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:17:21.139 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:17:21.139 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:17:21.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:17:21.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:17:21.140 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:17:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:21 smithi078 ceph-mon[91136]: pgmap v1723: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:21 smithi078 ceph-mon[91136]: from='client.47931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:21 smithi078 ceph-mon[99959]: pgmap v1723: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:21 smithi078 ceph-mon[99959]: from='client.47931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:21.419 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:17:21.419 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:17:21.419 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:17:21.419 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:17:21.419 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:17:21.420 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:17:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:21 smithi146 ceph-mon[103278]: pgmap v1723: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:21 smithi146 ceph-mon[103278]: from='client.47931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:21.734 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:17:21.734 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:21.734 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:21.735 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:21.735 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[91136]: from='client.47937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[91136]: from='client.67862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1614002303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1647624849' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[99959]: from='client.47937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[99959]: from='client.67862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1614002303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1647624849' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:22 smithi146 ceph-mon[103278]: from='client.47937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:22 smithi146 ceph-mon[103278]: from='client.67862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1614002303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1647624849' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:23 smithi146 ceph-mon[103278]: from='client.47955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:23 smithi146 ceph-mon[103278]: pgmap v1724: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-mon[91136]: from='client.47955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-mon[91136]: pgmap v1724: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-mon[99959]: from='client.47955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-mon[99959]: pgmap v1724: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:23.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:23.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:23.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:25 smithi146 ceph-mon[103278]: pgmap v1725: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:25 smithi078 ceph-mon[91136]: pgmap v1725: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:25 smithi078 ceph-mon[99959]: pgmap v1725: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:27 smithi146 ceph-mon[103278]: pgmap v1726: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:27 smithi078 ceph-mon[91136]: pgmap v1726: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:27 smithi078 ceph-mon[99959]: pgmap v1726: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:28 smithi146 ceph-mon[103278]: pgmap v1727: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[91136]: pgmap v1727: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[99959]: pgmap v1727: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:29.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:30.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:30.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:30.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:30.991 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:17:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:17:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:30 smithi146 ceph-mon[103278]: pgmap v1728: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:30 smithi078 ceph-mon[91136]: pgmap v1728: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:30 smithi078 ceph-mon[99959]: pgmap v1728: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:32 smithi146 ceph-mon[103278]: pgmap v1729: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[91136]: pgmap v1729: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[99959]: pgmap v1729: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:33.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:33.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:33.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:33.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:34 smithi146 ceph-mon[103278]: pgmap v1730: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:34 smithi078 ceph-mon[91136]: pgmap v1730: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:34 smithi078 ceph-mon[99959]: pgmap v1730: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:37 smithi146 ceph-mon[103278]: pgmap v1731: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:36 smithi078 ceph-mon[91136]: pgmap v1731: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:37 smithi078 ceph-mon[99959]: pgmap v1731: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:39 smithi146 ceph-mon[103278]: pgmap v1732: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:39 smithi078 ceph-mon[91136]: pgmap v1732: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:39 smithi078 ceph-mon[99959]: pgmap v1732: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:40.329Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:40.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:40.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:41.007 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:17:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:17:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:41 smithi146 ceph-mon[103278]: pgmap v1733: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:41.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:41 smithi078 ceph-mon[91136]: pgmap v1733: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:41.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:41 smithi078 ceph-mon[99959]: pgmap v1733: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:43 smithi146 ceph-mon[103278]: pgmap v1734: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-mon[99959]: pgmap v1734: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-mon[91136]: pgmap v1734: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:43.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:43.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:44.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:43.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:44.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:45.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:45 smithi146 ceph-mon[103278]: pgmap v1735: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:45 smithi078 ceph-mon[91136]: pgmap v1735: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:45 smithi078 ceph-mon[99959]: pgmap v1735: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:47 smithi078 ceph-mon[91136]: pgmap v1736: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:47 smithi078 ceph-mon[99959]: pgmap v1736: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:47 smithi146 ceph-mon[103278]: pgmap v1736: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:49 smithi146 ceph-mon[103278]: pgmap v1737: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[91136]: pgmap v1737: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[99959]: pgmap v1737: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:49.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:49.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:50.330Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:50.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:50.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:50.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:17:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:17:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:51 smithi078 ceph-mon[91136]: pgmap v1738: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:51 smithi078 ceph-mon[99959]: pgmap v1738: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:51 smithi146 ceph-mon[103278]: pgmap v1738: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 68m - - 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 68m - - 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (62m) 4s ago 63m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (63m) 20s ago 63m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (56m) 4s ago 61m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (54m) 20s ago 67m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (56m) 4s ago 69m 551M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (55m) 4s ago 69m 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (53m) 20s ago 68m 101M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (54m) 4s ago 67m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (63m) 4s ago 63m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:17:52.554 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (63m) 20s ago 63m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (52m) 4s ago 67m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (51m) 4s ago 66m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (51m) 4s ago 66m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (50m) 4s ago 66m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (50m) 20s ago 65m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (49m) 20s ago 65m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (48m) 20s ago 65m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (47m) 20s ago 64m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (56m) 20s ago 64m 69.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (47m) 4s ago 61m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (46m) 20s ago 61m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (46m) 4s ago 61m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:17:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (46m) 20s ago 61m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:17:52.867 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:17:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:17:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:17:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:17:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:17:52.869 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:17:53.137 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:17:53.137 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:17:53.137 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:17:53.138 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:17:53.445 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:17:53.445 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:17:53.446 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: pgmap v1739: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='client.67886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='client.67892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/616146756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: pgmap v1739: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='client.67886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='client.67892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/616146756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: pgmap v1739: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='client.67886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='client.67892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:17:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/616146756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:17:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:53.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:53.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:17:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:17:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:17:53.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:17:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:54 smithi146 ceph-mon[103278]: from='client.67898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:54 smithi146 ceph-mon[103278]: from='client.67910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2853739928' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:54 smithi078 ceph-mon[91136]: from='client.67898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:54 smithi078 ceph-mon[91136]: from='client.67910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2853739928' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:54 smithi078 ceph-mon[99959]: from='client.67898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:54 smithi078 ceph-mon[99959]: from='client.67910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:17:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2853739928' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:17:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:55 smithi146 ceph-mon[103278]: pgmap v1740: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:55 smithi078 ceph-mon[91136]: pgmap v1740: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:55 smithi078 ceph-mon[99959]: pgmap v1740: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:17:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:57 smithi146 ceph-mon[103278]: pgmap v1741: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:57 smithi078 ceph-mon[91136]: pgmap v1741: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:57 smithi078 ceph-mon[99959]: pgmap v1741: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:59 smithi146 ceph-mon[103278]: pgmap v1742: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:17:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:59 smithi078 ceph-mon[91136]: pgmap v1742: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:17:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:17:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:59 smithi078 ceph-mon[99959]: pgmap v1742: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:17:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:17:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:00.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:00.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:00.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:00.905 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:18:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:18:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:01 smithi078 ceph-mon[91136]: pgmap v1743: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:01 smithi078 ceph-mon[99959]: pgmap v1743: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:01 smithi146 ceph-mon[103278]: pgmap v1743: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:03.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:03.455 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-mon[99959]: pgmap v1744: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:03.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:03.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-mon[91136]: pgmap v1744: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:03 smithi146 ceph-mon[103278]: pgmap v1744: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:03.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:03.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:03.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:05.303 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:05 smithi146 ceph-mon[103278]: pgmap v1745: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:05 smithi078 ceph-mon[91136]: pgmap v1745: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:05 smithi078 ceph-mon[99959]: pgmap v1745: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:07 smithi146 ceph-mon[103278]: pgmap v1746: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:07 smithi078 ceph-mon[91136]: pgmap v1746: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:07 smithi078 ceph-mon[99959]: pgmap v1746: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:09 smithi146 ceph-mon[103278]: pgmap v1747: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[91136]: pgmap v1747: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[99959]: pgmap v1747: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:10.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:10.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:10.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:18:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:18:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:11 smithi146 ceph-mon[103278]: pgmap v1748: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:11 smithi078 ceph-mon[99959]: pgmap v1748: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:11 smithi078 ceph-mon[91136]: pgmap v1748: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:13 smithi146 ceph-mon[103278]: pgmap v1749: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[99959]: pgmap v1749: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[91136]: pgmap v1749: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:13.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:13.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:13.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:15 smithi146 ceph-mon[103278]: pgmap v1750: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:15 smithi078 ceph-mon[91136]: pgmap v1750: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:15 smithi078 ceph-mon[99959]: pgmap v1750: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:17 smithi146 ceph-mon[103278]: pgmap v1751: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:17 smithi078 ceph-mon[91136]: pgmap v1751: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:17 smithi078 ceph-mon[99959]: pgmap v1751: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:19 smithi146 ceph-mon[103278]: pgmap v1752: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:19 smithi078 ceph-mon[99959]: pgmap v1752: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:19 smithi078 ceph-mon[91136]: pgmap v1752: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:20.332Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:20.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:20.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:18:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:18:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:21 smithi078 ceph-mon[91136]: pgmap v1753: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:21 smithi078 ceph-mon[99959]: pgmap v1753: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:21 smithi146 ceph-mon[103278]: pgmap v1753: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:23.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:23.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-mon[99959]: pgmap v1754: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:23.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-mon[91136]: pgmap v1754: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:23.755 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:18:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:23 smithi146 ceph-mon[103278]: pgmap v1754: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:23.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:23.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:23.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:24.289 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:18:24.289 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 69m - - 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 68m - - 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (63m) 16s ago 64m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (64m) 11s ago 64m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (56m) 16s ago 61m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (55m) 11s ago 68m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (57m) 16s ago 69m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (55m) 16s ago 70m 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (54m) 11s ago 68m 102M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (54m) 16s ago 68m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (63m) 16s ago 63m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (63m) 11s ago 63m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (53m) 16s ago 67m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (52m) 16s ago 67m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:18:24.290 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (51m) 16s ago 67m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (51m) 16s ago 66m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (50m) 11s ago 66m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (49m) 11s ago 65m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (48m) 11s ago 65m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (47m) 11s ago 65m 59.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (56m) 11s ago 64m 69.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (47m) 16s ago 62m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (47m) 11s ago 62m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (47m) 16s ago 62m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:18:24.291 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (47m) 11s ago 62m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:18:24.602 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:18:24.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:18:24.602 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:18:24.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:18:24.604 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:18:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:18:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:18:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:18:24.875 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:18:24.875 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:18:25.183 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:18:25.184 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:25.184 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:25.184 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:25.184 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[91136]: from='client.48003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[91136]: pgmap v1755: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[91136]: from='client.48009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[91136]: from='client.48015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4036891733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1804613304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[99959]: from='client.48003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[99959]: pgmap v1755: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:25.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[99959]: from='client.48009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[99959]: from='client.48015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4036891733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:25.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1804613304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:25.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:25 smithi146 ceph-mon[103278]: from='client.48003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:25 smithi146 ceph-mon[103278]: pgmap v1755: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:25.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:25 smithi146 ceph-mon[103278]: from='client.48009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:25 smithi146 ceph-mon[103278]: from='client.48015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:25.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4036891733' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:25.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1804613304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:26 smithi078 ceph-mon[99959]: from='client.48027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:26 smithi078 ceph-mon[91136]: from='client.48027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:26.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:26 smithi146 ceph-mon[103278]: from='client.48027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:27 smithi078 ceph-mon[91136]: pgmap v1756: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:27 smithi078 ceph-mon[99959]: pgmap v1756: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:27 smithi146 ceph-mon[103278]: pgmap v1756: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:29 smithi146 ceph-mon[103278]: pgmap v1757: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[91136]: pgmap v1757: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[99959]: pgmap v1757: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:29.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:30.333Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:30.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:30.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:18:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:18:31.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:31 smithi078 ceph-mon[91136]: pgmap v1758: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:31.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:31 smithi078 ceph-mon[99959]: pgmap v1758: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:31 smithi146 ceph-mon[103278]: pgmap v1758: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:33 smithi146 ceph-mon[103278]: pgmap v1759: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[91136]: pgmap v1759: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[99959]: pgmap v1759: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:33.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:33.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:33.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:33.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:35 smithi146 ceph-mon[103278]: pgmap v1760: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:35 smithi078 ceph-mon[91136]: pgmap v1760: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:35 smithi078 ceph-mon[99959]: pgmap v1760: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:37 smithi146 ceph-mon[103278]: pgmap v1761: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:37 smithi078 ceph-mon[91136]: pgmap v1761: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:37 smithi078 ceph-mon[99959]: pgmap v1761: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:39 smithi146 ceph-mon[103278]: pgmap v1762: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:39 smithi078 ceph-mon[91136]: pgmap v1762: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:39 smithi078 ceph-mon[99959]: pgmap v1762: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:40.334Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:40.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:40.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:18:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:18:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:41 smithi078 ceph-mon[91136]: pgmap v1763: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:41 smithi078 ceph-mon[99959]: pgmap v1763: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:41 smithi146 ceph-mon[103278]: pgmap v1763: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:43.564 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:43.564 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-mon[99959]: pgmap v1764: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:43.565 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:43.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:43.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-mon[91136]: pgmap v1764: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:43.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:43.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:43.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:43 smithi146 ceph-mon[103278]: pgmap v1764: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:43.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:43.867 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:43.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:43.867 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:43.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:43.867 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:43.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:45.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:45 smithi146 ceph-mon[103278]: pgmap v1765: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:45 smithi078 ceph-mon[91136]: pgmap v1765: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:45 smithi078 ceph-mon[99959]: pgmap v1765: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:47 smithi146 ceph-mon[103278]: pgmap v1766: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:47 smithi078 ceph-mon[91136]: pgmap v1766: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:47 smithi078 ceph-mon[99959]: pgmap v1766: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:49 smithi146 ceph-mon[103278]: pgmap v1767: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[91136]: pgmap v1767: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[99959]: pgmap v1767: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:50.334Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:50.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:50.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:18:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:18:51.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:51 smithi078 ceph-mon[91136]: pgmap v1768: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:51.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:51 smithi078 ceph-mon[99959]: pgmap v1768: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:51 smithi146 ceph-mon[103278]: pgmap v1768: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:53 smithi146 ceph-mon[103278]: pgmap v1769: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:53.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[99959]: pgmap v1769: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[91136]: pgmap v1769: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:53.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:18:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:53.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:53.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:18:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:18:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:18:53.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:18:55.479 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:18:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:55 smithi146 ceph-mon[103278]: pgmap v1770: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:55 smithi078 ceph-mon[99959]: pgmap v1770: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:55 smithi078 ceph-mon[91136]: pgmap v1770: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 69m - - 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 69m - - 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (63m) 7s ago 64m 30.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (64m) 2s ago 64m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (57m) 7s ago 62m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:18:56.003 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (55m) 2s ago 68m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (57m) 7s ago 70m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (56m) 7s ago 70m 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (54m) 2s ago 69m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (55m) 7s ago 69m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (64m) 7s ago 64m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (64m) 2s ago 64m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (53m) 7s ago 68m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (52m) 7s ago 68m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (52m) 7s ago 67m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (51m) 7s ago 67m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:18:56.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (51m) 2s ago 66m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (50m) 2s ago 66m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (49m) 2s ago 66m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (48m) 2s ago 65m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (57m) 2s ago 65m 69.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (48m) 7s ago 62m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (47m) 2s ago 62m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (47m) 7s ago 62m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:18:56.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (47m) 2s ago 62m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:56.313 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:18:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:18:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:18:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:18:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:18:56.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:18:56.314 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:18:56.585 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:18:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:18:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:18:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:18:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:18:56.586 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:18:56.586 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:18:56.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4274678094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:56.587 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4274678094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4274678094' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:18:56.895 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:18:56.895 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:18:56.895 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:18:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:57 smithi146 ceph-mon[103278]: from='client.48039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:57 smithi146 ceph-mon[103278]: from='client.67964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:57 smithi146 ceph-mon[103278]: pgmap v1771: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:57 smithi146 ceph-mon[103278]: from='client.48051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1963888593' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[91136]: from='client.48039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[91136]: from='client.67964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[91136]: pgmap v1771: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[91136]: from='client.48051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1963888593' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[99959]: from='client.48039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[99959]: from='client.67964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[99959]: pgmap v1771: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:57.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[99959]: from='client.48051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:57.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1963888593' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:18:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:58 smithi146 ceph-mon[103278]: from='client.67982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:58 smithi078 ceph-mon[91136]: from='client.67982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:58 smithi078 ceph-mon[99959]: from='client.67982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:18:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:18:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:18:59 smithi146 ceph-mon[103278]: pgmap v1772: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:18:59 smithi078 ceph-mon[91136]: pgmap v1772: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:18:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:18:59 smithi078 ceph-mon[99959]: pgmap v1772: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:19:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:00.334Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:00.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:00.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:19:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:19:01.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:01 smithi078 ceph-mon[91136]: pgmap v1773: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:01.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:01 smithi078 ceph-mon[99959]: pgmap v1773: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:01 smithi146 ceph-mon[103278]: pgmap v1773: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-mon[91136]: pgmap v1774: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:19:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-mon[99959]: pgmap v1774: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:19:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:03 smithi146 ceph-mon[103278]: pgmap v1774: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:19:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:03.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:03.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:03.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:05.620 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:05 smithi078 ceph-mon[99959]: pgmap v1775: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:05.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:05 smithi078 ceph-mon[91136]: pgmap v1775: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:05.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:05 smithi146 ceph-mon[103278]: pgmap v1775: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:07 smithi146 ceph-mon[103278]: pgmap v1776: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:07 smithi078 ceph-mon[91136]: pgmap v1776: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:07 smithi078 ceph-mon[99959]: pgmap v1776: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:09 smithi146 ceph-mon[103278]: pgmap v1777: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[91136]: pgmap v1777: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[99959]: pgmap v1777: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:10.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:10.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:10.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:10.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:19:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:19:11.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:11 smithi078 ceph-mon[91136]: pgmap v1778: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:11.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:11 smithi078 ceph-mon[99959]: pgmap v1778: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:11 smithi146 ceph-mon[103278]: pgmap v1778: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:12 smithi146 ceph-mon[103278]: pgmap v1779: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:12 smithi078 ceph-mon[91136]: pgmap v1779: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:12 smithi078 ceph-mon[99959]: pgmap v1779: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:13.770Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:13.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:13.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:14.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:15 smithi146 ceph-mon[103278]: pgmap v1780: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:15 smithi078 ceph-mon[99959]: pgmap v1780: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:15 smithi078 ceph-mon[91136]: pgmap v1780: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:17 smithi146 ceph-mon[103278]: pgmap v1781: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:17 smithi078 ceph-mon[99959]: pgmap v1781: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:17 smithi078 ceph-mon[91136]: pgmap v1781: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:19 smithi146 ceph-mon[103278]: pgmap v1782: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:19 smithi078 ceph-mon[99959]: pgmap v1782: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:19 smithi078 ceph-mon[91136]: pgmap v1782: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:20.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:20.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:20.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:19:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:19:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:21 smithi146 ceph-mon[103278]: pgmap v1783: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:21 smithi078 ceph-mon[91136]: pgmap v1783: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:21 smithi078 ceph-mon[99959]: pgmap v1783: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:23 smithi146 ceph-mon[103278]: pgmap v1784: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-mon[91136]: pgmap v1784: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-mon[99959]: pgmap v1784: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:24.001 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:23.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:24.002 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:23.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:24.002 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:23.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:25.313 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:25 smithi078 ceph-mon[99959]: pgmap v1785: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:25.313 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:25 smithi078 ceph-mon[91136]: pgmap v1785: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:25 smithi146 ceph-mon[103278]: pgmap v1785: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:27.196 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:19:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:27 smithi078 ceph-mon[99959]: pgmap v1786: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:27 smithi078 ceph-mon[91136]: pgmap v1786: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:27.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:27 smithi146 ceph-mon[103278]: pgmap v1786: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 70m - - 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 69m - - 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (64m) 19s ago 65m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (65m) 14s ago 65m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (57m) 19s ago 63m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (56m) 14s ago 69m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (58m) 19s ago 70m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (56m) 19s ago 71m 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (55m) 14s ago 69m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:19:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (55m) 19s ago 69m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (64m) 19s ago 64m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (64m) 14s ago 64m 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (54m) 19s ago 68m 66.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (53m) 19s ago 68m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (52m) 19s ago 68m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (52m) 19s ago 67m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (51m) 14s ago 67m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (50m) 14s ago 66m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (49m) 14s ago 66m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (48m) 14s ago 66m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (57m) 14s ago 65m 69.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (48m) 19s ago 63m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (48m) 14s ago 63m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (48m) 19s ago 63m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:19:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (48m) 14s ago 63m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:19:28.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:19:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:19:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:19:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:19:28.324 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:19:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:19:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:19:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:19:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:19:28.325 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:19:28.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:28 smithi078 ceph-mon[99959]: from='client.48075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:28.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3860614557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:28.325 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:28.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:28 smithi078 ceph-mon[91136]: from='client.48075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:28.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3860614557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:28.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:28 smithi146 ceph-mon[103278]: from='client.48075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3860614557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:19:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:28.634 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:19:28.634 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:19:28.634 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:19:28.634 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:19:28.634 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='client.68000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='client.68006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: pgmap v1787: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='client.68018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1126471840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='client.68000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='client.68006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: pgmap v1787: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='client.68018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1126471840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='client.68000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='client.68006 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: pgmap v1787: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='client.68018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1126471840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:30.336Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:30.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:30.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:30.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:19:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:19:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:31 smithi146 ceph-mon[103278]: pgmap v1788: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:31 smithi078 ceph-mon[91136]: pgmap v1788: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:31 smithi078 ceph-mon[99959]: pgmap v1788: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:33 smithi146 ceph-mon[103278]: pgmap v1789: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-mon[91136]: pgmap v1789: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-mon[99959]: pgmap v1789: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:33.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:33.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:33.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:35 smithi078 ceph-mon[91136]: pgmap v1790: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:35 smithi078 ceph-mon[99959]: pgmap v1790: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:35 smithi146 ceph-mon[103278]: pgmap v1790: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:37 smithi078 ceph-mon[91136]: pgmap v1791: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:37 smithi078 ceph-mon[99959]: pgmap v1791: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:37 smithi146 ceph-mon[103278]: pgmap v1791: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:39.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:39 smithi146 ceph-mon[103278]: pgmap v1792: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:39 smithi078 ceph-mon[91136]: pgmap v1792: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:39 smithi078 ceph-mon[99959]: pgmap v1792: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:40.337Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:40.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:40.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:19:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:19:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:41 smithi078 ceph-mon[99959]: pgmap v1793: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:41 smithi078 ceph-mon[91136]: pgmap v1793: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:41 smithi146 ceph-mon[103278]: pgmap v1793: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:43.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:43.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:43 smithi146 ceph-mon[103278]: pgmap v1794: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:43.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:43.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:43.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-mon[99959]: pgmap v1794: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:43.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:43.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:43.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-mon[91136]: pgmap v1794: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:43.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:43.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:43.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:43.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:45.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:45 smithi078 ceph-mon[91136]: pgmap v1795: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:45.633 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:45 smithi078 ceph-mon[99959]: pgmap v1795: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:45 smithi146 ceph-mon[103278]: pgmap v1795: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:47.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:47 smithi146 ceph-mon[103278]: pgmap v1796: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:47 smithi078 ceph-mon[91136]: pgmap v1796: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:47 smithi078 ceph-mon[99959]: pgmap v1796: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:49 smithi146 ceph-mon[103278]: pgmap v1797: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[99959]: pgmap v1797: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[91136]: pgmap v1797: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:49.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:50.337Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:50.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:50.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:19:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:19:51.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:51 smithi078 ceph-mon[91136]: pgmap v1798: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:51.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:51 smithi078 ceph-mon[99959]: pgmap v1798: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:51 smithi146 ceph-mon[103278]: pgmap v1798: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:53 smithi146 ceph-mon[103278]: pgmap v1799: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:53.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:53.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-mon[99959]: pgmap v1799: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:53.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:19:53.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-mon[91136]: pgmap v1799: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:53.774Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:53.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:19:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:19:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:19:53.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:19:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:54 smithi146 ceph-mon[103278]: pgmap v1800: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[99959]: pgmap v1800: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:19:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:54 smithi078 ceph-mon[91136]: pgmap v1800: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:57 smithi146 ceph-mon[103278]: pgmap v1801: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:57 smithi078 ceph-mon[91136]: pgmap v1801: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:57 smithi078 ceph-mon[99959]: pgmap v1801: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:19:58.932 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:19:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:59 smithi078 ceph-mon[99959]: pgmap v1802: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:19:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:59 smithi078 ceph-mon[91136]: pgmap v1802: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:19:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 70m - - 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 70m - - 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (64m) 10s ago 65m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (65m) 6s ago 65m 86.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (58m) 10s ago 63m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (56m) 6s ago 69m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (58m) 10s ago 71m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:19:59.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (57m) 10s ago 71m 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (55m) 6s ago 70m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (56m) 10s ago 70m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (65m) 10s ago 65m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (65m) 6s ago 65m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (54m) 10s ago 69m 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (53m) 10s ago 69m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (53m) 10s ago 68m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (52m) 10s ago 68m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (52m) 6s ago 67m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (51m) 6s ago 67m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (50m) 6s ago 67m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (49m) 6s ago 66m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (58m) 6s ago 66m 69.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (49m) 10s ago 63m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:19:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (48m) 6s ago 64m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:19:59.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (49m) 10s ago 63m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:19:59.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (48m) 6s ago 63m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:19:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:59 smithi146 ceph-mon[103278]: pgmap v1802: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:19:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:19:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:20:00.035 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:20:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:20:00.036 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:20:00.036 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:20:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T21:19:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[99959]: from='client.68030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[99959]: from='client.48117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1382693391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[91136]: from='client.68030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[91136]: from='client.48117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1382693391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.348 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.349 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:00.349 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:00.349 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.349 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:00 smithi146 ceph-mon[103278]: from='client.68030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:00 smithi146 ceph-mon[103278]: from='client.48117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1382693391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:00.339Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:00.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:00.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:20:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:20:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:01 smithi146 ceph-mon[103278]: from='client.48123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:01 smithi146 ceph-mon[103278]: pgmap v1803: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:01 smithi146 ceph-mon[103278]: from='client.68054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/412983698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[91136]: from='client.48123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[91136]: pgmap v1803: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[91136]: from='client.68054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/412983698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[99959]: from='client.48123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[99959]: pgmap v1803: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[99959]: from='client.68054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/412983698' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:03 smithi146 ceph-mon[103278]: pgmap v1804: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-mon[91136]: pgmap v1804: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-mon[99959]: pgmap v1804: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:03.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:03.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:03.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:05.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:05 smithi078 ceph-mon[91136]: pgmap v1805: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:05.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:05 smithi078 ceph-mon[99959]: pgmap v1805: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:05 smithi146 ceph-mon[103278]: pgmap v1805: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:07.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:07 smithi078 ceph-mon[91136]: pgmap v1806: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:07.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:07 smithi078 ceph-mon[99959]: pgmap v1806: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:07.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:07 smithi146 ceph-mon[103278]: pgmap v1806: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[91136]: pgmap v1807: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[99959]: pgmap v1807: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:09 smithi146 ceph-mon[103278]: pgmap v1807: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:09.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:09.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:10.339Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:10.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:10.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:20:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:20:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:11 smithi146 ceph-mon[103278]: pgmap v1808: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:11 smithi078 ceph-mon[91136]: pgmap v1808: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:11 smithi078 ceph-mon[99959]: pgmap v1808: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:13 smithi146 ceph-mon[103278]: pgmap v1809: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-mon[91136]: pgmap v1809: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-mon[99959]: pgmap v1809: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:13.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:13.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:13.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:15 smithi146 ceph-mon[103278]: pgmap v1810: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:15 smithi078 ceph-mon[91136]: pgmap v1810: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:15 smithi078 ceph-mon[99959]: pgmap v1810: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:17 smithi146 ceph-mon[103278]: pgmap v1811: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:17 smithi078 ceph-mon[91136]: pgmap v1811: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:17 smithi078 ceph-mon[99959]: pgmap v1811: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:19 smithi146 ceph-mon[103278]: pgmap v1812: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:19 smithi078 ceph-mon[91136]: pgmap v1812: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:19 smithi078 ceph-mon[99959]: pgmap v1812: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:20.340Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:20.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:20.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:20:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:20:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:21 smithi146 ceph-mon[103278]: pgmap v1813: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:21 smithi078 ceph-mon[91136]: pgmap v1813: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:21 smithi078 ceph-mon[99959]: pgmap v1813: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:23.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:23.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:23 smithi146 ceph-mon[103278]: pgmap v1814: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-mon[99959]: pgmap v1814: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-mon[91136]: pgmap v1814: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:23.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:23.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:23.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:25 smithi146 ceph-mon[103278]: pgmap v1815: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:25 smithi078 ceph-mon[91136]: pgmap v1815: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:25 smithi078 ceph-mon[99959]: pgmap v1815: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:27.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:27 smithi078 ceph-mon[91136]: pgmap v1816: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:27.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:27 smithi078 ceph-mon[99959]: pgmap v1816: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:27 smithi146 ceph-mon[103278]: pgmap v1816: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:29.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:29 smithi146 ceph-mon[103278]: pgmap v1817: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:29.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:29 smithi078 ceph-mon[99959]: pgmap v1817: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:29 smithi078 ceph-mon[91136]: pgmap v1817: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:30.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:30.341Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:30.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:30.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:30.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:30.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:30.643 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:20:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:20:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 71m - - 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 70m - - 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (65m) 2s ago 66m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (66m) 17s ago 66m 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (58m) 2s ago 64m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (57m) 17s ago 70m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (59m) 2s ago 71m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:20:31.171 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (57m) 2s ago 72m 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (56m) 17s ago 70m 103M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (56m) 2s ago 70m 108M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (65m) 2s ago 65m 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (65m) 17s ago 65m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (55m) 2s ago 69m 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (54m) 2s ago 69m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (53m) 2s ago 69m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (53m) 2s ago 68m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (52m) 17s ago 68m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (51m) 17s ago 68m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (50m) 17s ago 67m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:20:31.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (49m) 17s ago 67m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:20:31.173 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (58m) 17s ago 66m 69.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:20:31.173 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (49m) 2s ago 64m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:20:31.173 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (49m) 17s ago 64m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:20:31.173 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (49m) 2s ago 64m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:20:31.173 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (49m) 17s ago 64m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:20:31.479 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:20:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:20:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:20:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:20:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:20:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:20:31.480 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:20:31.481 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:31 smithi078 ceph-mon[99959]: pgmap v1818: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:31.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:31 smithi078 ceph-mon[91136]: pgmap v1818: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:31 smithi146 ceph-mon[103278]: pgmap v1818: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:20:31.752 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:20:31.753 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:20:31.753 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:20:31.753 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:20:31.753 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:20:31.753 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:20:31.753 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:20:32.062 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:20:32.062 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:32.062 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:20:32.063 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:32.063 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:20:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:32 smithi146 ceph-mon[103278]: from='client.68066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:32 smithi146 ceph-mon[103278]: from='client.68072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:32 smithi146 ceph-mon[103278]: from='client.68078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1298558517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2871625728' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[91136]: from='client.68066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[91136]: from='client.68072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[91136]: from='client.68078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1298558517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2871625728' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[99959]: from='client.68066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[99959]: from='client.68072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[99959]: from='client.68078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1298558517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:20:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2871625728' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:20:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:33 smithi146 ceph-mon[103278]: from='client.68090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:33 smithi146 ceph-mon[103278]: pgmap v1819: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-mon[99959]: from='client.68090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-mon[99959]: pgmap v1819: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-mon[91136]: from='client.68090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:20:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:33.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-mon[91136]: pgmap v1819: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:33.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:33.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:33.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:34 smithi146 ceph-mon[103278]: pgmap v1820: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[91136]: pgmap v1820: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:34 smithi078 ceph-mon[99959]: pgmap v1820: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:37 smithi146 ceph-mon[103278]: pgmap v1821: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:37 smithi078 ceph-mon[91136]: pgmap v1821: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:37 smithi078 ceph-mon[99959]: pgmap v1821: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:39 smithi146 ceph-mon[103278]: pgmap v1822: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:39 smithi078 ceph-mon[91136]: pgmap v1822: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:39 smithi078 ceph-mon[99959]: pgmap v1822: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:40.342Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:40.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:40.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:40.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:20:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:20:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:41 smithi078 ceph-mon[91136]: pgmap v1823: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:41 smithi078 ceph-mon[99959]: pgmap v1823: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:41 smithi146 ceph-mon[103278]: pgmap v1823: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:43 smithi146 ceph-mon[103278]: pgmap v1824: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-mon[99959]: pgmap v1824: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-mon[91136]: pgmap v1824: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:43.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:43.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:43.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:45 smithi146 ceph-mon[103278]: pgmap v1825: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:45 smithi078 ceph-mon[91136]: pgmap v1825: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:45 smithi078 ceph-mon[99959]: pgmap v1825: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:47.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:47 smithi078 ceph-mon[99959]: pgmap v1826: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:47.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:47 smithi078 ceph-mon[91136]: pgmap v1826: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:47 smithi146 ceph-mon[103278]: pgmap v1826: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:49 smithi146 ceph-mon[103278]: pgmap v1827: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:49.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:49 smithi078 ceph-mon[99959]: pgmap v1827: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:49.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:49 smithi078 ceph-mon[91136]: pgmap v1827: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:20:50.534 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.535 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.535 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:50.342Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:50.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:50.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:50.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:20:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:20:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:51 smithi146 ceph-mon[103278]: pgmap v1828: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:51 smithi078 ceph-mon[99959]: pgmap v1828: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:51 smithi078 ceph-mon[91136]: pgmap v1828: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-mon[91136]: pgmap v1829: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-mon[99959]: pgmap v1829: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:20:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:53 smithi146 ceph-mon[103278]: pgmap v1829: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:20:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:53.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:53.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:20:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:20:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:20:53.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:20:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[91136]: pgmap v1830: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T21:20:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:54 smithi078 ceph-mon[99959]: pgmap v1830: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T21:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:20:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:54 smithi146 ceph-mon[103278]: pgmap v1830: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T21:20:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:57 smithi146 ceph-mon[103278]: pgmap v1831: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T21:20:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:57 smithi078 ceph-mon[91136]: pgmap v1831: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T21:20:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:57 smithi078 ceph-mon[99959]: pgmap v1831: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-20T21:20:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:59 smithi146 ceph-mon[103278]: pgmap v1832: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:20:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:59 smithi078 ceph-mon[99959]: pgmap v1832: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:20:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:20:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:59 smithi078 ceph-mon[91136]: pgmap v1832: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:20:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:00.343Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:00.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:00.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:21:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:21:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:01 smithi146 ceph-mon[103278]: pgmap v1833: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:21:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:01 smithi078 ceph-mon[91136]: pgmap v1833: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:21:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:01 smithi078 ceph-mon[99959]: pgmap v1833: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:21:02.364 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:21:02.892 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 71m - - 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 71m - - 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (65m) 13s ago 66m 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (66m) 9s ago 66m 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (59m) 13s ago 64m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (57m) 9s ago 70m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (59m) 13s ago 72m 552M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (58m) 13s ago 72m 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (56m) 9s ago 71m 104M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (57m) 13s ago 71m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:21:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (66m) 13s ago 66m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (66m) 9s ago 66m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (55m) 13s ago 70m 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (54m) 13s ago 70m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (54m) 13s ago 69m 61.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (53m) 13s ago 69m 67.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (53m) 9s ago 68m 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (52m) 9s ago 68m 60.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (51m) 9s ago 68m 64.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (50m) 9s ago 67m 59.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (59m) 9s ago 67m 69.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (50m) 13s ago 64m 87.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (50m) 9s ago 65m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (50m) 13s ago 64m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:21:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (49m) 9s ago 64m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:21:03.211 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:21:03.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:21:03.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:21:03.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:03.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:21:03.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:21:03.212 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:21:03.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:03.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-mon[91136]: pgmap v1834: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:21:03.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-mon[91136]: from='client.48183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:03.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:03.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-mon[99959]: pgmap v1834: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:21:03.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-mon[99959]: from='client.48183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:03.491 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:21:03.491 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:21:03.491 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:21:03.491 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:21:03.492 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:21:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:03 smithi146 ceph-mon[103278]: pgmap v1834: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:21:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:03 smithi146 ceph-mon[103278]: from='client.48183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:03.832 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:21:03.832 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:21:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:21:03.833 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:21:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:21:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:03.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:03.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:03.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:04 smithi146 ceph-mon[103278]: from='client.68108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:04 smithi146 ceph-mon[103278]: from='client.48195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2550076438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3514910076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[91136]: from='client.68108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[91136]: from='client.48195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2550076438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3514910076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[99959]: from='client.68108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[99959]: from='client.48195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2550076438' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3514910076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:05 smithi146 ceph-mon[103278]: from='client.48207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:05 smithi146 ceph-mon[103278]: pgmap v1835: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:05 smithi078 ceph-mon[91136]: from='client.48207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:05 smithi078 ceph-mon[91136]: pgmap v1835: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:21:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:05 smithi078 ceph-mon[99959]: from='client.48207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:05 smithi078 ceph-mon[99959]: pgmap v1835: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:21:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:07 smithi078 ceph-mon[91136]: pgmap v1836: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-20T21:21:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:07 smithi078 ceph-mon[99959]: pgmap v1836: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-20T21:21:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:07 smithi146 ceph-mon[103278]: pgmap v1836: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-20T21:21:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:09 smithi078 ceph-mon[91136]: pgmap v1837: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T21:21:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:09 smithi078 ceph-mon[99959]: pgmap v1837: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T21:21:09.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:09 smithi146 ceph-mon[103278]: pgmap v1837: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T21:21:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[91136]: pgmap v1838: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-mon[99959]: pgmap v1838: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:21:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:10.343Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:10.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:10.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:10.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:10.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:10 smithi146 ceph-mon[103278]: pgmap v1838: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:21:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:21:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:21:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:13 smithi146 ceph-mon[103278]: pgmap v1839: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T21:21:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-mon[91136]: pgmap v1839: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T21:21:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-mon[99959]: pgmap v1839: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T21:21:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:13.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:13.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:13.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:14.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:15 smithi146 ceph-mon[103278]: pgmap v1840: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:21:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:15 smithi078 ceph-mon[91136]: pgmap v1840: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:21:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:15 smithi078 ceph-mon[99959]: pgmap v1840: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:21:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:17 smithi146 ceph-mon[103278]: pgmap v1841: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:21:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:17 smithi078 ceph-mon[91136]: pgmap v1841: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:21:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:17 smithi078 ceph-mon[99959]: pgmap v1841: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:21:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:19 smithi146 ceph-mon[103278]: pgmap v1842: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:21:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:19 smithi078 ceph-mon[91136]: pgmap v1842: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:21:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:19 smithi078 ceph-mon[99959]: pgmap v1842: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:21:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:20.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:20.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:20.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:21:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:21:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:21 smithi146 ceph-mon[103278]: pgmap v1843: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:21:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:21 smithi078 ceph-mon[99959]: pgmap v1843: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:21:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:21 smithi078 ceph-mon[91136]: pgmap v1843: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:21:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:23 smithi146 ceph-mon[103278]: pgmap v1844: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:21:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-mon[91136]: pgmap v1844: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:21:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-mon[99959]: pgmap v1844: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:21:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:23.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:23.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:23.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:25 smithi146 ceph-mon[103278]: pgmap v1845: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:21:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:25 smithi078 ceph-mon[91136]: pgmap v1845: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:21:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:25 smithi078 ceph-mon[99959]: pgmap v1845: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T21:21:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:27 smithi146 ceph-mon[103278]: pgmap v1846: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:27 smithi078 ceph-mon[91136]: pgmap v1846: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:27 smithi078 ceph-mon[99959]: pgmap v1846: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:29.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:29 smithi078 ceph-mon[99959]: pgmap v1847: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:29.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:29.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:29 smithi078 ceph-mon[91136]: pgmap v1847: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:29.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:29 smithi146 ceph-mon[103278]: pgmap v1847: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:30.511 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:30.345Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:30.511 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:30.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:30.511 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:30.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:30 smithi146 ceph-mon[103278]: pgmap v1848: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[91136]: pgmap v1848: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:21:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:21:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:30 smithi078 ceph-mon[99959]: pgmap v1848: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:33 smithi146 ceph-mon[103278]: pgmap v1849: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-mon[91136]: pgmap v1849: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-mon[99959]: pgmap v1849: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:33.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:33.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:33.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:34.130 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 72m - - 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 71m - - 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (66m) 5s ago 67m 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (67m) 0s ago 67m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (59m) 5s ago 65m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (58m) 0s ago 71m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (60m) 5s ago 72m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (58m) 5s ago 73m 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:21:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (57m) 0s ago 71m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (57m) 5s ago 71m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (66m) 5s ago 66m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (66m) 0s ago 66m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (56m) 5s ago 71m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (55m) 5s ago 70m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (54m) 5s ago 70m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (54m) 5s ago 69m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (53m) 0s ago 69m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (52m) 0s ago 69m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (51m) 0s ago 68m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (51m) 0s ago 68m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (59m) 0s ago 67m 69.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (50m) 5s ago 65m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:21:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (50m) 0s ago 65m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:21:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (50m) 5s ago 65m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:21:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (50m) 0s ago 65m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:21:34.963 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:21:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:21:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:21:34.965 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:21:35.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: pgmap v1850: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: from='client.68138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: from='client.68144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/105323536' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: pgmap v1850: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: from='client.68138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:35.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: from='client.68144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:35.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/105323536' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:21:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:21:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:21:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:21:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:21:35.239 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: pgmap v1850: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: from='client.68138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: from='client.68144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/105323536' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:21:35.546 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:21:35.546 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:21:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:21:35.546 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:21:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:21:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:36 smithi146 ceph-mon[103278]: from='client.68150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:36 smithi146 ceph-mon[103278]: from='client.68162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3925869183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:36 smithi078 ceph-mon[91136]: from='client.68150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:36 smithi078 ceph-mon[91136]: from='client.68162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3925869183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:36 smithi078 ceph-mon[99959]: from='client.68150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:36 smithi078 ceph-mon[99959]: from='client.68162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:21:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3925869183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:21:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:37 smithi146 ceph-mon[103278]: pgmap v1851: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:37 smithi078 ceph-mon[91136]: pgmap v1851: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:37 smithi078 ceph-mon[99959]: pgmap v1851: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:39 smithi146 ceph-mon[103278]: pgmap v1852: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:39 smithi078 ceph-mon[91136]: pgmap v1852: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:39 smithi078 ceph-mon[99959]: pgmap v1852: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:40.346Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:40.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:40.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:21:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:21:41.393 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:41 smithi078 ceph-mon[99959]: pgmap v1853: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:41.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:41 smithi078 ceph-mon[91136]: pgmap v1853: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:41 smithi146 ceph-mon[103278]: pgmap v1853: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:43 smithi146 ceph-mon[103278]: pgmap v1854: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-mon[99959]: pgmap v1854: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-mon[91136]: pgmap v1854: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:43.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:43.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:43.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:45 smithi146 ceph-mon[103278]: pgmap v1855: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:45 smithi078 ceph-mon[99959]: pgmap v1855: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:45 smithi078 ceph-mon[91136]: pgmap v1855: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:47 smithi146 ceph-mon[103278]: pgmap v1856: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:47 smithi078 ceph-mon[91136]: pgmap v1856: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:47 smithi078 ceph-mon[99959]: pgmap v1856: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:49.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:49 smithi078 ceph-mon[99959]: pgmap v1857: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:49.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:49 smithi078 ceph-mon[91136]: pgmap v1857: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:49 smithi146 ceph-mon[103278]: pgmap v1857: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:50.347Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:50.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:50.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:50.878 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:21:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:21:50.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:50.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:50.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:50.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[99959]: pgmap v1858: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:50.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:50.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:50.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:50.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:50 smithi078 ceph-mon[91136]: pgmap v1858: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:51.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:51.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:51.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:51.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:50 smithi146 ceph-mon[103278]: pgmap v1858: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:53 smithi146 ceph-mon[103278]: pgmap v1859: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-mon[91136]: pgmap v1859: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:21:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-mon[99959]: pgmap v1859: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:53.784Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:53.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:21:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:21:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:21:53.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:21:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:55 smithi146 ceph-mon[103278]: pgmap v1860: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[91136]: pgmap v1860: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[99959]: pgmap v1860: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:21:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:57 smithi146 ceph-mon[103278]: pgmap v1861: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:57 smithi078 ceph-mon[91136]: pgmap v1861: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:57 smithi078 ceph-mon[99959]: pgmap v1861: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:21:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:59 smithi146 ceph-mon[103278]: pgmap v1862: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:21:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:59 smithi078 ceph-mon[91136]: pgmap v1862: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:21:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:21:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:59 smithi078 ceph-mon[99959]: pgmap v1862: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:21:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:21:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:00.347Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:00.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:00.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:22:00] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T21:22:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:01 smithi146 ceph-mon[103278]: pgmap v1863: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:01 smithi078 ceph-mon[91136]: pgmap v1863: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:01 smithi078 ceph-mon[99959]: pgmap v1863: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:03 smithi146 ceph-mon[103278]: pgmap v1864: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-mon[91136]: pgmap v1864: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-mon[99959]: pgmap v1864: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:03.785Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:03.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:03.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:05 smithi146 ceph-mon[103278]: pgmap v1865: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:05 smithi078 ceph-mon[91136]: pgmap v1865: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:05 smithi078 ceph-mon[99959]: pgmap v1865: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:05.841 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 72m - - 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 72m - - 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (66m) 16s ago 67m 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (67m) 12s ago 67m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (60m) 16s ago 65m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (59m) 12s ago 71m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (60m) 16s ago 73m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (59m) 16s ago 73m 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (57m) 12s ago 72m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:22:06.373 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (58m) 16s ago 72m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (67m) 16s ago 67m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (67m) 12s ago 67m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (56m) 16s ago 71m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (55m) 16s ago 71m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (55m) 16s ago 70m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (54m) 16s ago 70m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (54m) 12s ago 70m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (53m) 12s ago 69m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (52m) 12s ago 69m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (51m) 12s ago 68m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (60m) 12s ago 68m 70.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (51m) 16s ago 66m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:22:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (51m) 12s ago 66m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:22:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (51m) 16s ago 65m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:22:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (50m) 12s ago 65m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:22:06.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:22:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:22:06.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:22:06.684 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:22:06.957 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:22:06.958 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:22:07.265 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:22:07.363 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:07.363 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:07.363 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:07.363 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:07.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[99959]: from='client.48255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[99959]: pgmap v1866: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:07.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[99959]: from='client.48261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.365 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[99959]: from='client.68186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.365 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2302591729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:07.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[91136]: from='client.48255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[91136]: pgmap v1866: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:07.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[91136]: from='client.48261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[91136]: from='client.68186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2302591729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:07 smithi146 ceph-mon[103278]: from='client.48255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:07 smithi146 ceph-mon[103278]: pgmap v1866: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:07 smithi146 ceph-mon[103278]: from='client.48261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:07 smithi146 ceph-mon[103278]: from='client.68186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2302591729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:08 smithi146 ceph-mon[103278]: from='client.48279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3709682019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:08 smithi078 ceph-mon[91136]: from='client.48279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3709682019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:08 smithi078 ceph-mon[99959]: from='client.48279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3709682019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:09 smithi078 ceph-mon[91136]: pgmap v1867: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:09 smithi078 ceph-mon[99959]: pgmap v1867: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:09 smithi146 ceph-mon[103278]: pgmap v1867: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:10.348Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:10.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:10.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:10.907 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:22:10] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-20T21:22:10.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:10.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:10.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:10.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[99959]: pgmap v1868: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:10.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:10.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:10.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:10.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:10 smithi078 ceph-mon[91136]: pgmap v1868: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:11.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:11.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:11.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:11.064 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:10 smithi146 ceph-mon[103278]: pgmap v1868: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:13 smithi146 ceph-mon[103278]: pgmap v1869: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-mon[91136]: pgmap v1869: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-mon[99959]: pgmap v1869: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:14.114 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:13.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:14.114 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:13.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:14.114 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:13.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[99959]: pgmap v1870: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[91136]: pgmap v1870: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:15 smithi146 ceph-mon[103278]: pgmap v1870: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:17 smithi078 ceph-mon[91136]: pgmap v1871: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:17 smithi078 ceph-mon[99959]: pgmap v1871: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:17 smithi146 ceph-mon[103278]: pgmap v1871: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:19 smithi078 ceph-mon[99959]: pgmap v1872: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:19 smithi078 ceph-mon[91136]: pgmap v1872: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:19 smithi146 ceph-mon[103278]: pgmap v1872: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:20.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:20.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:20.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:22:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:22:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:21 smithi146 ceph-mon[103278]: pgmap v1873: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:21 smithi078 ceph-mon[91136]: pgmap v1873: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:21 smithi078 ceph-mon[99959]: pgmap v1873: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-mon[91136]: pgmap v1874: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-mon[99959]: pgmap v1874: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:23 smithi146 ceph-mon[103278]: pgmap v1874: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:23.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:23.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:23.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:25 smithi146 ceph-mon[103278]: pgmap v1875: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:25 smithi078 ceph-mon[91136]: pgmap v1875: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:25 smithi078 ceph-mon[99959]: pgmap v1875: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:27 smithi146 ceph-mon[103278]: pgmap v1876: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:27 smithi078 ceph-mon[91136]: pgmap v1876: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:27 smithi078 ceph-mon[99959]: pgmap v1876: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:29.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:29 smithi078 ceph-mon[91136]: pgmap v1877: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:29.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:29 smithi078 ceph-mon[99959]: pgmap v1877: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:29 smithi146 ceph-mon[103278]: pgmap v1877: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:30.350Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:30.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:30.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:30.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:22:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:22:31.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[91136]: pgmap v1878: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:31.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:30 smithi078 ceph-mon[99959]: pgmap v1878: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:30 smithi146 ceph-mon[103278]: pgmap v1878: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:33 smithi146 ceph-mon[103278]: pgmap v1879: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-mon[91136]: pgmap v1879: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-mon[99959]: pgmap v1879: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:33.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:33.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:33.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:35 smithi146 ceph-mon[103278]: pgmap v1880: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[91136]: pgmap v1880: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[99959]: pgmap v1880: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:37 smithi146 ceph-mon[103278]: pgmap v1881: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:37 smithi078 ceph-mon[99959]: pgmap v1881: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:37 smithi078 ceph-mon[91136]: pgmap v1881: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:37.562 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:22:38.088 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:22:38.088 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 73m - - 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 73m - - 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (67m) 8s ago 68m 30.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (68m) 3s ago 68m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (60m) 8s ago 66m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (59m) 3s ago 72m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (61m) 8s ago 74m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (59m) 8s ago 74m 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (58m) 3s ago 72m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (58m) 8s ago 72m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (67m) 8s ago 67m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (67m) 3s ago 67m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (57m) 8s ago 72m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (56m) 8s ago 71m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:22:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (55m) 8s ago 71m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (55m) 8s ago 70m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (54m) 3s ago 70m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (53m) 3s ago 70m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (52m) 3s ago 69m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (52m) 3s ago 69m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (60m) 3s ago 68m 70.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (51m) 8s ago 66m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (51m) 3s ago 66m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (51m) 8s ago 66m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:22:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (51m) 3s ago 66m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:22:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:22:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:22:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:22:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:22:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:22:38.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:22:38.667 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:22:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:22:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:22:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:22:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:22:38.668 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:22:38.977 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:22:38.977 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:22:38.977 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:22:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[99959]: from='client.48291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[99959]: from='client.48297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[99959]: pgmap v1882: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[99959]: from='client.48303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/949985727' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/583184526' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[91136]: from='client.48291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[91136]: from='client.48297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[91136]: pgmap v1882: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[91136]: from='client.48303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/949985727' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/583184526' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:39 smithi146 ceph-mon[103278]: from='client.48291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:39 smithi146 ceph-mon[103278]: from='client.48297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:39 smithi146 ceph-mon[103278]: pgmap v1882: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:39 smithi146 ceph-mon[103278]: from='client.48303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/949985727' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:22:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/583184526' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:22:40.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:40 smithi078 ceph-mon[91136]: from='client.48315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:40.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:40 smithi078 ceph-mon[99959]: from='client.48315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:40.433 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:40.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:40.433 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:40.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:40.433 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:40.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:40 smithi146 ceph-mon[103278]: from='client.48315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:22:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:22:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:22:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:41 smithi146 ceph-mon[103278]: pgmap v1883: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:41 smithi078 ceph-mon[91136]: pgmap v1883: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:41 smithi078 ceph-mon[99959]: pgmap v1883: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:43 smithi146 ceph-mon[103278]: pgmap v1884: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-mon[91136]: pgmap v1884: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-mon[99959]: pgmap v1884: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:44.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:43.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:44.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:43.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:44.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:43.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:45 smithi146 ceph-mon[103278]: pgmap v1885: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:45 smithi078 ceph-mon[91136]: pgmap v1885: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:45 smithi078 ceph-mon[99959]: pgmap v1885: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:47 smithi146 ceph-mon[103278]: pgmap v1886: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:47 smithi078 ceph-mon[91136]: pgmap v1886: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:47 smithi078 ceph-mon[99959]: pgmap v1886: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:49 smithi146 ceph-mon[103278]: pgmap v1887: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:49 smithi078 ceph-mon[91136]: pgmap v1887: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:49 smithi078 ceph-mon[99959]: pgmap v1887: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:50.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:50.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:50.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:50.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:22:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:22:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[91136]: pgmap v1888: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[99959]: pgmap v1888: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:51 smithi146 ceph-mon[103278]: pgmap v1888: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:53.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:53.227 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:53 smithi146 ceph-mon[103278]: pgmap v1889: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:53.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:53.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-mon[91136]: pgmap v1889: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:53.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:22:53.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-mon[99959]: pgmap v1889: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:53.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:53.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:22:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:22:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:22:53.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:22:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:55 smithi146 ceph-mon[103278]: pgmap v1890: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[91136]: pgmap v1890: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[99959]: pgmap v1890: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:22:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:57 smithi146 ceph-mon[103278]: pgmap v1891: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:57 smithi078 ceph-mon[91136]: pgmap v1891: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:57 smithi078 ceph-mon[99959]: pgmap v1891: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:22:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:59 smithi146 ceph-mon[103278]: pgmap v1892: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:22:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:59 smithi078 ceph-mon[99959]: pgmap v1892: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:22:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:22:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:59 smithi078 ceph-mon[91136]: pgmap v1892: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:22:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:22:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:00.352Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:00.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:00.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:23:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:23:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:01 smithi146 ceph-mon[103278]: pgmap v1893: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:01 smithi078 ceph-mon[91136]: pgmap v1893: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:01 smithi078 ceph-mon[99959]: pgmap v1893: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-mon[99959]: pgmap v1894: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-mon[91136]: pgmap v1894: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:03 smithi146 ceph-mon[103278]: pgmap v1894: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:03.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:03.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:03.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:05 smithi146 ceph-mon[103278]: pgmap v1895: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:05 smithi078 ceph-mon[99959]: pgmap v1895: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:05 smithi078 ceph-mon[91136]: pgmap v1895: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:07 smithi146 ceph-mon[103278]: pgmap v1896: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:07 smithi078 ceph-mon[91136]: pgmap v1896: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:07 smithi078 ceph-mon[99959]: pgmap v1896: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:09.277 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:23:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:09 smithi146 ceph-mon[103278]: pgmap v1897: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:09 smithi078 ceph-mon[91136]: pgmap v1897: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:09 smithi078 ceph-mon[99959]: pgmap v1897: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 73m - - 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 73m - - 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (67m) 19s ago 68m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (68m) 15s ago 68m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (61m) 19s ago 66m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (60m) 15s ago 72m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (62m) 19s ago 74m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (60m) 19s ago 74m 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (58m) 15s ago 73m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:23:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (59m) 19s ago 73m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (68m) 19s ago 68m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (68m) 15s ago 68m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (58m) 19s ago 72m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (57m) 19s ago 72m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (56m) 19s ago 71m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (55m) 19s ago 71m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (55m) 15s ago 71m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (54m) 15s ago 70m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (53m) 15s ago 70m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (52m) 15s ago 69m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (61m) 15s ago 69m 70.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (52m) 19s ago 67m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (52m) 15s ago 67m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (52m) 19s ago 67m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:23:09.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (52m) 15s ago 66m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:23:10.117 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:23:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:23:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:23:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:23:10.118 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:23:10.355 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:10 smithi078 ceph-mon[99959]: from='client.48327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:10.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:10 smithi078 ceph-mon[91136]: from='client.48327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:10.393 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:23:10.393 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:23:10.394 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:23:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:10 smithi146 ceph-mon[103278]: from='client.48327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:10.353Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:10.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:10.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:10.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:10.702 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:23:10.702 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:10.703 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:10.703 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:10.933 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:23:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:23:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='client.68252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='client.68258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: pgmap v1898: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3130488390' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='client.68270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2265050390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='client.68252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='client.68258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: pgmap v1898: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3130488390' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='client.68270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2265050390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='client.68252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='client.68258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: pgmap v1898: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3130488390' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='client.68270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2265050390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:13.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:13.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:13 smithi146 ceph-mon[103278]: pgmap v1899: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:13.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:13.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-mon[91136]: pgmap v1899: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:13.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:13.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-mon[99959]: pgmap v1899: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:14.113 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:13.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:14.113 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:13.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:14.113 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:13.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:14.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:15 smithi146 ceph-mon[103278]: pgmap v1900: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[91136]: pgmap v1900: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[99959]: pgmap v1900: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:17 smithi146 ceph-mon[103278]: pgmap v1901: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:17 smithi078 ceph-mon[91136]: pgmap v1901: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:17 smithi078 ceph-mon[99959]: pgmap v1901: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:19 smithi146 ceph-mon[103278]: pgmap v1902: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:19 smithi078 ceph-mon[91136]: pgmap v1902: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:19 smithi078 ceph-mon[99959]: pgmap v1902: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:20.354Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:20.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:20.612 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:20.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:23:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:23:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:21 smithi146 ceph-mon[103278]: pgmap v1903: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:21 smithi078 ceph-mon[91136]: pgmap v1903: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:21 smithi078 ceph-mon[99959]: pgmap v1903: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:23 smithi146 ceph-mon[103278]: pgmap v1904: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-mon[91136]: pgmap v1904: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-mon[99959]: pgmap v1904: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:23.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:23.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:23.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:25 smithi146 ceph-mon[103278]: pgmap v1905: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:25 smithi078 ceph-mon[91136]: pgmap v1905: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:25 smithi078 ceph-mon[99959]: pgmap v1905: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:27 smithi146 ceph-mon[103278]: pgmap v1906: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:27 smithi078 ceph-mon[91136]: pgmap v1906: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:27 smithi078 ceph-mon[99959]: pgmap v1906: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:29 smithi146 ceph-mon[103278]: pgmap v1907: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:29 smithi078 ceph-mon[99959]: pgmap v1907: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:29 smithi078 ceph-mon[91136]: pgmap v1907: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:30.354Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:30.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:30.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:23:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[99959]: pgmap v1908: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[91136]: pgmap v1908: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:31 smithi146 ceph-mon[103278]: pgmap v1908: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:33.339 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:33.340 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:33 smithi146 ceph-mon[103278]: pgmap v1909: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-mon[91136]: pgmap v1909: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-mon[99959]: pgmap v1909: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:33.886 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:33.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:33.886 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:33.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:33.887 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:33.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:35.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:35 smithi146 ceph-mon[103278]: pgmap v1910: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:35.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[91136]: pgmap v1910: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[99959]: pgmap v1910: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:37 smithi146 ceph-mon[103278]: pgmap v1911: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:37 smithi078 ceph-mon[91136]: pgmap v1911: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:37 smithi078 ceph-mon[99959]: pgmap v1911: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:39 smithi146 ceph-mon[103278]: pgmap v1912: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:39 smithi078 ceph-mon[91136]: pgmap v1912: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:39 smithi078 ceph-mon[99959]: pgmap v1912: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:40.355Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:40.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:40.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:41.000 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:23:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:23:41.001 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 74m - - 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 74m - - 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (68m) 11s ago 69m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (69m) 6s ago 69m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (61m) 11s ago 67m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (60m) 6s ago 73m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (62m) 11s ago 75m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (60m) 11s ago 75m 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:23:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (59m) 6s ago 73m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (59m) 11s ago 73m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (69m) 11s ago 69m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (68m) 6s ago 68m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (58m) 11s ago 73m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (57m) 11s ago 72m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (56m) 11s ago 72m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (56m) 11s ago 72m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (55m) 6s ago 71m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (54m) 6s ago 71m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (54m) 6s ago 70m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (53m) 6s ago 70m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (62m) 6s ago 69m 70.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (52m) 11s ago 67m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:23:41.529 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (52m) 6s ago 67m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:23:41.530 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (52m) 11s ago 67m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:23:41.530 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (52m) 6s ago 67m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:23:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:41 smithi146 ceph-mon[103278]: pgmap v1913: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:41 smithi078 ceph-mon[91136]: pgmap v1913: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:41 smithi078 ceph-mon[99959]: pgmap v1913: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:41.839 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:23:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:23:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:23:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:23:41.840 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:23:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:23:42.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:23:42.109 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:23:42.109 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:23:42.109 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:23:42.109 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:23:42.109 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:23:42.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:42 smithi078 ceph-mon[99959]: from='client.48363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:42.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:42 smithi078 ceph-mon[99959]: from='client.68288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:42.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3597049660' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:42.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:42 smithi078 ceph-mon[91136]: from='client.48363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:42.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:42 smithi078 ceph-mon[91136]: from='client.68288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:42.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3597049660' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:42 smithi146 ceph-mon[103278]: from='client.48363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:42 smithi146 ceph-mon[103278]: from='client.68288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3597049660' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:23:43.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:43 smithi146 ceph-mon[103278]: from='client.48375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:43 smithi146 ceph-mon[103278]: pgmap v1914: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:43 smithi146 ceph-mon[103278]: from='client.48387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/627450070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[91136]: from='client.48375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[91136]: pgmap v1914: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[91136]: from='client.48387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/627450070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[99959]: from='client.48375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[99959]: pgmap v1914: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[99959]: from='client.48387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:23:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/627450070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:23:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:43.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:43.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:43.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:45 smithi146 ceph-mon[103278]: pgmap v1915: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:45 smithi078 ceph-mon[91136]: pgmap v1915: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:45 smithi078 ceph-mon[99959]: pgmap v1915: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:47 smithi146 ceph-mon[103278]: pgmap v1916: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:47 smithi078 ceph-mon[91136]: pgmap v1916: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:47 smithi078 ceph-mon[99959]: pgmap v1916: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:49 smithi146 ceph-mon[103278]: pgmap v1917: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:49 smithi078 ceph-mon[99959]: pgmap v1917: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:49 smithi078 ceph-mon[91136]: pgmap v1917: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:50.356Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:50.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:50.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:50.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:23:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:23:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:51 smithi146 ceph-mon[103278]: pgmap v1918: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[91136]: pgmap v1918: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[99959]: pgmap v1918: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:53.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:53.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-mon[91136]: pgmap v1919: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:53.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:53.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-mon[99959]: pgmap v1919: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:53.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:23:53.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:53 smithi146 ceph-mon[103278]: pgmap v1919: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:53.966 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:53.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:53.966 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:53.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:23:53.966 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:23:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:23:53.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:23:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:55 smithi146 ceph-mon[103278]: pgmap v1920: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[91136]: pgmap v1920: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[99959]: pgmap v1920: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:55.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:23:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:57 smithi146 ceph-mon[103278]: pgmap v1921: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:57 smithi078 ceph-mon[91136]: pgmap v1921: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:57 smithi078 ceph-mon[99959]: pgmap v1921: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:23:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:23:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:23:59 smithi146 ceph-mon[103278]: pgmap v1922: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:23:59 smithi078 ceph-mon[91136]: pgmap v1922: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:23:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:23:59 smithi078 ceph-mon[99959]: pgmap v1922: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:00.357Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:00.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:00.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:24:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:24:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:01 smithi146 ceph-mon[103278]: pgmap v1923: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:01 smithi078 ceph-mon[91136]: pgmap v1923: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:01 smithi078 ceph-mon[99959]: pgmap v1923: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:03 smithi146 ceph-mon[103278]: pgmap v1924: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-mon[91136]: pgmap v1924: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-mon[99959]: pgmap v1924: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:03.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:03.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:03.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:05 smithi146 ceph-mon[103278]: pgmap v1925: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:05 smithi078 ceph-mon[91136]: pgmap v1925: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:05 smithi078 ceph-mon[99959]: pgmap v1925: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:07 smithi146 ceph-mon[103278]: pgmap v1926: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:07 smithi078 ceph-mon[91136]: pgmap v1926: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:07 smithi078 ceph-mon[99959]: pgmap v1926: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:09 smithi146 ceph-mon[103278]: pgmap v1927: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:09 smithi078 ceph-mon[91136]: pgmap v1927: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:09 smithi078 ceph-mon[99959]: pgmap v1927: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:10.358Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:10.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:10.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:24:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:11 smithi146 ceph-mon[103278]: pgmap v1928: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[91136]: pgmap v1928: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[99959]: pgmap v1928: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:11.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:12.703 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 74m - - 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 74m - - 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (68m) 2s ago 70m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (69m) 18s ago 69m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (62m) 2s ago 67m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (61m) 18s ago 74m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (63m) 2s ago 75m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (61m) 2s ago 75m 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:24:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (60m) 18s ago 74m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (60m) 2s ago 74m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (69m) 2s ago 69m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (69m) 18s ago 69m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (59m) 2s ago 73m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (58m) 2s ago 73m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (57m) 2s ago 72m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (56m) 2s ago 72m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (56m) 18s ago 72m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (55m) 18s ago 71m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (54m) 18s ago 71m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (53m) 18s ago 70m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (62m) 18s ago 70m 70.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (53m) 2s ago 68m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (53m) 18s ago 68m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:24:13.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (53m) 2s ago 68m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:24:13.258 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (53m) 18s ago 68m 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:24:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-mon[91136]: pgmap v1929: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-mon[99959]: pgmap v1929: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:13.573 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:24:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:24:13.573 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:24:13.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:24:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:24:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:13 smithi146 ceph-mon[103278]: pgmap v1929: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:24:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:24:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:24:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:24:13.848 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:24:13.848 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:13.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:13.848 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:13.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:13.848 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:13.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:14.189 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:24:14.189 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:14.189 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:14 smithi146 ceph-mon[103278]: from='client.68318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:14 smithi146 ceph-mon[103278]: from='client.68324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:14 smithi146 ceph-mon[103278]: from='client.48411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/848092628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1138300645' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[99959]: from='client.68318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[99959]: from='client.68324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[99959]: from='client.48411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/848092628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1138300645' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[91136]: from='client.68318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[91136]: from='client.68324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[91136]: from='client.48411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/848092628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:14.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1138300645' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:15.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[91136]: from='client.48423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:15.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[91136]: pgmap v1930: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:15.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[99959]: from='client.48423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:15.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[99959]: pgmap v1930: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:15.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:15 smithi146 ceph-mon[103278]: from='client.48423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:15.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:15 smithi146 ceph-mon[103278]: pgmap v1930: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:15.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:15.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:17 smithi146 ceph-mon[103278]: pgmap v1931: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:17 smithi078 ceph-mon[91136]: pgmap v1931: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:17 smithi078 ceph-mon[99959]: pgmap v1931: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:19 smithi146 ceph-mon[103278]: pgmap v1932: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:19 smithi078 ceph-mon[91136]: pgmap v1932: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:19 smithi078 ceph-mon[99959]: pgmap v1932: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:20.358Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:20.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:20.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:24:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:24:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:21 smithi146 ceph-mon[103278]: pgmap v1933: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:21 smithi078 ceph-mon[91136]: pgmap v1933: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:21 smithi078 ceph-mon[99959]: pgmap v1933: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:22 smithi146 ceph-mon[103278]: pgmap v1934: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:22 smithi078 ceph-mon[91136]: pgmap v1934: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:22 smithi078 ceph-mon[99959]: pgmap v1934: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:23.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:23.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:23.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:25 smithi146 ceph-mon[103278]: pgmap v1935: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:25 smithi078 ceph-mon[91136]: pgmap v1935: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:25 smithi078 ceph-mon[99959]: pgmap v1935: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 21:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T21:24:25.847639299Z level=info msg="Completed cleanup jobs" duration=11.924552ms 2024-09-20T21:24:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:27 smithi146 ceph-mon[103278]: pgmap v1936: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:27 smithi078 ceph-mon[91136]: pgmap v1936: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:27 smithi078 ceph-mon[99959]: pgmap v1936: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:29 smithi146 ceph-mon[103278]: pgmap v1937: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:29 smithi078 ceph-mon[91136]: pgmap v1937: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:29 smithi078 ceph-mon[99959]: pgmap v1937: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:30.720 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:24:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:24:30.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:30.359Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:30.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:30.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:30.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:30.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:31 smithi146 ceph-mon[103278]: pgmap v1938: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[91136]: pgmap v1938: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[99959]: pgmap v1938: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:33 smithi146 ceph-mon[103278]: pgmap v1939: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-mon[99959]: pgmap v1939: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-mon[91136]: pgmap v1939: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:33.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:33.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:33.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:33.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:33.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:33.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:35.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:35 smithi078 ceph-mon[91136]: pgmap v1940: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:35.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:35 smithi078 ceph-mon[99959]: pgmap v1940: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:35.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:35 smithi146 ceph-mon[103278]: pgmap v1940: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:37 smithi146 ceph-mon[103278]: pgmap v1941: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:37 smithi078 ceph-mon[91136]: pgmap v1941: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:37 smithi078 ceph-mon[99959]: pgmap v1941: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:39 smithi146 ceph-mon[103278]: pgmap v1942: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:39 smithi078 ceph-mon[91136]: pgmap v1942: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:39 smithi078 ceph-mon[99959]: pgmap v1942: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:40.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:24:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:24:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:40.360Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:40.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:40.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:41 smithi146 ceph-mon[103278]: pgmap v1943: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:41 smithi078 ceph-mon[99959]: pgmap v1943: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:41 smithi078 ceph-mon[91136]: pgmap v1943: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:43 smithi146 ceph-mon[103278]: pgmap v1944: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-mon[91136]: pgmap v1944: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-mon[99959]: pgmap v1944: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:43.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:43.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:43.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:44.480 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 75m - - 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 75m - - 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (69m) 14s ago 70m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (70m) 9s ago 70m 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (62m) 14s ago 68m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (61m) 9s ago 74m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (63m) 14s ago 76m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (61m) 14s ago 76m 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (60m) 9s ago 74m 106M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (60m) 14s ago 74m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:24:45.008 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (70m) 14s ago 70m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (70m) 9s ago 70m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (59m) 14s ago 74m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (58m) 14s ago 73m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (57m) 14s ago 73m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (57m) 14s ago 73m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (56m) 9s ago 72m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (55m) 9s ago 72m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (55m) 9s ago 71m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (54m) 9s ago 71m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (63m) 9s ago 70m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (53m) 14s ago 68m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (53m) 9s ago 68m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:24:45.009 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (53m) 14s ago 68m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:24:45.010 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (53m) 9s ago 68m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:24:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:45 smithi078 ceph-mon[99959]: pgmap v1945: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:45 smithi078 ceph-mon[91136]: pgmap v1945: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:45.322 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:24:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:24:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:24:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:24:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:24:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:24:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:24:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:24:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:24:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:24:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:24:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:24:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:24:45.324 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:24:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:45 smithi146 ceph-mon[103278]: pgmap v1945: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:45.589 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:24:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:24:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:24:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:24:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:24:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:24:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:24:45.898 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:24:45.898 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:24:45.898 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:24:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:46 smithi146 ceph-mon[103278]: from='client.48435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:46 smithi146 ceph-mon[103278]: from='client.48441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:46 smithi146 ceph-mon[103278]: from='client.68366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/188626561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4275702435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[91136]: from='client.48435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[91136]: from='client.48441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[91136]: from='client.68366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/188626561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4275702435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[99959]: from='client.48435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[99959]: from='client.48441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[99959]: from='client.68366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/188626561' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:24:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4275702435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:24:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:47 smithi146 ceph-mon[103278]: from='client.68378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:47 smithi146 ceph-mon[103278]: pgmap v1946: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:47 smithi078 ceph-mon[99959]: from='client.68378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:47 smithi078 ceph-mon[99959]: pgmap v1946: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:47 smithi078 ceph-mon[91136]: from='client.68378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:24:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:47 smithi078 ceph-mon[91136]: pgmap v1946: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:49 smithi146 ceph-mon[103278]: pgmap v1947: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:49 smithi078 ceph-mon[91136]: pgmap v1947: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:49 smithi078 ceph-mon[99959]: pgmap v1947: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:24:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:24:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:50.360Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:50.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:50.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:51 smithi146 ceph-mon[103278]: pgmap v1948: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[91136]: pgmap v1948: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[99959]: pgmap v1948: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:53 smithi146 ceph-mon[103278]: pgmap v1949: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-mon[91136]: pgmap v1949: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:24:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-mon[99959]: pgmap v1949: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:54.134 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:53.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:54.135 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:53.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:24:54.135 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:24:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:24:53.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:24:55.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:55 smithi146 ceph-mon[103278]: pgmap v1950: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:55.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:55 smithi078 ceph-mon[99959]: pgmap v1950: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:55.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:55 smithi078 ceph-mon[91136]: pgmap v1950: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:56.535 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.535 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.535 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:24:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:57 smithi078 ceph-mon[91136]: pgmap v1951: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:57 smithi078 ceph-mon[99959]: pgmap v1951: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:57.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:57 smithi146 ceph-mon[103278]: pgmap v1951: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:24:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:24:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:24:59 smithi146 ceph-mon[103278]: pgmap v1952: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:24:59 smithi078 ceph-mon[91136]: pgmap v1952: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:24:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:24:59 smithi078 ceph-mon[99959]: pgmap v1952: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:25:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:25:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:00.361Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:00.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:00.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:01 smithi146 ceph-mon[103278]: pgmap v1953: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:01 smithi078 ceph-mon[91136]: pgmap v1953: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:01 smithi078 ceph-mon[99959]: pgmap v1953: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:03 smithi146 ceph-mon[103278]: pgmap v1954: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-mon[91136]: pgmap v1954: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:03.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-mon[99959]: pgmap v1954: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:03.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:03.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:03.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:05 smithi146 ceph-mon[103278]: pgmap v1955: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:05 smithi078 ceph-mon[91136]: pgmap v1955: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:05 smithi078 ceph-mon[99959]: pgmap v1955: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:07 smithi078 ceph-mon[91136]: pgmap v1956: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:07 smithi078 ceph-mon[99959]: pgmap v1956: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:07.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:07 smithi146 ceph-mon[103278]: pgmap v1956: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:09 smithi146 ceph-mon[103278]: pgmap v1957: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:09 smithi078 ceph-mon[91136]: pgmap v1957: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:09 smithi078 ceph-mon[99959]: pgmap v1957: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:10.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:25:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:25:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:10.362Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:10.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:10.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:11 smithi146 ceph-mon[103278]: pgmap v1958: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[99959]: pgmap v1958: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[91136]: pgmap v1958: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:13 smithi146 ceph-mon[103278]: pgmap v1959: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-mon[99959]: pgmap v1959: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-mon[91136]: pgmap v1959: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:13.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:13.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:13.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:15 smithi146 ceph-mon[103278]: pgmap v1960: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:15 smithi078 ceph-mon[99959]: pgmap v1960: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:15 smithi078 ceph-mon[91136]: pgmap v1960: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:16.194 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:25:16.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[99959]: pgmap v1961: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:16.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[99959]: from='client.48471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:16.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[91136]: pgmap v1961: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:16.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:16 smithi078 ceph-mon[91136]: from='client.48471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:16.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:16.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:16 smithi146 ceph-mon[103278]: pgmap v1961: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:16.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:16 smithi146 ceph-mon[103278]: from='client.48471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:16.757 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:25:16.757 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 75m - - 2024-09-20T21:25:16.757 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 75m - - 2024-09-20T21:25:16.757 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (70m) 5s ago 71m 31.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (70m) 1s ago 70m 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (63m) 5s ago 68m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (62m) 1s ago 75m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (64m) 5s ago 76m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (62m) 5s ago 76m 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (61m) 1s ago 75m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (61m) 5s ago 75m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (70m) 5s ago 70m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (70m) 1s ago 70m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (60m) 5s ago 74m 67.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:25:16.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (59m) 5s ago 74m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (58m) 5s ago 73m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (58m) 5s ago 73m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (57m) 1s ago 73m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (56m) 1s ago 72m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (55m) 1s ago 72m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (54m) 1s ago 72m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (63m) 1s ago 71m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (54m) 5s ago 69m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (54m) 1s ago 69m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (54m) 5s ago 69m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:25:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (54m) 1s ago 69m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:25:17.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:25:17.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:25:17.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:17.079 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:25:17.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:25:17.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:25:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:25:17.352 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:25:17.353 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:25:17.666 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:25:17.666 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:17.666 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:17.666 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:17.666 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:17.666 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[99959]: from='client.68396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.666 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[99959]: from='client.48483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.666 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1639396278' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:17.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[99959]: from='client.48495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[91136]: from='client.68396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[91136]: from='client.48483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1639396278' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:17.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:17 smithi078 ceph-mon[91136]: from='client.48495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:17 smithi146 ceph-mon[103278]: from='client.68396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:17 smithi146 ceph-mon[103278]: from='client.48483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1639396278' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:17 smithi146 ceph-mon[103278]: from='client.48495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3583459187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:18 smithi146 ceph-mon[103278]: pgmap v1962: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3583459187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:18 smithi078 ceph-mon[91136]: pgmap v1962: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3583459187' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:18 smithi078 ceph-mon[99959]: pgmap v1962: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:25:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:25:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:20.362Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:20.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:20.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:21 smithi146 ceph-mon[103278]: pgmap v1963: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:21 smithi078 ceph-mon[91136]: pgmap v1963: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:21 smithi078 ceph-mon[99959]: pgmap v1963: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:23 smithi146 ceph-mon[103278]: pgmap v1964: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-mon[91136]: pgmap v1964: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-mon[99959]: pgmap v1964: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:23.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:23.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:23.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:25 smithi146 ceph-mon[103278]: pgmap v1965: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:25 smithi078 ceph-mon[91136]: pgmap v1965: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:25 smithi078 ceph-mon[99959]: pgmap v1965: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:27 smithi146 ceph-mon[103278]: pgmap v1966: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:27 smithi078 ceph-mon[91136]: pgmap v1966: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:27 smithi078 ceph-mon[99959]: pgmap v1966: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:29 smithi146 ceph-mon[103278]: pgmap v1967: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:29 smithi078 ceph-mon[91136]: pgmap v1967: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:29 smithi078 ceph-mon[99959]: pgmap v1967: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:25:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:25:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:30.364Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:30.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:30.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:31 smithi146 ceph-mon[103278]: pgmap v1968: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[91136]: pgmap v1968: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[99959]: pgmap v1968: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:31.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:33 smithi146 ceph-mon[103278]: pgmap v1969: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-mon[91136]: pgmap v1969: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-mon[99959]: pgmap v1969: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:33.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:33.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:33.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:35.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:35 smithi078 ceph-mon[91136]: pgmap v1970: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:35.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:35 smithi078 ceph-mon[99959]: pgmap v1970: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:35 smithi146 ceph-mon[103278]: pgmap v1970: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:36.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[99959]: pgmap v1971: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:36.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:36 smithi078 ceph-mon[91136]: pgmap v1971: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:36.851 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.851 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:36.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:36 smithi146 ceph-mon[103278]: pgmap v1971: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:39 smithi146 ceph-mon[103278]: pgmap v1972: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:39 smithi078 ceph-mon[91136]: pgmap v1972: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:39 smithi078 ceph-mon[99959]: pgmap v1972: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:25:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:25:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:40.365Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:40.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:40.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:41 smithi146 ceph-mon[103278]: pgmap v1973: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:41 smithi078 ceph-mon[91136]: pgmap v1973: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:41 smithi078 ceph-mon[99959]: pgmap v1973: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:43 smithi146 ceph-mon[103278]: pgmap v1974: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-mon[91136]: pgmap v1974: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-mon[99959]: pgmap v1974: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:43.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:43.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:43.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:45 smithi146 ceph-mon[103278]: pgmap v1975: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:45 smithi078 ceph-mon[91136]: pgmap v1975: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:45 smithi078 ceph-mon[99959]: pgmap v1975: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:47 smithi146 ceph-mon[103278]: pgmap v1976: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:47 smithi078 ceph-mon[91136]: pgmap v1976: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:47 smithi078 ceph-mon[99959]: pgmap v1976: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:47.964 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 76m - - 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 76m - - 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (70m) 17s ago 71m 31.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (71m) 12s ago 71m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (63m) 17s ago 69m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (62m) 12s ago 75m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (64m) 17s ago 77m 553M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (62m) 17s ago 77m 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (61m) 12s ago 76m 107M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:25:48.495 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (62m) 17s ago 75m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (71m) 17s ago 71m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (71m) 12s ago 71m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (60m) 17s ago 75m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (59m) 17s ago 74m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (59m) 17s ago 74m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (58m) 17s ago 74m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (58m) 12s ago 73m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (56m) 12s ago 73m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (56m) 12s ago 72m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (55m) 12s ago 72m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (64m) 12s ago 71m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:25:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (54m) 17s ago 69m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:25:48.497 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (54m) 12s ago 69m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:25:48.497 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (54m) 17s ago 69m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:25:48.497 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (54m) 12s ago 69m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:48.806 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:25:48.807 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:25:49.077 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:25:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:25:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:25:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:25:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:25:49.078 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:25:49.078 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:25:49.388 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:25:49.388 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:49.388 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:25:49.389 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:49.389 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:25:49.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[99959]: from='client.48507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:49.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[99959]: pgmap v1977: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:49.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[99959]: from='client.68432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:49.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3139983730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:49.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[91136]: from='client.48507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:49.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[91136]: pgmap v1977: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:49.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[91136]: from='client.68432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:49.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3139983730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:49 smithi146 ceph-mon[103278]: from='client.48507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:49 smithi146 ceph-mon[103278]: pgmap v1977: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:49 smithi146 ceph-mon[103278]: from='client.68432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3139983730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:25:50.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-mon[99959]: from='client.68438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:50.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-mon[99959]: from='client.68450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:50.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3966576279' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:50.497 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:50.366Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:50.497 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:50.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:50.497 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:50.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:50.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-mon[91136]: from='client.68438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:50.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-mon[91136]: from='client.68450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:50.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3966576279' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:50 smithi146 ceph-mon[103278]: from='client.68438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:50 smithi146 ceph-mon[103278]: from='client.68450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:25:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3966576279' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:25:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:25:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T21:25:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:51 smithi146 ceph-mon[103278]: pgmap v1978: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:51 smithi078 ceph-mon[91136]: pgmap v1978: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:51 smithi078 ceph-mon[99959]: pgmap v1978: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:53.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:53.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:53 smithi146 ceph-mon[103278]: pgmap v1979: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-mon[91136]: pgmap v1979: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:25:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-mon[99959]: pgmap v1979: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:25:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:53.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:53.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:25:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:25:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:25:53.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:25:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:55 smithi146 ceph-mon[103278]: pgmap v1980: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:55 smithi078 ceph-mon[91136]: pgmap v1980: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:55 smithi078 ceph-mon[99959]: pgmap v1980: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:25:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:56 smithi146 ceph-mon[103278]: pgmap v1981: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:25:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[91136]: pgmap v1981: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:25:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:56 smithi078 ceph-mon[99959]: pgmap v1981: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:25:58.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:58.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:25:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:25:59 smithi146 ceph-mon[103278]: pgmap v1982: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:25:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:25:59 smithi078 ceph-mon[91136]: pgmap v1982: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:25:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:25:59 smithi078 ceph-mon[99959]: pgmap v1982: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:26:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:26:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:26:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:00.367Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:00.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:00.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:01.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:01 smithi146 ceph-mon[103278]: pgmap v1983: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:01 smithi078 ceph-mon[91136]: pgmap v1983: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:01 smithi078 ceph-mon[99959]: pgmap v1983: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:03 smithi146 ceph-mon[103278]: pgmap v1984: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-mon[91136]: pgmap v1984: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-mon[99959]: pgmap v1984: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:03.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:03.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:03.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:05 smithi146 ceph-mon[103278]: pgmap v1985: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:05 smithi078 ceph-mon[91136]: pgmap v1985: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:05 smithi078 ceph-mon[99959]: pgmap v1985: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:07 smithi146 ceph-mon[103278]: pgmap v1986: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T21:26:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:07 smithi078 ceph-mon[91136]: pgmap v1986: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T21:26:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:07 smithi078 ceph-mon[99959]: pgmap v1986: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T21:26:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:09 smithi146 ceph-mon[103278]: pgmap v1987: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:09 smithi078 ceph-mon[91136]: pgmap v1987: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:09 smithi078 ceph-mon[99959]: pgmap v1987: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:26:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:26:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:10.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:10.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:10.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:11 smithi146 ceph-mon[103278]: pgmap v1988: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:11 smithi078 ceph-mon[99959]: pgmap v1988: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:11 smithi078 ceph-mon[91136]: pgmap v1988: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:12.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:13.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:13.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:13 smithi146 ceph-mon[103278]: pgmap v1989: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T21:26:13.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-mon[91136]: pgmap v1989: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T21:26:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-mon[99959]: pgmap v1989: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T21:26:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:13.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:13.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:13.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:15 smithi146 ceph-mon[103278]: pgmap v1990: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:15 smithi078 ceph-mon[91136]: pgmap v1990: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:15 smithi078 ceph-mon[99959]: pgmap v1990: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T21:26:16.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:16.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:16.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:16.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[99959]: pgmap v1991: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:16.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:16.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:16.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:16.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:16 smithi078 ceph-mon[91136]: pgmap v1991: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:17.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:17.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:17.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:17.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:16 smithi146 ceph-mon[103278]: pgmap v1991: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T21:26:19.377 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:19 smithi146 ceph-mon[103278]: pgmap v1992: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:26:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:19 smithi078 ceph-mon[91136]: pgmap v1992: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:26:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:19 smithi078 ceph-mon[99959]: pgmap v1992: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T21:26:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:26:20.208 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:26:20.208 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 76m - - 2024-09-20T21:26:20.208 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 76m - - 2024-09-20T21:26:20.208 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (71m) 8s ago 72m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:26:20.208 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (71m) 4s ago 71m 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:26:20.208 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (64m) 8s ago 69m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (63m) 4s ago 76m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (65m) 8s ago 77m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (63m) 8s ago 77m 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (62m) 4s ago 76m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (62m) 8s ago 76m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (71m) 8s ago 71m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (71m) 4s ago 71m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (61m) 8s ago 75m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (60m) 8s ago 75m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (59m) 8s ago 75m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (59m) 8s ago 74m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (58m) 4s ago 74m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (57m) 4s ago 73m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (56m) 4s ago 73m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:26:20.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (55m) 4s ago 73m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:26:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (64m) 4s ago 72m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:26:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (55m) 8s ago 70m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:26:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (55m) 4s ago 70m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:26:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (55m) 8s ago 70m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:26:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (55m) 4s ago 70m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:26:20.516 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:26:20.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:26:20.518 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:26:20.518 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:20.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:20.518 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:20.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:20.519 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:20.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:20.784 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:26:20.785 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:26:20.786 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:26:20.786 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:26:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:26:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:26:21.097 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:26:21.097 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:21.097 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:21.097 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:21.097 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:21 smithi146 ceph-mon[103278]: from='client.68462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:21 smithi146 ceph-mon[103278]: from='client.68468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:21 smithi146 ceph-mon[103278]: pgmap v1993: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:21 smithi146 ceph-mon[103278]: from='client.48555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/128874350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1015238067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[91136]: from='client.68462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[91136]: from='client.68468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[91136]: pgmap v1993: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[91136]: from='client.48555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/128874350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1015238067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[99959]: from='client.68462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[99959]: from='client.68468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[99959]: pgmap v1993: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[99959]: from='client.48555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/128874350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1015238067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:22 smithi146 ceph-mon[103278]: from='client.68486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:22 smithi078 ceph-mon[91136]: from='client.68486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:22 smithi078 ceph-mon[99959]: from='client.68486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:23 smithi146 ceph-mon[103278]: pgmap v1994: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-mon[99959]: pgmap v1994: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-mon[91136]: pgmap v1994: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T21:26:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:23.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:23.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:23.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:25 smithi146 ceph-mon[103278]: pgmap v1995: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:25 smithi078 ceph-mon[91136]: pgmap v1995: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:25 smithi078 ceph-mon[99959]: pgmap v1995: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:27 smithi146 ceph-mon[103278]: pgmap v1996: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:27 smithi078 ceph-mon[91136]: pgmap v1996: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:27 smithi078 ceph-mon[99959]: pgmap v1996: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:29 smithi146 ceph-mon[103278]: pgmap v1997: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:29 smithi078 ceph-mon[91136]: pgmap v1997: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:29 smithi078 ceph-mon[99959]: pgmap v1997: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:26:30] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T21:26:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:30.369Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:30.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:30.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:31.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:31 smithi078 ceph-mon[91136]: pgmap v1998: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:31.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:31 smithi078 ceph-mon[99959]: pgmap v1998: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:31 smithi146 ceph-mon[103278]: pgmap v1998: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:32 smithi146 ceph-mon[103278]: pgmap v1999: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[91136]: pgmap v1999: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:32 smithi078 ceph-mon[99959]: pgmap v1999: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:33.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:33.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:33.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:35 smithi146 ceph-mon[103278]: pgmap v2000: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:35 smithi078 ceph-mon[91136]: pgmap v2000: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:35 smithi078 ceph-mon[99959]: pgmap v2000: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:37.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[91136]: pgmap v2001: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:36 smithi078 ceph-mon[99959]: pgmap v2001: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:36 smithi146 ceph-mon[103278]: pgmap v2001: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:39.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:39 smithi146 ceph-mon[103278]: pgmap v2002: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:39.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:39 smithi078 ceph-mon[91136]: pgmap v2002: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:39.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:39 smithi078 ceph-mon[99959]: pgmap v2002: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:26:40] "GET /metrics HTTP/1.1" 200 38985 "" "Prometheus/2.43.0" 2024-09-20T21:26:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:40.370Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:40.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:40.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:41.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:41 smithi078 ceph-mon[91136]: pgmap v2003: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:41.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:41 smithi078 ceph-mon[99959]: pgmap v2003: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:41 smithi146 ceph-mon[103278]: pgmap v2003: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:43.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:43.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:43 smithi146 ceph-mon[103278]: pgmap v2004: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:43.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-mon[99959]: pgmap v2004: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-mon[91136]: pgmap v2004: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:43.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:43.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:43.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:45.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:45 smithi146 ceph-mon[103278]: pgmap v2005: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:45 smithi078 ceph-mon[91136]: pgmap v2005: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:45 smithi078 ceph-mon[99959]: pgmap v2005: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:47 smithi146 ceph-mon[103278]: pgmap v2006: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:47 smithi078 ceph-mon[91136]: pgmap v2006: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:47 smithi078 ceph-mon[99959]: pgmap v2006: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:49 smithi146 ceph-mon[103278]: pgmap v2007: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:49 smithi078 ceph-mon[99959]: pgmap v2007: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:49 smithi078 ceph-mon[91136]: pgmap v2007: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:50.374 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:50.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:50.374 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:50.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:50.374 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:50.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:26:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:26:51.399 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:26:51.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:51 smithi146 ceph-mon[103278]: pgmap v2008: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:51 smithi078 ceph-mon[91136]: pgmap v2008: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:51 smithi078 ceph-mon[99959]: pgmap v2008: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:26:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 77m - - 2024-09-20T21:26:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 77m - - 2024-09-20T21:26:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (71m) 0s ago 72m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:26:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (72m) 15s ago 72m 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:26:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (64m) 0s ago 70m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (63m) 15s ago 76m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (65m) 0s ago 78m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (64m) 0s ago 78m 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (62m) 15s ago 77m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (63m) 0s ago 76m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (72m) 0s ago 72m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (72m) 15s ago 72m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (61m) 0s ago 76m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (60m) 0s ago 75m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (60m) 0s ago 75m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (59m) 0s ago 75m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (59m) 15s ago 74m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (58m) 15s ago 74m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:26:51.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (57m) 15s ago 74m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:26:51.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (56m) 15s ago 73m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:26:51.943 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (65m) 15s ago 73m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:26:51.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (56m) 0s ago 70m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:26:51.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (55m) 15s ago 70m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:26:51.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (55m) 0s ago 70m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:26:51.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (55m) 15s ago 70m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:26:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:26:52.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:26:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:52 smithi146 ceph-mon[103278]: from='client.48579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:26:52.529 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:26:52.530 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:26:52.530 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:26:52.530 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:26:52.530 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:26:52.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:26:52.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[99959]: from='client.48579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:52.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[91136]: from='client.48579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:52.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:52.841 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:26:52.841 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:26:52.841 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:53 smithi146 ceph-mon[103278]: from='client.48585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:53 smithi146 ceph-mon[103278]: from='client.68510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:53 smithi146 ceph-mon[103278]: pgmap v2009: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1278815196' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3422690447' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[91136]: from='client.48585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[91136]: from='client.68510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[91136]: pgmap v2009: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1278815196' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3422690447' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[99959]: from='client.48585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[99959]: from='client.68510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:26:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[99959]: pgmap v2009: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1278815196' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:26:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3422690447' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:26:54.230 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:53.806Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:54.230 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:53.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:26:54.230 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:26:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:26:53.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:26:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:54 smithi146 ceph-mon[103278]: from='client.48603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:54 smithi078 ceph-mon[91136]: from='client.48603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:54 smithi078 ceph-mon[99959]: from='client.48603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:26:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:55 smithi146 ceph-mon[103278]: pgmap v2010: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:55 smithi078 ceph-mon[91136]: pgmap v2010: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:55 smithi078 ceph-mon[99959]: pgmap v2010: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:57 smithi146 ceph-mon[103278]: pgmap v2011: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[91136]: pgmap v2011: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:26:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:57 smithi078 ceph-mon[99959]: pgmap v2011: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:26:58.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:26:59.328 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:26:59 smithi146 ceph-mon[103278]: pgmap v2012: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:59.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:26:59 smithi078 ceph-mon[91136]: pgmap v2012: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:26:59.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:26:59 smithi078 ceph-mon[99959]: pgmap v2012: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:00.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:00.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:00.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:01.010 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:27:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:27:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:01 smithi078 ceph-mon[91136]: pgmap v2013: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:01 smithi078 ceph-mon[99959]: pgmap v2013: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:01 smithi146 ceph-mon[103278]: pgmap v2013: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:03.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:03 smithi146 ceph-mon[103278]: pgmap v2014: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-mon[91136]: pgmap v2014: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-mon[99959]: pgmap v2014: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:03.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:03.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:03.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:05 smithi146 ceph-mon[103278]: pgmap v2015: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:05 smithi078 ceph-mon[91136]: pgmap v2015: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:05 smithi078 ceph-mon[99959]: pgmap v2015: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:07 smithi146 ceph-mon[103278]: pgmap v2016: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:07 smithi078 ceph-mon[91136]: pgmap v2016: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:07 smithi078 ceph-mon[99959]: pgmap v2016: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:09 smithi146 ceph-mon[103278]: pgmap v2017: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:09 smithi078 ceph-mon[99959]: pgmap v2017: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:09 smithi078 ceph-mon[91136]: pgmap v2017: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:27:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:27:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:10.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:10.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:10.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:11 smithi146 ceph-mon[103278]: pgmap v2018: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:11 smithi078 ceph-mon[91136]: pgmap v2018: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:11 smithi078 ceph-mon[99959]: pgmap v2018: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:12 smithi146 ceph-mon[103278]: pgmap v2019: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[91136]: pgmap v2019: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:13.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:12 smithi078 ceph-mon[99959]: pgmap v2019: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:13.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:13.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:13.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:14 smithi146 ceph-mon[103278]: pgmap v2020: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:14 smithi078 ceph-mon[91136]: pgmap v2020: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:14 smithi078 ceph-mon[99959]: pgmap v2020: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:17 smithi146 ceph-mon[103278]: pgmap v2021: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[99959]: pgmap v2021: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[91136]: pgmap v2021: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:19 smithi146 ceph-mon[103278]: pgmap v2022: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:19 smithi078 ceph-mon[91136]: pgmap v2022: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:19 smithi078 ceph-mon[99959]: pgmap v2022: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:20.403 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:20.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:20.403 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:20.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:20.403 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:20.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:20.747 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:27:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:27:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:21 smithi146 ceph-mon[103278]: pgmap v2023: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:21 smithi078 ceph-mon[91136]: pgmap v2023: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:21 smithi078 ceph-mon[99959]: pgmap v2023: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:23.139 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:27:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:23 smithi146 ceph-mon[103278]: pgmap v2024: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-mon[91136]: pgmap v2024: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-mon[99959]: pgmap v2024: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:23.667 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:27:23.667 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 78m - - 2024-09-20T21:27:23.667 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 77m - - 2024-09-20T21:27:23.667 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (72m) 11s ago 73m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:27:23.667 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (72m) 7s ago 72m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (65m) 11s ago 70m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (64m) 7s ago 77m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (66m) 11s ago 78m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (64m) 11s ago 79m 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (63m) 7s ago 77m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (63m) 11s ago 77m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (72m) 11s ago 72m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (72m) 7s ago 72m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (62m) 11s ago 76m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (61m) 11s ago 76m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (60m) 11s ago 76m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (60m) 11s ago 75m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:27:23.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (59m) 7s ago 75m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (58m) 7s ago 74m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (57m) 7s ago 74m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (56m) 7s ago 74m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (65m) 7s ago 73m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (56m) 11s ago 71m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (56m) 7s ago 71m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (56m) 11s ago 71m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:27:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (56m) 7s ago 71m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:27:23.978 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:27:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:27:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:27:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:27:23.980 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:27:23.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:23.809Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:23.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:23.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:23.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:23.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:24.251 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:27:24.251 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:27:24.251 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:27:24.252 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:27:24.252 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:27:24.252 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:27:24.252 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:27:24.252 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:27:24.252 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:27:24.253 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:27:24.253 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:27:24.253 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:27:24.253 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:27:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:27:24.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:24 smithi078 ceph-mon[99959]: from='client.68534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:24.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:24 smithi078 ceph-mon[99959]: from='client.68540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:24.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3175614362' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:24.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:24 smithi078 ceph-mon[91136]: from='client.68534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:24.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:24 smithi078 ceph-mon[91136]: from='client.68540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:24.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3175614362' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:24 smithi146 ceph-mon[103278]: from='client.68534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:24 smithi146 ceph-mon[103278]: from='client.68540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3175614362' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:24.558 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:27:24.559 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:24.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:24.559 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:24.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:25 smithi146 ceph-mon[103278]: from='client.68546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:25 smithi146 ceph-mon[103278]: pgmap v2025: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:25 smithi146 ceph-mon[103278]: from='client.48639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1845942064' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[91136]: from='client.68546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[91136]: pgmap v2025: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[91136]: from='client.48639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1845942064' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[99959]: from='client.68546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[99959]: pgmap v2025: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[99959]: from='client.48639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1845942064' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:27 smithi146 ceph-mon[103278]: pgmap v2026: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:27 smithi078 ceph-mon[99959]: pgmap v2026: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:27 smithi078 ceph-mon[91136]: pgmap v2026: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:29 smithi146 ceph-mon[103278]: pgmap v2027: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:29 smithi078 ceph-mon[99959]: pgmap v2027: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:29 smithi078 ceph-mon[91136]: pgmap v2027: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:27:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T21:27:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:30.373Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:30.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:30.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:31 smithi146 ceph-mon[103278]: pgmap v2028: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:31 smithi078 ceph-mon[91136]: pgmap v2028: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:31 smithi078 ceph-mon[99959]: pgmap v2028: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:32 smithi146 ceph-mon[103278]: pgmap v2029: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[99959]: pgmap v2029: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:32 smithi078 ceph-mon[91136]: pgmap v2029: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:33.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:33.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:33.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:35 smithi146 ceph-mon[103278]: pgmap v2030: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:35 smithi078 ceph-mon[91136]: pgmap v2030: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:35 smithi078 ceph-mon[99959]: pgmap v2030: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:37 smithi146 ceph-mon[103278]: pgmap v2031: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[91136]: pgmap v2031: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[99959]: pgmap v2031: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:39.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:39 smithi078 ceph-mon[99959]: pgmap v2032: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:39.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:39 smithi078 ceph-mon[91136]: pgmap v2032: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:39 smithi146 ceph-mon[103278]: pgmap v2032: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:40.480 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:40.374Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:40.480 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:40.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:40.480 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:40.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:27:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T21:27:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:41 smithi078 ceph-mon[91136]: pgmap v2033: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:41 smithi078 ceph-mon[99959]: pgmap v2033: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:41.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:41 smithi146 ceph-mon[103278]: pgmap v2033: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:43 smithi146 ceph-mon[103278]: pgmap v2034: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-mon[99959]: pgmap v2034: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-mon[91136]: pgmap v2034: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:43.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:43.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:43.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:45 smithi146 ceph-mon[103278]: pgmap v2035: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:45 smithi078 ceph-mon[91136]: pgmap v2035: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:45 smithi078 ceph-mon[99959]: pgmap v2035: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:47 smithi146 ceph-mon[103278]: pgmap v2036: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:47 smithi078 ceph-mon[91136]: pgmap v2036: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:47 smithi078 ceph-mon[99959]: pgmap v2036: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:49 smithi146 ceph-mon[103278]: pgmap v2037: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:49 smithi078 ceph-mon[91136]: pgmap v2037: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:49 smithi078 ceph-mon[99959]: pgmap v2037: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:27:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:27:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:50.375Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:50.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:50.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:51.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:51 smithi146 ceph-mon[103278]: pgmap v2038: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:51.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:51 smithi078 ceph-mon[91136]: pgmap v2038: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:51.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:51 smithi078 ceph-mon[99959]: pgmap v2038: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:52 smithi146 ceph-mon[103278]: pgmap v2039: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[91136]: pgmap v2039: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:27:53.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:52 smithi078 ceph-mon[99959]: pgmap v2039: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:27:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:53.812Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:53.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:27:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:27:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:27:53.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:27:54.854 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:27:55.380 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:27:55.380 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 78m - - 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 78m - - 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (72m) 3s ago 73m 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (73m) 19s ago 73m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (66m) 3s ago 71m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (64m) 19s ago 77m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (66m) 3s ago 79m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (65m) 3s ago 79m 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (63m) 19s ago 78m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (64m) 3s ago 78m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (73m) 3s ago 73m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (73m) 19s ago 73m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (62m) 3s ago 77m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:27:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (61m) 3s ago 77m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (61m) 3s ago 76m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (60m) 3s ago 76m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (60m) 19s ago 75m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (59m) 19s ago 75m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (58m) 19s ago 75m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (57m) 19s ago 74m 60.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (66m) 19s ago 74m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (57m) 3s ago 71m 87.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (56m) 19s ago 71m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (56m) 3s ago 71m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:27:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (56m) 19s ago 71m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:27:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:55 smithi146 ceph-mon[103278]: pgmap v2040: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:55 smithi078 ceph-mon[99959]: pgmap v2040: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:55 smithi078 ceph-mon[91136]: pgmap v2040: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:55.692 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:27:55.715 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:27:55.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:27:55.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:27:55.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:27:55.717 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:27:55.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:27:55.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:27:55.717 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:27:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:27:55.961 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:27:56.276 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:27:56.276 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:27:56.276 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:56.276 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:27:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:56 smithi146 ceph-mon[103278]: from='client.68570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:56 smithi146 ceph-mon[103278]: from='client.48657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:56 smithi146 ceph-mon[103278]: from='client.68582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1053924850' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[99959]: from='client.68570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[99959]: from='client.48657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[99959]: from='client.68582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1053924850' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[91136]: from='client.68570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[91136]: from='client.48657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[91136]: from='client.68582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:56.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1053924850' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:27:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:57 smithi146 ceph-mon[103278]: from='client.48675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:57 smithi146 ceph-mon[103278]: pgmap v2041: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:27:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3661778285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[91136]: from='client.48675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[91136]: pgmap v2041: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3661778285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[99959]: from='client.48675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[99959]: pgmap v2041: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3661778285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:27:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:57.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:27:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:27:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:27:59 smithi146 ceph-mon[103278]: pgmap v2042: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:27:59 smithi078 ceph-mon[91136]: pgmap v2042: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:27:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:27:59 smithi078 ceph-mon[99959]: pgmap v2042: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:00.780 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:28:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:28:00.781 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:00.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:00.781 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:00.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:00.781 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:00.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:01.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:01 smithi078 ceph-mon[91136]: pgmap v2043: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:01.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:01 smithi078 ceph-mon[99959]: pgmap v2043: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:01 smithi146 ceph-mon[103278]: pgmap v2043: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:03 smithi146 ceph-mon[103278]: pgmap v2044: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-mon[91136]: pgmap v2044: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-mon[99959]: pgmap v2044: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:03.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:03.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:03.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:05 smithi146 ceph-mon[103278]: pgmap v2045: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:05 smithi078 ceph-mon[91136]: pgmap v2045: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:05 smithi078 ceph-mon[99959]: pgmap v2045: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:07 smithi146 ceph-mon[103278]: pgmap v2046: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:07 smithi078 ceph-mon[91136]: pgmap v2046: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:07 smithi078 ceph-mon[99959]: pgmap v2046: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:09 smithi146 ceph-mon[103278]: pgmap v2047: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:09 smithi078 ceph-mon[91136]: pgmap v2047: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:09 smithi078 ceph-mon[99959]: pgmap v2047: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:10.689 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:28:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:28:10.689 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:10.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:10.690 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:10.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:10.690 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:10.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:11 smithi146 ceph-mon[103278]: pgmap v2048: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:11 smithi078 ceph-mon[99959]: pgmap v2048: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:11 smithi078 ceph-mon[91136]: pgmap v2048: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:13 smithi146 ceph-mon[103278]: pgmap v2049: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[91136]: pgmap v2049: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:13.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-mon[99959]: pgmap v2049: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:14.123 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:13.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:14.123 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:13.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:14.123 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:13.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:15.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:15 smithi146 ceph-mon[103278]: pgmap v2050: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:15 smithi078 ceph-mon[91136]: pgmap v2050: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:15 smithi078 ceph-mon[99959]: pgmap v2050: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:17 smithi146 ceph-mon[103278]: pgmap v2051: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[91136]: pgmap v2051: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[99959]: pgmap v2051: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:19 smithi146 ceph-mon[103278]: pgmap v2052: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:19 smithi078 ceph-mon[99959]: pgmap v2052: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:19 smithi078 ceph-mon[91136]: pgmap v2052: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:20.675 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:28:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:28:20.675 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:20.377Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:20.675 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:20.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:20.675 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:20.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:21.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:21 smithi078 ceph-mon[91136]: pgmap v2053: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:21.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:21 smithi078 ceph-mon[99959]: pgmap v2053: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:21 smithi146 ceph-mon[103278]: pgmap v2053: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:23 smithi146 ceph-mon[103278]: pgmap v2054: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-mon[91136]: pgmap v2054: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-mon[99959]: pgmap v2054: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:23.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:23.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:23.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:25 smithi146 ceph-mon[103278]: pgmap v2055: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:25 smithi078 ceph-mon[99959]: pgmap v2055: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:25 smithi078 ceph-mon[91136]: pgmap v2055: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:26.568 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:28:27.094 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:28:27.094 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 79m - - 2024-09-20T21:28:27.094 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 78m - - 2024-09-20T21:28:27.094 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (73m) 14s ago 74m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (74m) 10s ago 74m 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (66m) 14s ago 72m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (65m) 10s ago 78m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (67m) 14s ago 79m 554M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (65m) 14s ago 80m 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (64m) 10s ago 78m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (64m) 14s ago 78m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (73m) 14s ago 73m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (73m) 10s ago 73m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (63m) 14s ago 77m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (62m) 14s ago 77m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (61m) 14s ago 77m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:28:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (61m) 14s ago 76m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (60m) 10s ago 76m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (59m) 10s ago 75m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (58m) 10s ago 75m 65.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (57m) 10s ago 75m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (66m) 10s ago 74m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (57m) 14s ago 72m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (57m) 10s ago 72m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (57m) 14s ago 72m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:28:27.096 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (57m) 10s ago 72m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:28:27.409 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:27.410 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:28:27.411 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:28:27.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:27 smithi078 ceph-mon[99959]: pgmap v2056: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:27.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:27 smithi078 ceph-mon[91136]: pgmap v2056: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:27 smithi146 ceph-mon[103278]: pgmap v2056: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:27.681 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:28:27.682 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:28:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:28:27.683 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:28:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:28:27.989 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:28:27.989 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:27.989 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:28 smithi146 ceph-mon[103278]: from='client.48687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:28 smithi146 ceph-mon[103278]: from='client.68612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:28 smithi146 ceph-mon[103278]: from='client.48699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/268218414' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1704010471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[91136]: from='client.48687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[91136]: from='client.68612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[91136]: from='client.48699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/268218414' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1704010471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[99959]: from='client.48687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[99959]: from='client.68612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[99959]: from='client.48699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/268218414' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1704010471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:28:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:29 smithi146 ceph-mon[103278]: from='client.48711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:29.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:29 smithi146 ceph-mon[103278]: pgmap v2057: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:29 smithi078 ceph-mon[99959]: from='client.48711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:29 smithi078 ceph-mon[99959]: pgmap v2057: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:29 smithi078 ceph-mon[91136]: from='client.48711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:29 smithi078 ceph-mon[91136]: pgmap v2057: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:28:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:28:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:30.378Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:30.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:30.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:31 smithi146 ceph-mon[103278]: pgmap v2058: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:31 smithi078 ceph-mon[99959]: pgmap v2058: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:31 smithi078 ceph-mon[91136]: pgmap v2058: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:33.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:33.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[99959]: pgmap v2059: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:33.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:33.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[91136]: pgmap v2059: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:33.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:33 smithi146 ceph-mon[103278]: pgmap v2059: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:33.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:33.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:33.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:35 smithi146 ceph-mon[103278]: pgmap v2060: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:35 smithi078 ceph-mon[91136]: pgmap v2060: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:35.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:35 smithi078 ceph-mon[99959]: pgmap v2060: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:37 smithi146 ceph-mon[103278]: pgmap v2061: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[91136]: pgmap v2061: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[99959]: pgmap v2061: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:39 smithi146 ceph-mon[103278]: pgmap v2062: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:39 smithi078 ceph-mon[91136]: pgmap v2062: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:39 smithi078 ceph-mon[99959]: pgmap v2062: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:40.683 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:28:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:28:40.684 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:40.380Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:40.684 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:40.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:40.684 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:40.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:41.344 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:41 smithi146 ceph-mon[103278]: pgmap v2063: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:41.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:41 smithi078 ceph-mon[91136]: pgmap v2063: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:41.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:41 smithi078 ceph-mon[99959]: pgmap v2063: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:43 smithi146 ceph-mon[103278]: pgmap v2064: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-mon[91136]: pgmap v2064: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-mon[99959]: pgmap v2064: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:43.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:43.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:43.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:45 smithi146 ceph-mon[103278]: pgmap v2065: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:45 smithi078 ceph-mon[99959]: pgmap v2065: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:45 smithi078 ceph-mon[91136]: pgmap v2065: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:47 smithi078 ceph-mon[91136]: pgmap v2066: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:47 smithi078 ceph-mon[99959]: pgmap v2066: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:47 smithi146 ceph-mon[103278]: pgmap v2066: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:49 smithi078 ceph-mon[91136]: pgmap v2067: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:49 smithi078 ceph-mon[99959]: pgmap v2067: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:49 smithi146 ceph-mon[103278]: pgmap v2067: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:28:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:28:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:50.381Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:50.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:50.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:51 smithi078 ceph-mon[91136]: pgmap v2068: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:51 smithi078 ceph-mon[99959]: pgmap v2068: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:51 smithi146 ceph-mon[103278]: pgmap v2068: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:53 smithi146 ceph-mon[103278]: pgmap v2069: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[91136]: pgmap v2069: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:53.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[99959]: pgmap v2069: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:53.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:28:53.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:53.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:53.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:53.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:28:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:28:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:28:53.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:28:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:55 smithi146 ceph-mon[103278]: pgmap v2070: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:55 smithi078 ceph-mon[91136]: pgmap v2070: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:55 smithi078 ceph-mon[99959]: pgmap v2070: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:57 smithi146 ceph-mon[103278]: pgmap v2071: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[91136]: pgmap v2071: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[99959]: pgmap v2071: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:28:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:28:58.287 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:28:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 79m - - 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 79m - - 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (73m) 6s ago 74m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (74m) 1s ago 74m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (67m) 6s ago 72m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (65m) 1s ago 78m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (67m) 6s ago 80m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (66m) 6s ago 80m 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (64m) 1s ago 79m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:28:58.816 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (65m) 6s ago 79m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (74m) 6s ago 74m 15.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (74m) 1s ago 74m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (63m) 6s ago 78m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (62m) 6s ago 78m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (62m) 6s ago 77m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (61m) 6s ago 77m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (61m) 1s ago 76m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (60m) 1s ago 76m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (59m) 1s ago 76m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (58m) 1s ago 75m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (67m) 1s ago 75m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (58m) 6s ago 72m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (57m) 1s ago 72m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:28:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (58m) 6s ago 72m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:28:58.818 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (57m) 1s ago 72m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:28:59.126 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:28:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:28:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:28:59.127 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:28:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:28:59.399 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:28:59.399 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:28:59.399 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:28:59.711 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:28:59.711 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:59.712 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:28:59.712 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:59.712 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:28:59.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:59 smithi078 ceph-mon[99959]: pgmap v2072: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:59.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:59 smithi078 ceph-mon[99959]: from='client.68642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:59.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:28:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1047289183' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:59.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:59 smithi078 ceph-mon[91136]: pgmap v2072: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:59.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:59 smithi078 ceph-mon[91136]: from='client.68642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:59.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:28:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1047289183' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:28:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:59 smithi146 ceph-mon[103278]: pgmap v2072: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:28:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:59 smithi146 ceph-mon[103278]: from='client.68642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:28:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:28:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1047289183' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:00 smithi146 ceph-mon[103278]: from='client.68648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:00 smithi146 ceph-mon[103278]: from='client.48735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:00 smithi146 ceph-mon[103278]: from='client.68666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/991889948' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:29:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:29:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[99959]: from='client.68648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[99959]: from='client.48735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[99959]: from='client.68666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/991889948' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[91136]: from='client.68648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[91136]: from='client.48735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[91136]: from='client.68666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:00.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/991889948' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:00.382Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:00.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:00.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:01.653 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:01 smithi146 ceph-mon[103278]: pgmap v2073: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:01.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:01 smithi078 ceph-mon[91136]: pgmap v2073: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:01.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:01 smithi078 ceph-mon[99959]: pgmap v2073: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:03.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:03 smithi146 ceph-mon[103278]: pgmap v2074: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:03.520 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:03.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-mon[99959]: pgmap v2074: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:03.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:03.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-mon[91136]: pgmap v2074: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:03.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:04.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:03.817Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:04.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:03.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:04.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:03.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:05 smithi146 ceph-mon[103278]: pgmap v2075: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:05 smithi078 ceph-mon[99959]: pgmap v2075: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:05 smithi078 ceph-mon[91136]: pgmap v2075: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:07 smithi146 ceph-mon[103278]: pgmap v2076: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:07 smithi078 ceph-mon[91136]: pgmap v2076: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:07 smithi078 ceph-mon[99959]: pgmap v2076: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:09 smithi146 ceph-mon[103278]: pgmap v2077: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:09 smithi078 ceph-mon[91136]: pgmap v2077: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:09 smithi078 ceph-mon[99959]: pgmap v2077: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:29:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:29:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:10.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:10.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:10.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:11 smithi146 ceph-mon[103278]: pgmap v2078: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:11 smithi078 ceph-mon[91136]: pgmap v2078: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:11 smithi078 ceph-mon[99959]: pgmap v2078: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:13 smithi146 ceph-mon[103278]: pgmap v2079: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[91136]: pgmap v2079: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[99959]: pgmap v2079: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:13.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:13.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:13.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:15 smithi146 ceph-mon[103278]: pgmap v2080: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:15 smithi078 ceph-mon[91136]: pgmap v2080: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:15 smithi078 ceph-mon[99959]: pgmap v2080: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:17 smithi146 ceph-mon[103278]: pgmap v2081: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[99959]: pgmap v2081: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[91136]: pgmap v2081: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:19 smithi146 ceph-mon[103278]: pgmap v2082: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:19 smithi078 ceph-mon[91136]: pgmap v2082: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:19 smithi078 ceph-mon[99959]: pgmap v2082: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:20.740 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:29:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:29:20.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:20.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:20.741 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:20.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:20.741 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:20.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:21 smithi146 ceph-mon[103278]: pgmap v2083: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:21 smithi078 ceph-mon[91136]: pgmap v2083: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:21 smithi078 ceph-mon[99959]: pgmap v2083: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:23 smithi146 ceph-mon[103278]: pgmap v2084: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:23.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-mon[99959]: pgmap v2084: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:23.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-mon[91136]: pgmap v2084: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:23.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:23.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:23.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:23.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:25 smithi146 ceph-mon[103278]: pgmap v2085: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:25 smithi078 ceph-mon[91136]: pgmap v2085: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:25 smithi078 ceph-mon[99959]: pgmap v2085: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:27 smithi146 ceph-mon[103278]: pgmap v2086: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:27 smithi078 ceph-mon[91136]: pgmap v2086: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:27 smithi078 ceph-mon[99959]: pgmap v2086: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:29 smithi146 ceph-mon[103278]: pgmap v2087: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:29 smithi078 ceph-mon[91136]: pgmap v2087: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:29 smithi078 ceph-mon[99959]: pgmap v2087: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:30.004 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:29:30.534 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 80m - - 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 79m - - 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (74m) 18s ago 75m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (75m) 13s ago 75m 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (67m) 18s ago 73m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (66m) 13s ago 79m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (68m) 18s ago 80m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (66m) 18s ago 81m 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (65m) 13s ago 79m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (65m) 18s ago 79m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (74m) 18s ago 74m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:29:30.535 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (74m) 13s ago 74m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (64m) 18s ago 78m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (63m) 18s ago 78m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (62m) 18s ago 78m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (62m) 18s ago 77m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (61m) 13s ago 77m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (60m) 13s ago 77m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (59m) 13s ago 76m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (58m) 13s ago 76m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (67m) 13s ago 75m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (58m) 18s ago 73m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (58m) 13s ago 73m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (58m) 18s ago 73m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:29:30.536 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (58m) 13s ago 73m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:29:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:29:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:29:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:30.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:30.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:30.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:29:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:29:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:29:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:29:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:29:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:29:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:29:31.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:29:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:29:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:29:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:29:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:29:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:29:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:31 smithi146 ceph-mon[103278]: from='client.68678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:31 smithi146 ceph-mon[103278]: pgmap v2088: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:31 smithi146 ceph-mon[103278]: from='client.68684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2157027054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2188927832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[99959]: from='client.68678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[99959]: pgmap v2088: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[99959]: from='client.68684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2157027054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2188927832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[91136]: from='client.68678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[91136]: pgmap v2088: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:31.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[91136]: from='client.68684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:31.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2157027054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:29:31.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2188927832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:29:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:32 smithi078 ceph-mon[91136]: from='client.48771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:32 smithi078 ceph-mon[91136]: from='client.48783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:32 smithi078 ceph-mon[99959]: from='client.48771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:32 smithi078 ceph-mon[99959]: from='client.48783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:32 smithi146 ceph-mon[103278]: from='client.48771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:32 smithi146 ceph-mon[103278]: from='client.48783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:29:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:33 smithi146 ceph-mon[103278]: pgmap v2089: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[91136]: pgmap v2089: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[99959]: pgmap v2089: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:33.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:33.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:33.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:35 smithi146 ceph-mon[103278]: pgmap v2090: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:35 smithi078 ceph-mon[91136]: pgmap v2090: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:35 smithi078 ceph-mon[99959]: pgmap v2090: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:37 smithi146 ceph-mon[103278]: pgmap v2091: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[99959]: pgmap v2091: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[91136]: pgmap v2091: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:37.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:39 smithi146 ceph-mon[103278]: pgmap v2092: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:39 smithi078 ceph-mon[99959]: pgmap v2092: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:39 smithi078 ceph-mon[91136]: pgmap v2092: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:40.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:29:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:29:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:40.386Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:40.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:40.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:41 smithi146 ceph-mon[103278]: pgmap v2093: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:41 smithi078 ceph-mon[91136]: pgmap v2093: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:41 smithi078 ceph-mon[99959]: pgmap v2093: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:43 smithi146 ceph-mon[103278]: pgmap v2094: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-mon[99959]: pgmap v2094: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-mon[91136]: pgmap v2094: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:44.139 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:43.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:44.139 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:43.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:44.139 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:43.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:44.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:44 smithi146 ceph-mon[103278]: pgmap v2095: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:44 smithi078 ceph-mon[91136]: pgmap v2095: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:44 smithi078 ceph-mon[99959]: pgmap v2095: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:47 smithi146 ceph-mon[103278]: pgmap v2096: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:47 smithi078 ceph-mon[91136]: pgmap v2096: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:47 smithi078 ceph-mon[99959]: pgmap v2096: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:49 smithi078 ceph-mon[99959]: pgmap v2097: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:49 smithi078 ceph-mon[91136]: pgmap v2097: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:49 smithi146 ceph-mon[103278]: pgmap v2097: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:29:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:29:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:50.386Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:50.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:50.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:51 smithi078 ceph-mon[99959]: pgmap v2098: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:51 smithi078 ceph-mon[91136]: pgmap v2098: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:51 smithi146 ceph-mon[103278]: pgmap v2098: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[99959]: pgmap v2099: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[91136]: pgmap v2099: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:53.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:53.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:53 smithi146 ceph-mon[103278]: pgmap v2099: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:29:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:53.821Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:53.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:29:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:29:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:29:53.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:29:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:55 smithi146 ceph-mon[103278]: pgmap v2100: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:55 smithi078 ceph-mon[91136]: pgmap v2100: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:55 smithi078 ceph-mon[99959]: pgmap v2100: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:57 smithi146 ceph-mon[103278]: pgmap v2101: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[99959]: pgmap v2101: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[91136]: pgmap v2101: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:29:57.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:57.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:29:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:29:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:29:59 smithi146 ceph-mon[103278]: pgmap v2102: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:29:59 smithi078 ceph-mon[99959]: pgmap v2102: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:29:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:29:59 smithi078 ceph-mon[91136]: pgmap v2102: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T21:29:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:00.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:00.387Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:00.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:00.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:00.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:30:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:30:01.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:01 smithi078 ceph-mon[99959]: pgmap v2103: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:01.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:01 smithi078 ceph-mon[91136]: pgmap v2103: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:01.719 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:30:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:01 smithi146 ceph-mon[103278]: pgmap v2103: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 80m - - 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 80m - - 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (74m) 9s ago 75m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (75m) 5s ago 75m 87.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (68m) 9s ago 73m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (66m) 5s ago 79m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (68m) 9s ago 81m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (67m) 9s ago 81m 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (65m) 5s ago 80m 110M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (66m) 9s ago 80m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (75m) 9s ago 75m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:30:02.247 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (75m) 5s ago 75m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (64m) 9s ago 79m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (63m) 9s ago 79m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (63m) 9s ago 78m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (62m) 9s ago 78m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (62m) 5s ago 77m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (61m) 5s ago 77m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (60m) 5s ago 77m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (59m) 5s ago 76m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (68m) 5s ago 76m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (59m) 9s ago 73m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (59m) 5s ago 74m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (59m) 9s ago 73m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:30:02.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (58m) 5s ago 73m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:30:02.563 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:30:02.830 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:30:02.830 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:30:02.830 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:30:02.830 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:30:02.830 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:30:02.830 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:30:02.831 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:30:03.138 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:03.139 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:30:03.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:30:03.139 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:30:03.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: from='client.48795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: from='client.48801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: pgmap v2104: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: from='client.48807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/250298618' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:03.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/821502209' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: from='client.48795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: from='client.48801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: pgmap v2104: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: from='client.48807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/250298618' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/821502209' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: from='client.48795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: from='client.48801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: pgmap v2104: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: from='client.48807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/250298618' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/821502209' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:03.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:03.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:03.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:04.420 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:04 smithi146 ceph-mon[103278]: from='client.68738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:04.591 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:04 smithi078 ceph-mon[99959]: from='client.68738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:04.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:04 smithi078 ceph-mon[91136]: from='client.68738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:05 smithi078 ceph-mon[91136]: pgmap v2105: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:05 smithi078 ceph-mon[99959]: pgmap v2105: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:05 smithi146 ceph-mon[103278]: pgmap v2105: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:07 smithi146 ceph-mon[103278]: pgmap v2106: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:07 smithi078 ceph-mon[99959]: pgmap v2106: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:07 smithi078 ceph-mon[91136]: pgmap v2106: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:09 smithi146 ceph-mon[103278]: pgmap v2107: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:09 smithi078 ceph-mon[99959]: pgmap v2107: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:09 smithi078 ceph-mon[91136]: pgmap v2107: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:30:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:30:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:10.388Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:10.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:10.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:11 smithi078 ceph-mon[91136]: pgmap v2108: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:11 smithi078 ceph-mon[99959]: pgmap v2108: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:11 smithi146 ceph-mon[103278]: pgmap v2108: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:13.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:13 smithi146 ceph-mon[103278]: pgmap v2109: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:13.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:13.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[91136]: pgmap v2109: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[99959]: pgmap v2109: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:13.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:13.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:13.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:13.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:15 smithi146 ceph-mon[103278]: pgmap v2110: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:15 smithi078 ceph-mon[99959]: pgmap v2110: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:15 smithi078 ceph-mon[91136]: pgmap v2110: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:17 smithi146 ceph-mon[103278]: pgmap v2111: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:17 smithi078 ceph-mon[91136]: pgmap v2111: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:17 smithi078 ceph-mon[99959]: pgmap v2111: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:19 smithi146 ceph-mon[103278]: pgmap v2112: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:19 smithi078 ceph-mon[91136]: pgmap v2112: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:19 smithi078 ceph-mon[99959]: pgmap v2112: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:30:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:30:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:20.388Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:20.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:20.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:21 smithi146 ceph-mon[103278]: pgmap v2113: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:21 smithi078 ceph-mon[91136]: pgmap v2113: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:21 smithi078 ceph-mon[99959]: pgmap v2113: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:23.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:23 smithi146 ceph-mon[103278]: pgmap v2114: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:23.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-mon[91136]: pgmap v2114: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-mon[99959]: pgmap v2114: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:24.096 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:23.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:24.096 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:23.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:24.096 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:23.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:25.462 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:25 smithi078 ceph-mon[99959]: pgmap v2115: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:25.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:25 smithi078 ceph-mon[91136]: pgmap v2115: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:25.538 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:25 smithi146 ceph-mon[103278]: pgmap v2115: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:27 smithi146 ceph-mon[103278]: pgmap v2116: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:27 smithi078 ceph-mon[91136]: pgmap v2116: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:27 smithi078 ceph-mon[99959]: pgmap v2116: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:29 smithi146 ceph-mon[103278]: pgmap v2117: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:29 smithi078 ceph-mon[99959]: pgmap v2117: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:29 smithi078 ceph-mon[91136]: pgmap v2117: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:30:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:30:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:30.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:30.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:30.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:31 smithi146 ceph-mon[103278]: pgmap v2118: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:31 smithi078 ceph-mon[91136]: pgmap v2118: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:31 smithi078 ceph-mon[99959]: pgmap v2118: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:33.439 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:30:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:33 smithi146 ceph-mon[103278]: pgmap v2119: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[91136]: pgmap v2119: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[99959]: pgmap v2119: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:33.965 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:30:33.965 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 81m - - 2024-09-20T21:30:33.965 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 80m - - 2024-09-20T21:30:33.965 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (75m) 0s ago 76m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:30:33.965 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (76m) 16s ago 76m 87.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (68m) 0s ago 74m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (67m) 16s ago 80m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (69m) 0s ago 81m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (67m) 0s ago 82m 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (66m) 16s ago 80m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (66m) 0s ago 80m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (75m) 0s ago 75m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (75m) 16s ago 75m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (65m) 0s ago 80m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (64m) 0s ago 79m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (63m) 0s ago 79m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (63m) 0s ago 78m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:30:33.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (62m) 16s ago 78m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (61m) 16s ago 78m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (60m) 16s ago 77m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (60m) 16s ago 77m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (68m) 16s ago 76m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (59m) 0s ago 74m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (59m) 16s ago 74m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (59m) 0s ago 74m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:30:33.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (59m) 16s ago 74m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:30:34.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:30:34.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:30:34.280 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:33.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:34.281 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:33.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:34.281 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:33.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:34 smithi078 ceph-mon[91136]: from='client.48831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/729187928' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:34 smithi078 ceph-mon[99959]: from='client.48831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/729187928' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:30:34.554 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:30:34.555 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:30:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:34 smithi146 ceph-mon[103278]: from='client.48831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/729187928' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:30:34.860 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:30:34.860 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:30:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:30:34.860 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:30:34.860 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:30:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:35 smithi146 ceph-mon[103278]: from='client.68756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:35 smithi146 ceph-mon[103278]: from='client.48843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:35 smithi146 ceph-mon[103278]: pgmap v2120: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3734337483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[99959]: from='client.68756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[99959]: from='client.48843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[99959]: pgmap v2120: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3734337483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[91136]: from='client.68756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[91136]: from='client.48843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[91136]: pgmap v2120: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3734337483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:30:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:36 smithi146 ceph-mon[103278]: from='client.48855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:36 smithi078 ceph-mon[91136]: from='client.48855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:36 smithi078 ceph-mon[99959]: from='client.48855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:30:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:37 smithi146 ceph-mon[103278]: pgmap v2121: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:37 smithi078 ceph-mon[91136]: pgmap v2121: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:37 smithi078 ceph-mon[99959]: pgmap v2121: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:38 smithi146 ceph-mon[103278]: pgmap v2122: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[91136]: pgmap v2122: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:38 smithi078 ceph-mon[99959]: pgmap v2122: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:30:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:30:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:40.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:40.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:40.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:41 smithi078 ceph-mon[91136]: pgmap v2123: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:41 smithi078 ceph-mon[99959]: pgmap v2123: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:41 smithi146 ceph-mon[103278]: pgmap v2123: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-mon[91136]: pgmap v2124: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-mon[99959]: pgmap v2124: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:43.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:43 smithi146 ceph-mon[103278]: pgmap v2124: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:43.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:43.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:43.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:43.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:43.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:45.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:45 smithi146 ceph-mon[103278]: pgmap v2125: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:45.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:45 smithi078 ceph-mon[91136]: pgmap v2125: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:45.778 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:45 smithi078 ceph-mon[99959]: pgmap v2125: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:47 smithi146 ceph-mon[103278]: pgmap v2126: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:47 smithi078 ceph-mon[91136]: pgmap v2126: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:47 smithi078 ceph-mon[99959]: pgmap v2126: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:49 smithi146 ceph-mon[103278]: pgmap v2127: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:49 smithi078 ceph-mon[91136]: pgmap v2127: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:49 smithi078 ceph-mon[99959]: pgmap v2127: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:30:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:30:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:30:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:50.391Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:50.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:50.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:51 smithi146 ceph-mon[103278]: pgmap v2128: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:51 smithi078 ceph-mon[99959]: pgmap v2128: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:51 smithi078 ceph-mon[91136]: pgmap v2128: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[91136]: pgmap v2129: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[99959]: pgmap v2129: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:53 smithi146 ceph-mon[103278]: pgmap v2129: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:30:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:30:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:53.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:53.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:30:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:30:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:30:53.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:30:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:55 smithi146 ceph-mon[103278]: pgmap v2130: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T21:30:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:55 smithi078 ceph-mon[91136]: pgmap v2130: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T21:30:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:55 smithi078 ceph-mon[99959]: pgmap v2130: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T21:30:57.652 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:57 smithi146 ceph-mon[103278]: pgmap v2131: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:30:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:57 smithi078 ceph-mon[91136]: pgmap v2131: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:30:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:57 smithi078 ceph-mon[99959]: pgmap v2131: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:30:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:30:58 smithi146 ceph-mon[103278]: pgmap v2132: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[91136]: pgmap v2132: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:30:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:30:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:30:58 smithi078 ceph-mon[99959]: pgmap v2132: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-20T21:31:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:31:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:31:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:00.391Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:00.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:00.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:01 smithi146 ceph-mon[103278]: pgmap v2133: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:31:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:01 smithi078 ceph-mon[99959]: pgmap v2133: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:31:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:01 smithi078 ceph-mon[91136]: pgmap v2133: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:31:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:03 smithi146 ceph-mon[103278]: pgmap v2134: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T21:31:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-mon[99959]: pgmap v2134: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T21:31:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-mon[91136]: pgmap v2134: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T21:31:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:03.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:03.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:03.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:05.163 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:31:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:05 smithi078 ceph-mon[91136]: pgmap v2135: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T21:31:05.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:05 smithi078 ceph-mon[99959]: pgmap v2135: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 81m - - 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 81m - - 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (75m) 12s ago 76m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (76m) 8s ago 76m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (69m) 12s ago 74m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (68m) 8s ago 80m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (69m) 12s ago 82m 555M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (68m) 12s ago 82m 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:31:05.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (66m) 8s ago 81m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (67m) 12s ago 81m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (76m) 12s ago 76m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (76m) 8s ago 76m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (65m) 12s ago 80m 67.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (64m) 12s ago 80m 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (64m) 12s ago 79m 62.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (63m) 12s ago 79m 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (63m) 8s ago 79m 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (62m) 8s ago 78m 62.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (61m) 8s ago 78m 65.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (60m) 8s ago 77m 60.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (69m) 8s ago 77m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (60m) 12s ago 75m 87.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:31:05.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (60m) 8s ago 75m 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:31:05.705 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (60m) 12s ago 74m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:31:05.705 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (59m) 8s ago 74m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:31:05.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:05 smithi146 ceph-mon[103278]: pgmap v2135: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T21:31:06.020 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:31:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:31:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:31:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:31:06.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:31:06.021 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:31:06.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:31:06.295 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:31:06.295 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:31:06.295 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:31:06.295 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:31:06.295 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:31:06.295 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:31:06.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:06 smithi078 ceph-mon[91136]: from='client.48867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:06.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:06 smithi078 ceph-mon[91136]: from='client.68792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:06.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4080973967' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:06.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:06 smithi078 ceph-mon[99959]: from='client.48867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:06.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:06 smithi078 ceph-mon[99959]: from='client.68792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:06.614 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4080973967' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:06.614 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:31:06.614 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:06.614 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:06.614 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:06.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:06 smithi146 ceph-mon[103278]: from='client.48867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:06.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:06 smithi146 ceph-mon[103278]: from='client.68792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:06.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4080973967' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:07.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:07 smithi146 ceph-mon[103278]: from='client.48879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:07.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:07 smithi146 ceph-mon[103278]: pgmap v2136: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 195 op/s 2024-09-20T21:31:07.492 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:07 smithi146 ceph-mon[103278]: from='client.48891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:07.492 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4216014790' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[99959]: from='client.48879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[99959]: pgmap v2136: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 195 op/s 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[99959]: from='client.48891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4216014790' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[91136]: from='client.48879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[91136]: pgmap v2136: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 195 op/s 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[91136]: from='client.48891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:07.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4216014790' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:09 smithi146 ceph-mon[103278]: pgmap v2137: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T21:31:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:09 smithi078 ceph-mon[99959]: pgmap v2137: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T21:31:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:09 smithi078 ceph-mon[91136]: pgmap v2137: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T21:31:10.688 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:31:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:31:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:10.392Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:10.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:10.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:11.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:11 smithi078 ceph-mon[99959]: pgmap v2138: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T21:31:11.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:11 smithi078 ceph-mon[91136]: pgmap v2138: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T21:31:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:11 smithi146 ceph-mon[103278]: pgmap v2138: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T21:31:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:13 smithi146 ceph-mon[103278]: pgmap v2139: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:31:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-mon[99959]: pgmap v2139: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:31:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-mon[91136]: pgmap v2139: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:31:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:13.829Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:13.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:13.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:15 smithi146 ceph-mon[103278]: pgmap v2140: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T21:31:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:15 smithi078 ceph-mon[91136]: pgmap v2140: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T21:31:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:15 smithi078 ceph-mon[99959]: pgmap v2140: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-09-20T21:31:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:17 smithi146 ceph-mon[103278]: pgmap v2141: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T21:31:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:17 smithi078 ceph-mon[91136]: pgmap v2141: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T21:31:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:17 smithi078 ceph-mon[99959]: pgmap v2141: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T21:31:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[91136]: pgmap v2142: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:31:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:18 smithi078 ceph-mon[99959]: pgmap v2142: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:31:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:18 smithi146 ceph-mon[103278]: pgmap v2142: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T21:31:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:31:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:31:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:20.393Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:20.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:20.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:21.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:21 smithi078 ceph-mon[91136]: pgmap v2143: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T21:31:21.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:21 smithi078 ceph-mon[99959]: pgmap v2143: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T21:31:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:21 smithi146 ceph-mon[103278]: pgmap v2143: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T21:31:23.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:23 smithi146 ceph-mon[103278]: pgmap v2144: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T21:31:23.566 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-mon[91136]: pgmap v2144: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T21:31:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-mon[99959]: pgmap v2144: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-20T21:31:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:23.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:23.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:23.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:25.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:25 smithi146 ceph-mon[103278]: pgmap v2145: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T21:31:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:25 smithi078 ceph-mon[99959]: pgmap v2145: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T21:31:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:25 smithi078 ceph-mon[91136]: pgmap v2145: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T21:31:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:27 smithi146 ceph-mon[103278]: pgmap v2146: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T21:31:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:27 smithi078 ceph-mon[99959]: pgmap v2146: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T21:31:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:27 smithi078 ceph-mon[91136]: pgmap v2146: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-20T21:31:28.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:28.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:29 smithi146 ceph-mon[103278]: pgmap v2147: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:29 smithi078 ceph-mon[99959]: pgmap v2147: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:29 smithi078 ceph-mon[91136]: pgmap v2147: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:31:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:31:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:30.394Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:30.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:30.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:31 smithi146 ceph-mon[103278]: pgmap v2148: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:31 smithi078 ceph-mon[91136]: pgmap v2148: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:31 smithi078 ceph-mon[99959]: pgmap v2148: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:33 smithi146 ceph-mon[103278]: pgmap v2149: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-mon[91136]: pgmap v2149: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-mon[99959]: pgmap v2149: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:33.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:33.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:33.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:35 smithi146 ceph-mon[103278]: pgmap v2150: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:35 smithi078 ceph-mon[91136]: pgmap v2150: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:35 smithi078 ceph-mon[99959]: pgmap v2150: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:36.906 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:31:37.433 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:31:37.433 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 82m - - 2024-09-20T21:31:37.433 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 82m - - 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (76m) 4s ago 77m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (77m) 19s ago 77m 87.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (69m) 4s ago 75m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (68m) 19s ago 81m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (70m) 4s ago 83m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (68m) 4s ago 83m 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (67m) 19s ago 81m 113M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (67m) 4s ago 81m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (76m) 4s ago 76m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (76m) 19s ago 76m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (66m) 4s ago 81m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (65m) 4s ago 80m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (64m) 4s ago 80m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:31:37.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (64m) 4s ago 79m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (63m) 19s ago 79m 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (62m) 19s ago 79m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (61m) 19s ago 78m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (61m) 19s ago 78m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (69m) 19s ago 77m 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (60m) 4s ago 75m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (60m) 19s ago 75m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (60m) 4s ago 75m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:31:37.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (60m) 19s ago 75m 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:31:37.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:37 smithi078 ceph-mon[99959]: pgmap v2151: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:37.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:37 smithi078 ceph-mon[91136]: pgmap v2151: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:37.743 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:31:37.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:31:37.745 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:31:37.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:31:37.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:31:37.745 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:31:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:37 smithi146 ceph-mon[103278]: pgmap v2151: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:38.018 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:31:38.018 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:31:38.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:31:38.332 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:31:38.332 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:31:38.332 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='client.48903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='client.48909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='client.48915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/821724790' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1711257499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='client.48903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='client.48909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='client.48915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/821724790' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1711257499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='client.48903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='client.48909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='client.48915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/821724790' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1711257499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:31:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:39 smithi146 ceph-mon[103278]: from='client.48927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:39 smithi146 ceph-mon[103278]: pgmap v2152: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:39 smithi078 ceph-mon[91136]: from='client.48927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:39 smithi078 ceph-mon[91136]: pgmap v2152: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:39 smithi078 ceph-mon[99959]: from='client.48927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:31:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:39 smithi078 ceph-mon[99959]: pgmap v2152: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:31:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:31:40.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:40.395Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:40.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:40.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:41 smithi146 ceph-mon[103278]: pgmap v2153: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:41.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:41 smithi078 ceph-mon[91136]: pgmap v2153: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:41.859 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:41 smithi078 ceph-mon[99959]: pgmap v2153: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:43 smithi146 ceph-mon[103278]: pgmap v2154: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-mon[91136]: pgmap v2154: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-mon[99959]: pgmap v2154: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:43.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:43.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:43.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:45.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:45 smithi146 ceph-mon[103278]: pgmap v2155: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:45 smithi078 ceph-mon[99959]: pgmap v2155: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:45 smithi078 ceph-mon[91136]: pgmap v2155: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:47.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:47 smithi146 ceph-mon[103278]: pgmap v2156: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:47 smithi078 ceph-mon[91136]: pgmap v2156: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:47 smithi078 ceph-mon[99959]: pgmap v2156: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:49.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:49 smithi078 ceph-mon[99959]: pgmap v2157: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:49.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:49 smithi078 ceph-mon[91136]: pgmap v2157: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:49 smithi146 ceph-mon[103278]: pgmap v2157: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:31:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:31:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:50.396Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:50.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:50.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:51 smithi146 ceph-mon[103278]: pgmap v2158: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:51 smithi078 ceph-mon[91136]: pgmap v2158: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:51 smithi078 ceph-mon[99959]: pgmap v2158: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:53 smithi146 ceph-mon[103278]: pgmap v2159: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-mon[91136]: pgmap v2159: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-mon[99959]: pgmap v2159: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:31:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:53.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:53.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:31:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:31:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:31:53.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:31:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:54 smithi146 ceph-mon[103278]: pgmap v2160: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[99959]: pgmap v2160: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:54 smithi078 ceph-mon[91136]: pgmap v2160: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:57 smithi146 ceph-mon[103278]: pgmap v2161: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:57.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:57 smithi078 ceph-mon[91136]: pgmap v2161: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:57 smithi078 ceph-mon[99959]: pgmap v2161: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[91136]: pgmap v2162: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:31:59 smithi078 ceph-mon[99959]: pgmap v2162: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:31:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:31:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:31:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:31:59 smithi146 ceph-mon[103278]: pgmap v2162: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:32:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:32:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:00.396Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:00.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:00.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:01 smithi146 ceph-mon[103278]: pgmap v2163: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:01 smithi078 ceph-mon[99959]: pgmap v2163: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:01 smithi078 ceph-mon[91136]: pgmap v2163: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:03 smithi146 ceph-mon[103278]: pgmap v2164: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-mon[91136]: pgmap v2164: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-mon[99959]: pgmap v2164: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:03.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:03.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:03.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:05 smithi146 ceph-mon[103278]: pgmap v2165: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:05 smithi078 ceph-mon[99959]: pgmap v2165: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:05 smithi078 ceph-mon[91136]: pgmap v2165: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:07.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:07 smithi146 ceph-mon[103278]: pgmap v2166: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:07.655 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:07 smithi078 ceph-mon[99959]: pgmap v2166: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:07.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:07 smithi078 ceph-mon[91136]: pgmap v2166: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:08.630 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 82m - - 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 82m - - 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (76m) 15s ago 77m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (77m) 11s ago 77m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (70m) 15s ago 75m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:32:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (69m) 11s ago 81m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (71m) 15s ago 83m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (69m) 15s ago 83m 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (67m) 11s ago 82m 109M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (68m) 15s ago 82m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (77m) 15s ago 77m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (77m) 11s ago 77m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (67m) 15s ago 81m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (66m) 15s ago 81m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (65m) 15s ago 80m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (64m) 15s ago 80m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (64m) 11s ago 80m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (63m) 11s ago 79m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (62m) 11s ago 79m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:32:09.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (61m) 11s ago 78m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:32:09.187 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (70m) 11s ago 78m 67.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:32:09.187 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (61m) 15s ago 76m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:32:09.187 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (61m) 11s ago 76m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:32:09.187 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (61m) 15s ago 76m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:32:09.187 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (61m) 11s ago 75m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:32:09.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:32:09.522 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:32:09.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:09 smithi078 ceph-mon[91136]: pgmap v2167: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:09.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:09 smithi078 ceph-mon[99959]: pgmap v2167: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:09 smithi146 ceph-mon[103278]: pgmap v2167: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:32:09.799 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:32:09.800 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:32:09.800 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:32:10.112 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:32:10.112 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:10.112 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:10.112 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:10.112 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:10.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[99959]: from='client.48939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[99959]: from='client.68864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[99959]: from='client.68870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1775015501' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:10.400 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1236974507' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:10.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[91136]: from='client.48939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[91136]: from='client.68864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[91136]: from='client.68870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1775015501' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:10.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1236974507' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:10 smithi146 ceph-mon[103278]: from='client.48939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:10 smithi146 ceph-mon[103278]: from='client.68864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:10 smithi146 ceph-mon[103278]: from='client.68870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1775015501' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1236974507' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:32:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:32:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:10.397Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:10.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:10.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:11 smithi078 ceph-mon[91136]: from='client.68882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:11 smithi078 ceph-mon[91136]: pgmap v2168: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:11 smithi078 ceph-mon[99959]: from='client.68882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:11 smithi078 ceph-mon[99959]: pgmap v2168: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:11 smithi146 ceph-mon[103278]: from='client.68882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:11 smithi146 ceph-mon[103278]: pgmap v2168: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:13.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-mon[99959]: pgmap v2169: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:13.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:13.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:13.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-mon[91136]: pgmap v2169: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:13.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:13.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:13 smithi146 ceph-mon[103278]: pgmap v2169: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:13.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:13.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:13.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:14 smithi146 ceph-mon[103278]: pgmap v2170: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[91136]: pgmap v2170: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:15.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:14 smithi078 ceph-mon[99959]: pgmap v2170: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:17 smithi146 ceph-mon[103278]: pgmap v2171: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:17 smithi078 ceph-mon[91136]: pgmap v2171: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:17 smithi078 ceph-mon[99959]: pgmap v2171: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:19 smithi146 ceph-mon[103278]: pgmap v2172: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[99959]: pgmap v2172: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:19 smithi078 ceph-mon[91136]: pgmap v2172: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:32:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:32:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:20.398Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:20.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:20.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:21 smithi146 ceph-mon[103278]: pgmap v2173: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:21 smithi078 ceph-mon[99959]: pgmap v2173: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:21 smithi078 ceph-mon[91136]: pgmap v2173: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:23 smithi146 ceph-mon[103278]: pgmap v2174: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-mon[91136]: pgmap v2174: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-mon[99959]: pgmap v2174: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:23.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:23.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:23.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:25 smithi146 ceph-mon[103278]: pgmap v2175: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:25 smithi078 ceph-mon[99959]: pgmap v2175: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:25 smithi078 ceph-mon[91136]: pgmap v2175: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:27.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:27 smithi146 ceph-mon[103278]: pgmap v2176: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:27 smithi078 ceph-mon[91136]: pgmap v2176: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:27 smithi078 ceph-mon[99959]: pgmap v2176: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:28.387 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:28.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:28.548 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:29.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:29 smithi146 ceph-mon[103278]: pgmap v2177: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:29.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:29 smithi078 ceph-mon[91136]: pgmap v2177: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:29.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:29 smithi078 ceph-mon[99959]: pgmap v2177: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:30.456 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:30.399Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:30.457 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:30.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:30.457 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:30.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:32:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:32:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:31 smithi146 ceph-mon[103278]: pgmap v2178: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:31 smithi078 ceph-mon[91136]: pgmap v2178: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:31 smithi078 ceph-mon[99959]: pgmap v2178: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:33 smithi146 ceph-mon[103278]: pgmap v2179: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-mon[99959]: pgmap v2179: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-mon[91136]: pgmap v2179: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:33.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:33.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:33.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:34 smithi146 ceph-mon[103278]: pgmap v2180: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[91136]: pgmap v2180: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:35.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:34 smithi078 ceph-mon[99959]: pgmap v2180: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:37 smithi146 ceph-mon[103278]: pgmap v2181: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:37 smithi078 ceph-mon[99959]: pgmap v2181: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:37 smithi078 ceph-mon[91136]: pgmap v2181: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[91136]: pgmap v2182: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[99959]: pgmap v2182: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:39 smithi146 ceph-mon[103278]: pgmap v2182: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:40.411 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:32:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:32:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:32:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:40.399Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:40.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:40.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:40.937 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 83m - - 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 83m - - 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (77m) 7s ago 78m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (78m) 2s ago 78m 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (70m) 7s ago 76m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (69m) 2s ago 82m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (71m) 7s ago 84m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (69m) 7s ago 84m 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (68m) 2s ago 82m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (68m) 7s ago 82m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (78m) 7s ago 78m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:32:40.938 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (77m) 2s ago 77m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (67m) 7s ago 82m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (66m) 7s ago 81m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (65m) 7s ago 81m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (65m) 7s ago 80m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (64m) 2s ago 80m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (63m) 2s ago 80m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (63m) 2s ago 79m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (62m) 2s ago 79m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (71m) 2s ago 78m 67.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (61m) 7s ago 76m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (61m) 2s ago 76m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (61m) 7s ago 76m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:32:40.939 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (61m) 2s ago 76m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:32:41.250 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:32:41.250 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:32:41.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:32:41.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:41.250 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:32:41.251 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:32:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:41 smithi146 ceph-mon[103278]: pgmap v2183: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:41 smithi146 ceph-mon[103278]: from='client.68894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1522142343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:32:41.519 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:32:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:32:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:32:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:32:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:32:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:32:41.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:41 smithi078 ceph-mon[99959]: pgmap v2183: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:41.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:41 smithi078 ceph-mon[99959]: from='client.68894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:41.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1522142343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:41.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:41 smithi078 ceph-mon[91136]: pgmap v2183: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:41.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:41 smithi078 ceph-mon[91136]: from='client.68894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:41.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1522142343' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:32:41.838 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:32:41.839 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:32:41.839 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:32:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:42 smithi146 ceph-mon[103278]: from='client.68900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:42 smithi146 ceph-mon[103278]: from='client.68906 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/404456851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:42 smithi078 ceph-mon[91136]: from='client.68900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:42 smithi078 ceph-mon[91136]: from='client.68906 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/404456851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:42 smithi078 ceph-mon[99959]: from='client.68900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:42 smithi078 ceph-mon[99959]: from='client.68906 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/404456851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:32:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:43 smithi146 ceph-mon[103278]: from='client.68918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:43 smithi146 ceph-mon[103278]: pgmap v2184: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:43.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[91136]: from='client.68918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[91136]: pgmap v2184: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[99959]: from='client.68918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[99959]: pgmap v2184: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:43.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:43.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:43.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:45 smithi146 ceph-mon[103278]: pgmap v2185: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:45 smithi078 ceph-mon[99959]: pgmap v2185: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:45 smithi078 ceph-mon[91136]: pgmap v2185: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:47.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:47 smithi146 ceph-mon[103278]: pgmap v2186: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:47 smithi078 ceph-mon[99959]: pgmap v2186: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:47 smithi078 ceph-mon[91136]: pgmap v2186: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:49 smithi078 ceph-mon[91136]: pgmap v2187: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:49 smithi078 ceph-mon[99959]: pgmap v2187: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:49 smithi146 ceph-mon[103278]: pgmap v2187: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:50.723 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:32:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:32:50.723 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:50.401Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:50.723 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:50.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:50.723 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:50.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:51 smithi078 ceph-mon[91136]: pgmap v2188: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:51 smithi078 ceph-mon[99959]: pgmap v2188: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:51 smithi146 ceph-mon[103278]: pgmap v2188: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:53 smithi146 ceph-mon[103278]: pgmap v2189: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-mon[91136]: pgmap v2189: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-mon[99959]: pgmap v2189: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:32:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:53.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:53.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:32:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:32:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:32:53.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:32:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:54 smithi146 ceph-mon[103278]: pgmap v2190: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[91136]: pgmap v2190: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:54 smithi078 ceph-mon[99959]: pgmap v2190: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:57 smithi146 ceph-mon[103278]: pgmap v2191: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:57 smithi078 ceph-mon[91136]: pgmap v2191: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:57 smithi078 ceph-mon[99959]: pgmap v2191: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:32:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:32:59.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:59 smithi146 ceph-mon[103278]: pgmap v2192: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:59.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[99959]: pgmap v2192: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:59.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[91136]: pgmap v2192: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:32:59.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:32:59.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:33:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:33:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:00.402Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:00.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:00.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:01 smithi146 ceph-mon[103278]: pgmap v2193: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:01 smithi078 ceph-mon[91136]: pgmap v2193: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:01 smithi078 ceph-mon[99959]: pgmap v2193: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:03 smithi146 ceph-mon[103278]: pgmap v2194: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-mon[91136]: pgmap v2194: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-mon[99959]: pgmap v2194: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:03.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:03.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:03.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:05 smithi146 ceph-mon[103278]: pgmap v2195: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:05 smithi078 ceph-mon[91136]: pgmap v2195: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:05 smithi078 ceph-mon[99959]: pgmap v2195: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:07 smithi146 ceph-mon[103278]: pgmap v2196: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:07 smithi078 ceph-mon[99959]: pgmap v2196: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:07 smithi078 ceph-mon[91136]: pgmap v2196: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:09.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:09 smithi146 ceph-mon[103278]: pgmap v2197: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:09.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:09 smithi078 ceph-mon[91136]: pgmap v2197: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:09.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:09 smithi078 ceph-mon[99959]: pgmap v2197: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:33:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:33:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:10.402Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:10.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:10.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:11 smithi146 ceph-mon[103278]: pgmap v2198: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:11 smithi078 ceph-mon[91136]: pgmap v2198: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:11 smithi078 ceph-mon[99959]: pgmap v2198: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:12.146 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:33:12.679 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:33:12.679 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 83m - - 2024-09-20T21:33:12.679 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 83m - - 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (77m) 18s ago 79m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (78m) 14s ago 78m 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (71m) 18s ago 76m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (70m) 14s ago 83m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (72m) 18s ago 84m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (70m) 18s ago 84m 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (69m) 14s ago 83m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (69m) 18s ago 83m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (78m) 18s ago 78m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (78m) 14s ago 78m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (68m) 18s ago 82m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (67m) 18s ago 82m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (66m) 18s ago 81m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (65m) 18s ago 81m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:33:12.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (65m) 14s ago 81m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (64m) 14s ago 80m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (63m) 14s ago 80m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (62m) 14s ago 79m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (71m) 14s ago 79m 67.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (62m) 18s ago 77m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (62m) 14s ago 77m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (62m) 18s ago 77m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:33:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (62m) 14s ago 77m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:33:12.990 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:33:12.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:33:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:33:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:33:12.992 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:33:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:33:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:33:13.260 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[99959]: from='client.68930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[99959]: pgmap v2199: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[99959]: from='client.68936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/38527728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[91136]: from='client.68930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[91136]: pgmap v2199: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[91136]: from='client.68936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/38527728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:13.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:13.569 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:33:13.569 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:13.569 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:13.569 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:13 smithi146 ceph-mon[103278]: from='client.68930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:13 smithi146 ceph-mon[103278]: pgmap v2199: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:13 smithi146 ceph-mon[103278]: from='client.68936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/38527728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:13.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:13.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:13.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:14 smithi146 ceph-mon[103278]: from='client.68942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:14 smithi146 ceph-mon[103278]: from='client.49035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2407894640' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[91136]: from='client.68942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[91136]: from='client.49035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2407894640' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[99959]: from='client.68942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[99959]: from='client.49035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2407894640' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:15 smithi146 ceph-mon[103278]: pgmap v2200: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:15 smithi078 ceph-mon[91136]: pgmap v2200: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:15 smithi078 ceph-mon[99959]: pgmap v2200: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:17 smithi146 ceph-mon[103278]: pgmap v2201: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:17 smithi078 ceph-mon[91136]: pgmap v2201: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:17 smithi078 ceph-mon[99959]: pgmap v2201: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:19 smithi146 ceph-mon[103278]: pgmap v2202: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[91136]: pgmap v2202: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[99959]: pgmap v2202: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:19.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:33:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:33:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:20.402Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:20.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:20.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:21.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:21 smithi078 ceph-mon[91136]: pgmap v2203: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:21.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:21 smithi078 ceph-mon[99959]: pgmap v2203: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:21 smithi146 ceph-mon[103278]: pgmap v2203: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:23 smithi146 ceph-mon[103278]: pgmap v2204: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-mon[99959]: pgmap v2204: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-mon[91136]: pgmap v2204: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:23.839Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:23.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:23.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:25 smithi146 ceph-mon[103278]: pgmap v2205: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:25.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:25 smithi078 ceph-mon[91136]: pgmap v2205: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:25.793 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:25 smithi078 ceph-mon[99959]: pgmap v2205: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:27 smithi146 ceph-mon[103278]: pgmap v2206: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:27 smithi078 ceph-mon[91136]: pgmap v2206: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:27 smithi078 ceph-mon[99959]: pgmap v2206: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:29 smithi146 ceph-mon[103278]: pgmap v2207: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:29 smithi078 ceph-mon[91136]: pgmap v2207: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:29 smithi078 ceph-mon[99959]: pgmap v2207: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:30.403Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:30.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:30.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:30.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:33:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:33:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:31 smithi146 ceph-mon[103278]: pgmap v2208: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:31.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:31 smithi078 ceph-mon[91136]: pgmap v2208: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:31.768 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:31 smithi078 ceph-mon[99959]: pgmap v2208: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:33 smithi146 ceph-mon[103278]: pgmap v2209: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-mon[91136]: pgmap v2209: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-mon[99959]: pgmap v2209: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:34.207 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:33.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:34.207 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:33.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:34.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:33.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:35 smithi146 ceph-mon[103278]: pgmap v2210: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[99959]: pgmap v2210: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:35 smithi078 ceph-mon[91136]: pgmap v2210: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:37 smithi146 ceph-mon[103278]: pgmap v2211: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:37 smithi078 ceph-mon[91136]: pgmap v2211: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:37 smithi078 ceph-mon[99959]: pgmap v2211: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:39 smithi146 ceph-mon[103278]: pgmap v2212: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[91136]: pgmap v2212: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[99959]: pgmap v2212: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:33:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:33:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:40.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:40.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:40.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:41 smithi146 ceph-mon[103278]: pgmap v2213: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:41 smithi078 ceph-mon[91136]: pgmap v2213: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:41 smithi078 ceph-mon[99959]: pgmap v2213: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:43 smithi146 ceph-mon[103278]: pgmap v2214: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-mon[91136]: pgmap v2214: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-mon[99959]: pgmap v2214: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:43.864 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:33:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:43.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:43.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:43.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 84m - - 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 84m - - 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (78m) 10s ago 79m 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (79m) 5s ago 79m 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (71m) 10s ago 77m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:33:44.389 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (70m) 5s ago 83m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (72m) 10s ago 85m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (70m) 10s ago 85m 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (69m) 5s ago 83m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (69m) 10s ago 83m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (79m) 10s ago 79m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (79m) 5s ago 78m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (68m) 10s ago 83m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (67m) 10s ago 82m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (66m) 10s ago 82m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (66m) 10s ago 82m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (65m) 5s ago 81m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (64m) 5s ago 81m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:33:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (64m) 5s ago 80m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:33:44.391 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (63m) 5s ago 80m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:33:44.391 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (72m) 5s ago 79m 69.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:33:44.391 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (62m) 10s ago 77m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:33:44.391 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (62m) 5s ago 77m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:33:44.391 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (62m) 10s ago 77m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:33:44.391 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (62m) 5s ago 77m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:33:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:33:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:33:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:33:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:33:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:33:44.700 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:33:44.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:33:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:33:44.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:33:45.276 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:33:45.276 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:33:45.276 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:33:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[91136]: from='client.49047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[91136]: from='client.49053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[91136]: pgmap v2215: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[91136]: from='client.68978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3757501255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1551072955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[99959]: from='client.49047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[99959]: from='client.49053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[99959]: pgmap v2215: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[99959]: from='client.68978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3757501255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:45.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1551072955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:45 smithi146 ceph-mon[103278]: from='client.49047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:45 smithi146 ceph-mon[103278]: from='client.49053 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:45 smithi146 ceph-mon[103278]: pgmap v2215: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:45 smithi146 ceph-mon[103278]: from='client.68978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3757501255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:33:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1551072955' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:33:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:46 smithi146 ceph-mon[103278]: from='client.49071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:46 smithi078 ceph-mon[99959]: from='client.49071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:46 smithi078 ceph-mon[91136]: from='client.49071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:33:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:47 smithi146 ceph-mon[103278]: pgmap v2216: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:47 smithi078 ceph-mon[91136]: pgmap v2216: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:47 smithi078 ceph-mon[99959]: pgmap v2216: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:49 smithi146 ceph-mon[103278]: pgmap v2217: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:49 smithi078 ceph-mon[99959]: pgmap v2217: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:49 smithi078 ceph-mon[91136]: pgmap v2217: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:33:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:33:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:50.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:50.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:50.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:51.462 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:51 smithi146 ceph-mon[103278]: pgmap v2218: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:51.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:51 smithi078 ceph-mon[99959]: pgmap v2218: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:51.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:51 smithi078 ceph-mon[91136]: pgmap v2218: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:53.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-mon[91136]: pgmap v2219: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:53.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:53.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-mon[99959]: pgmap v2219: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:53.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:53 smithi146 ceph-mon[103278]: pgmap v2219: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:33:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:53.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:53.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:33:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:33:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:33:53.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:33:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:55 smithi146 ceph-mon[103278]: pgmap v2220: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[91136]: pgmap v2220: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[99959]: pgmap v2220: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:57 smithi146 ceph-mon[103278]: pgmap v2221: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:57 smithi078 ceph-mon[91136]: pgmap v2221: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:57 smithi078 ceph-mon[99959]: pgmap v2221: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:33:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:33:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:59 smithi146 ceph-mon[103278]: pgmap v2222: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[99959]: pgmap v2222: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[91136]: pgmap v2222: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:33:59.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:34:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:34:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:00.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:00.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:00.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:01 smithi146 ceph-mon[103278]: pgmap v2223: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:01 smithi078 ceph-mon[91136]: pgmap v2223: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:01 smithi078 ceph-mon[99959]: pgmap v2223: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-mon[99959]: pgmap v2224: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:03.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-mon[91136]: pgmap v2224: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:03 smithi146 ceph-mon[103278]: pgmap v2224: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:03.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:03.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:03.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:05 smithi146 ceph-mon[103278]: pgmap v2225: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:05 smithi078 ceph-mon[91136]: pgmap v2225: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:05 smithi078 ceph-mon[99959]: pgmap v2225: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:07 smithi146 ceph-mon[103278]: pgmap v2226: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:07 smithi078 ceph-mon[91136]: pgmap v2226: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:07 smithi078 ceph-mon[99959]: pgmap v2226: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:09 smithi146 ceph-mon[103278]: pgmap v2227: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:09 smithi078 ceph-mon[91136]: pgmap v2227: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:09 smithi078 ceph-mon[99959]: pgmap v2227: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:34:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:34:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:10.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:10.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:10.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:11.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:11 smithi146 ceph-mon[103278]: pgmap v2228: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:11 smithi078 ceph-mon[91136]: pgmap v2228: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:11 smithi078 ceph-mon[99959]: pgmap v2228: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:13.380 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-mon[99959]: pgmap v2229: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:13.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:13.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:13.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-mon[91136]: pgmap v2229: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:13.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:13.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:13 smithi146 ceph-mon[103278]: pgmap v2229: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:14.182 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:13.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:14.182 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:13.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:14.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:13.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:15.571 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:34:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:15 smithi146 ceph-mon[103278]: pgmap v2230: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[91136]: pgmap v2230: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[99959]: pgmap v2230: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 84m - - 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 84m - - 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (79m) 1s ago 80m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (79m) 17s ago 79m 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (72m) 1s ago 77m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (71m) 17s ago 84m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (73m) 1s ago 85m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:34:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (71m) 1s ago 85m 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (70m) 17s ago 84m 111M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (70m) 1s ago 84m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (79m) 1s ago 79m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (79m) 17s ago 79m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (69m) 1s ago 83m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (68m) 1s ago 83m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (67m) 1s ago 82m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (66m) 1s ago 82m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (66m) 17s ago 82m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (65m) 17s ago 81m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (64m) 17s ago 81m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (63m) 17s ago 81m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (72m) 17s ago 80m 67.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:34:16.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (63m) 1s ago 78m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:34:16.095 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (63m) 17s ago 78m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:34:16.095 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (63m) 1s ago 78m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:34:16.095 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (63m) 17s ago 78m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:34:16.407 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:34:16.676 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:34:16.677 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:34:16.677 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:34:16.677 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:34:16.677 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:34:16.677 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:34:16.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1360464556' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:16.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1360464556' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1360464556' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:16.987 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:34:16.987 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:16.987 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:17 smithi146 ceph-mon[103278]: from='client.49083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:17 smithi146 ceph-mon[103278]: from='client.49089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:17 smithi146 ceph-mon[103278]: from='client.69014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:17 smithi146 ceph-mon[103278]: pgmap v2231: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3600646126' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[91136]: from='client.49083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[91136]: from='client.49089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[91136]: from='client.69014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[91136]: pgmap v2231: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3600646126' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[99959]: from='client.49083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[99959]: from='client.49089 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[99959]: from='client.69014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[99959]: pgmap v2231: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3600646126' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:18 smithi146 ceph-mon[103278]: from='client.49107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:18 smithi078 ceph-mon[91136]: from='client.49107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:18 smithi078 ceph-mon[99959]: from='client.49107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:19 smithi146 ceph-mon[103278]: pgmap v2232: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[91136]: pgmap v2232: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[99959]: pgmap v2232: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:19.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:19.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:34:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:34:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:20.407Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:20.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:20.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:21 smithi146 ceph-mon[103278]: pgmap v2233: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:21 smithi078 ceph-mon[99959]: pgmap v2233: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:21 smithi078 ceph-mon[91136]: pgmap v2233: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:23 smithi146 ceph-mon[103278]: pgmap v2234: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:23 smithi078 ceph-mon[91136]: pgmap v2234: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:23 smithi078 ceph-mon[99959]: pgmap v2234: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:23.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:23.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:23.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:25 smithi146 ceph-mon[103278]: pgmap v2235: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:25 smithi078 ceph-mon[91136]: pgmap v2235: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:25 smithi078 ceph-mon[99959]: pgmap v2235: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:26.267 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 21:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T21:34:25.849088508Z level=info msg="Completed cleanup jobs" duration=13.23695ms 2024-09-20T21:34:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:27 smithi146 ceph-mon[103278]: pgmap v2236: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:27 smithi078 ceph-mon[91136]: pgmap v2236: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:27 smithi078 ceph-mon[99959]: pgmap v2236: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:29 smithi146 ceph-mon[103278]: pgmap v2237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:29 smithi078 ceph-mon[91136]: pgmap v2237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:29 smithi078 ceph-mon[99959]: pgmap v2237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:34:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:34:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:30.408Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:30.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:30.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:31.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:31 smithi146 ceph-mon[103278]: pgmap v2238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:31 smithi078 ceph-mon[91136]: pgmap v2238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:31 smithi078 ceph-mon[99959]: pgmap v2238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:33.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:33 smithi078 ceph-mon[99959]: pgmap v2239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:33.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:33 smithi078 ceph-mon[91136]: pgmap v2239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:33 smithi146 ceph-mon[103278]: pgmap v2239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:34.028 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:33.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:34.028 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:33.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:34.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:33.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:34.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:34.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:35 smithi146 ceph-mon[103278]: pgmap v2240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[91136]: pgmap v2240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[99959]: pgmap v2240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:37 smithi146 ceph-mon[103278]: pgmap v2241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:37 smithi078 ceph-mon[91136]: pgmap v2241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:37 smithi078 ceph-mon[99959]: pgmap v2241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:39 smithi146 ceph-mon[103278]: pgmap v2242: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[99959]: pgmap v2242: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[91136]: pgmap v2242: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:39.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:34:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:34:40.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:40.409Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:40.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:40.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:40.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:41 smithi146 ceph-mon[103278]: pgmap v2243: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:41 smithi078 ceph-mon[99959]: pgmap v2243: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:41 smithi078 ceph-mon[91136]: pgmap v2243: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:43 smithi146 ceph-mon[103278]: pgmap v2244: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-mon[99959]: pgmap v2244: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-mon[91136]: pgmap v2244: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:43.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:43.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:43.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:45 smithi078 ceph-mon[99959]: pgmap v2245: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:45 smithi078 ceph-mon[91136]: pgmap v2245: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:45 smithi146 ceph-mon[103278]: pgmap v2245: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:34:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:47 smithi078 ceph-mon[91136]: pgmap v2246: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:47 smithi078 ceph-mon[99959]: pgmap v2246: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 85m - - 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 85m - - 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (79m) 13s ago 80m 30.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (80m) 8s ago 80m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (72m) 13s ago 78m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (71m) 8s ago 84m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (73m) 13s ago 86m 556M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (71m) 13s ago 86m 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (70m) 8s ago 85m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:34:47.817 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (70m) 13s ago 84m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (80m) 13s ago 80m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (80m) 8s ago 80m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (69m) 13s ago 84m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (68m) 13s ago 83m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (68m) 13s ago 83m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (67m) 13s ago 83m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (66m) 8s ago 82m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (65m) 8s ago 82m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (65m) 8s ago 81m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (64m) 8s ago 81m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (73m) 8s ago 80m 67.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:34:47.818 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (63m) 13s ago 78m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:34:47.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (63m) 8s ago 78m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:34:47.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (63m) 13s ago 78m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:34:47.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (63m) 8s ago 78m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:34:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:47 smithi146 ceph-mon[103278]: pgmap v2246: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:34:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:34:48.133 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:34:48.400 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:34:48.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:34:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:34:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:48.713 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:34:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:48.713 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:34:48.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:48 smithi078 ceph-mon[99959]: from='client.49119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:48.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4042348177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:48.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:48 smithi078 ceph-mon[91136]: from='client.49119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:48.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4042348177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:48 smithi146 ceph-mon[103278]: from='client.49119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4042348177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:34:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[91136]: from='client.49125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[91136]: from='client.49131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[91136]: pgmap v2247: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[91136]: from='client.69062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2319063260' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[99959]: from='client.49125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[99959]: from='client.49131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[99959]: pgmap v2247: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[99959]: from='client.69062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2319063260' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:49 smithi146 ceph-mon[103278]: from='client.49125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:49 smithi146 ceph-mon[103278]: from='client.49131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:49 smithi146 ceph-mon[103278]: pgmap v2247: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:49 smithi146 ceph-mon[103278]: from='client.69062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:34:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2319063260' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:34:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:50 smithi078 ceph-mon[91136]: pgmap v2248: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:50 smithi078 ceph-mon[99959]: pgmap v2248: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:34:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:34:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:50.410Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:50.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:50.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:50.792 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:50 smithi146 ceph-mon[103278]: pgmap v2248: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:53 smithi078 ceph-mon[91136]: pgmap v2249: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:53 smithi078 ceph-mon[99959]: pgmap v2249: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:53 smithi146 ceph-mon[103278]: pgmap v2249: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:53.846Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:53.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:34:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:34:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:34:53.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:34:54.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:54.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:34:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[91136]: pgmap v2250: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[99959]: pgmap v2250: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:55 smithi146 ceph-mon[103278]: pgmap v2250: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:57 smithi146 ceph-mon[103278]: pgmap v2251: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:57 smithi078 ceph-mon[91136]: pgmap v2251: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:57 smithi078 ceph-mon[99959]: pgmap v2251: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:34:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:34:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:59 smithi146 ceph-mon[103278]: pgmap v2252: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[91136]: pgmap v2252: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[99959]: pgmap v2252: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:34:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:34:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:35:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:35:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:00.411Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:00.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:00.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:01 smithi146 ceph-mon[103278]: pgmap v2253: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:01 smithi078 ceph-mon[99959]: pgmap v2253: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:01 smithi078 ceph-mon[91136]: pgmap v2253: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:03 smithi146 ceph-mon[103278]: pgmap v2254: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:03 smithi078 ceph-mon[91136]: pgmap v2254: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:03 smithi078 ceph-mon[99959]: pgmap v2254: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:03.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:03.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:03.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:05 smithi146 ceph-mon[103278]: pgmap v2255: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:05 smithi078 ceph-mon[91136]: pgmap v2255: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:05 smithi078 ceph-mon[99959]: pgmap v2255: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:07 smithi146 ceph-mon[103278]: pgmap v2256: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:07 smithi078 ceph-mon[91136]: pgmap v2256: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:07 smithi078 ceph-mon[99959]: pgmap v2256: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:09 smithi146 ceph-mon[103278]: pgmap v2257: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:09 smithi078 ceph-mon[91136]: pgmap v2257: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:09 smithi078 ceph-mon[99959]: pgmap v2257: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:35:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:35:10.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:10.411Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:10.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:10.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:10.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:11.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:11 smithi146 ceph-mon[103278]: pgmap v2258: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:11 smithi078 ceph-mon[99959]: pgmap v2258: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:11 smithi078 ceph-mon[91136]: pgmap v2258: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:13.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-mon[99959]: pgmap v2259: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:13.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:13.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-mon[91136]: pgmap v2259: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:13.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:13.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:13 smithi146 ceph-mon[103278]: pgmap v2259: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:13.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:13.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:13.848Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:13.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:13.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:13.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:13.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:14.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:14.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:15.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[91136]: pgmap v2260: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:15.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[99959]: pgmap v2260: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:15.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:15 smithi146 ceph-mon[103278]: pgmap v2260: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:17 smithi146 ceph-mon[103278]: pgmap v2261: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:17 smithi078 ceph-mon[91136]: pgmap v2261: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:17 smithi078 ceph-mon[99959]: pgmap v2261: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:19.009 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 85m - - 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 85m - - 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (80m) 4s ago 81m 30.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (80m) 0s ago 80m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (73m) 4s ago 78m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (72m) 0s ago 85m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (74m) 4s ago 86m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (72m) 4s ago 86m 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (71m) 0s ago 85m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (71m) 4s ago 85m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (80m) 4s ago 80m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (80m) 0s ago 80m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (70m) 4s ago 84m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (69m) 4s ago 84m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (68m) 4s ago 84m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (68m) 4s ago 83m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (67m) 0s ago 83m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (66m) 0s ago 82m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (65m) 0s ago 82m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (64m) 0s ago 82m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (73m) 0s ago 81m 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (64m) 4s ago 79m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (64m) 0s ago 79m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (64m) 4s ago 79m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (64m) 0s ago 79m 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:35:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:19 smithi146 ceph-mon[103278]: pgmap v2262: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[99959]: pgmap v2262: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[91136]: pgmap v2262: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:19.847 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:35:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:35:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:35:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:35:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:35:19.848 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:35:20.116 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:35:20.116 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:35:20.117 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:35:20.427 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:35:20.428 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:20.428 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:20.428 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:20.428 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:20 smithi146 ceph-mon[103278]: from='client.49155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:20 smithi146 ceph-mon[103278]: from='client.49161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:20 smithi146 ceph-mon[103278]: from='client.69086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2185745197' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/976999222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:35:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[99959]: from='client.49155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[99959]: from='client.49161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[99959]: from='client.69086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2185745197' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/976999222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[91136]: from='client.49155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[91136]: from='client.49161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[91136]: from='client.69086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2185745197' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/976999222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:20.412Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:20.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:20.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:21 smithi146 ceph-mon[103278]: from='client.69098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:21 smithi146 ceph-mon[103278]: pgmap v2263: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:21 smithi078 ceph-mon[91136]: from='client.69098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:21 smithi078 ceph-mon[91136]: pgmap v2263: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:21 smithi078 ceph-mon[99959]: from='client.69098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:21 smithi078 ceph-mon[99959]: pgmap v2263: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:23 smithi146 ceph-mon[103278]: pgmap v2264: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:23 smithi078 ceph-mon[91136]: pgmap v2264: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:23 smithi078 ceph-mon[99959]: pgmap v2264: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:23.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:23.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:23.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:25 smithi146 ceph-mon[103278]: pgmap v2265: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:25 smithi078 ceph-mon[91136]: pgmap v2265: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:25 smithi078 ceph-mon[99959]: pgmap v2265: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:27 smithi146 ceph-mon[103278]: pgmap v2266: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:27 smithi078 ceph-mon[91136]: pgmap v2266: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:27 smithi078 ceph-mon[99959]: pgmap v2266: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:29 smithi146 ceph-mon[103278]: pgmap v2267: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:29 smithi078 ceph-mon[99959]: pgmap v2267: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:29 smithi078 ceph-mon[91136]: pgmap v2267: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:35:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:35:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:30.412Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:30.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:30.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:31 smithi146 ceph-mon[103278]: pgmap v2268: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:31 smithi078 ceph-mon[91136]: pgmap v2268: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:31 smithi078 ceph-mon[99959]: pgmap v2268: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:33 smithi146 ceph-mon[103278]: pgmap v2269: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:33 smithi078 ceph-mon[91136]: pgmap v2269: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:33 smithi078 ceph-mon[99959]: pgmap v2269: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:33.850Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:33.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:33.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:34.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:34.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:35 smithi146 ceph-mon[103278]: pgmap v2270: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[99959]: pgmap v2270: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:35.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[91136]: pgmap v2270: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:35.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:35.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:37 smithi146 ceph-mon[103278]: pgmap v2271: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:37 smithi078 ceph-mon[91136]: pgmap v2271: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:37 smithi078 ceph-mon[99959]: pgmap v2271: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:39 smithi146 ceph-mon[103278]: pgmap v2272: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[91136]: pgmap v2272: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[99959]: pgmap v2272: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:35:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:35:40.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:40.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:40.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:40.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:40.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:41 smithi146 ceph-mon[103278]: pgmap v2273: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:41 smithi078 ceph-mon[91136]: pgmap v2273: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:41 smithi078 ceph-mon[99959]: pgmap v2273: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:43 smithi146 ceph-mon[103278]: pgmap v2274: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-mon[99959]: pgmap v2274: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-mon[91136]: pgmap v2274: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:43.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:43.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:43.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:44.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:45 smithi146 ceph-mon[103278]: pgmap v2275: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:45 smithi078 ceph-mon[91136]: pgmap v2275: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:45 smithi078 ceph-mon[99959]: pgmap v2275: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:47 smithi146 ceph-mon[103278]: pgmap v2276: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:47 smithi078 ceph-mon[91136]: pgmap v2276: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:47 smithi078 ceph-mon[99959]: pgmap v2276: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:49 smithi146 ceph-mon[103278]: pgmap v2277: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:49 smithi078 ceph-mon[91136]: pgmap v2277: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:49 smithi078 ceph-mon[99959]: pgmap v2277: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:50.726 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:35:50] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:35:50.727 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:50.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:50.727 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:50.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:50.727 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:50.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:50.727 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:35:51.251 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 86m - - 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 86m - - 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (80m) 16s ago 81m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (81m) 11s ago 81m 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (73m) 16s ago 79m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (72m) 11s ago 85m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (74m) 16s ago 87m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (73m) 16s ago 87m 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (71m) 11s ago 86m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:35:51.252 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (72m) 16s ago 85m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (81m) 16s ago 81m 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (81m) 11s ago 81m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (70m) 16s ago 85m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (69m) 16s ago 84m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (69m) 16s ago 84m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (68m) 16s ago 84m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (68m) 11s ago 83m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (66m) 11s ago 83m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (66m) 11s ago 83m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (65m) 11s ago 82m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (74m) 11s ago 82m 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (65m) 16s ago 79m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (64m) 11s ago 79m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (64m) 16s ago 79m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:35:51.253 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (64m) 11s ago 79m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:35:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:35:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:35:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:35:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:35:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:35:51.565 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:35:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:51 smithi146 ceph-mon[103278]: pgmap v2278: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:51 smithi078 ceph-mon[91136]: pgmap v2278: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:51 smithi078 ceph-mon[99959]: pgmap v2278: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:35:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:35:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:35:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:35:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:35:51.834 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:35:52.161 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:35:52.161 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:52.162 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:35:52.162 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:52.162 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:35:52.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:52 smithi146 ceph-mon[103278]: from='client.49191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:52 smithi146 ceph-mon[103278]: from='client.49197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:52 smithi146 ceph-mon[103278]: from='client.69122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4280825990' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:52.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/821467430' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[91136]: from='client.49191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[91136]: from='client.49197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[91136]: from='client.69122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4280825990' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/821467430' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[99959]: from='client.49191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[99959]: from='client.49197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[99959]: from='client.69122 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:52.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4280825990' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:35:52.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/821467430' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:35:53.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:53 smithi146 ceph-mon[103278]: from='client.49215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:53.643 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:53 smithi146 ceph-mon[103278]: pgmap v2279: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-mon[91136]: from='client.49215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-mon[91136]: pgmap v2279: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-mon[99959]: from='client.49215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:35:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-mon[99959]: pgmap v2279: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:53.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:53.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:35:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:35:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:35:53.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:35:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:35:55.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[99959]: pgmap v2280: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:55.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:55.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:55.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:55.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[91136]: pgmap v2280: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:55.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:55.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:55.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:55 smithi146 ceph-mon[103278]: pgmap v2280: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:35:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:35:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:57 smithi078 ceph-mon[91136]: pgmap v2281: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:57 smithi078 ceph-mon[99959]: pgmap v2281: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:57 smithi146 ceph-mon[103278]: pgmap v2281: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:35:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:35:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:35:59 smithi146 ceph-mon[103278]: pgmap v2282: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:35:59 smithi078 ceph-mon[91136]: pgmap v2282: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:35:59 smithi078 ceph-mon[99959]: pgmap v2282: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:36:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:36:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[91136]: pgmap v2283: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-mon[99959]: pgmap v2283: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:00.414Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:00.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:00.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:00 smithi146 ceph-mon[103278]: pgmap v2283: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:03 smithi146 ceph-mon[103278]: pgmap v2284: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:03 smithi078 ceph-mon[91136]: pgmap v2284: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:03 smithi078 ceph-mon[99959]: pgmap v2284: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:03.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:03.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:03.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:05 smithi146 ceph-mon[103278]: pgmap v2285: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:05 smithi078 ceph-mon[91136]: pgmap v2285: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:05 smithi078 ceph-mon[99959]: pgmap v2285: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:07.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:07 smithi078 ceph-mon[99959]: pgmap v2286: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:07.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:07 smithi078 ceph-mon[91136]: pgmap v2286: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:07 smithi146 ceph-mon[103278]: pgmap v2286: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:09 smithi146 ceph-mon[103278]: pgmap v2287: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:09 smithi078 ceph-mon[91136]: pgmap v2287: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:09 smithi078 ceph-mon[99959]: pgmap v2287: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:36:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:36:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:10.415Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:10.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:10.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:11.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:11 smithi078 ceph-mon[99959]: pgmap v2288: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:11.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:11 smithi078 ceph-mon[91136]: pgmap v2288: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:11 smithi146 ceph-mon[103278]: pgmap v2288: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:13 smithi146 ceph-mon[103278]: pgmap v2289: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-mon[91136]: pgmap v2289: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-mon[99959]: pgmap v2289: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:13.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:13.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:13.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:15.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:15 smithi146 ceph-mon[103278]: pgmap v2290: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:15.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[91136]: pgmap v2290: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[99959]: pgmap v2290: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:15.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:17 smithi146 ceph-mon[103278]: pgmap v2291: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:17 smithi078 ceph-mon[91136]: pgmap v2291: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:17 smithi078 ceph-mon[99959]: pgmap v2291: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:19.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:19 smithi146 ceph-mon[103278]: pgmap v2292: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:19 smithi078 ceph-mon[99959]: pgmap v2292: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:19 smithi078 ceph-mon[91136]: pgmap v2292: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:20.705 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:36:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:36:20.706 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:20.415Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:20.706 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:20.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:20.706 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:20.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:20 smithi146 ceph-mon[103278]: pgmap v2293: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[91136]: pgmap v2293: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:20 smithi078 ceph-mon[99959]: pgmap v2293: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:22.467 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 87m - - 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 86m - - 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (81m) 7s ago 82m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (81m) 3s ago 81m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (74m) 7s ago 79m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (73m) 3s ago 86m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:36:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (75m) 7s ago 87m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (73m) 7s ago 88m 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (72m) 3s ago 86m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (72m) 7s ago 86m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (81m) 7s ago 81m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (81m) 3s ago 81m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (71m) 7s ago 85m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (70m) 7s ago 85m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (69m) 7s ago 85m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (69m) 7s ago 84m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (68m) 3s ago 84m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (67m) 3s ago 83m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (66m) 3s ago 83m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (65m) 3s ago 83m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:36:23.000 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (74m) 3s ago 82m 67.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:36:23.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (65m) 7s ago 80m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:36:23.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (65m) 3s ago 80m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:36:23.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (65m) 7s ago 80m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:36:23.001 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (65m) 3s ago 80m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:36:23.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:36:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:36:23.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:36:23.312 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:36:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:23 smithi146 ceph-mon[103278]: pgmap v2294: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:23 smithi146 ceph-mon[103278]: from='client.49227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-mon[91136]: pgmap v2294: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-mon[91136]: from='client.49227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-mon[99959]: pgmap v2294: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-mon[99959]: from='client.49227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:23.581 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:36:23.582 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:36:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:36:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:23.892 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:23.892 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:24.246 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:23.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:24.246 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:23.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:24.246 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:23.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:24 smithi146 ceph-mon[103278]: from='client.49233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:24 smithi146 ceph-mon[103278]: from='client.69158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/92758274' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3179754471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[91136]: from='client.49233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[91136]: from='client.69158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/92758274' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3179754471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[99959]: from='client.49233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[99959]: from='client.69158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/92758274' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:24.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3179754471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:25 smithi146 ceph-mon[103278]: from='client.69170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:25 smithi146 ceph-mon[103278]: pgmap v2295: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:25 smithi078 ceph-mon[91136]: from='client.69170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:25 smithi078 ceph-mon[91136]: pgmap v2295: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:25 smithi078 ceph-mon[99959]: from='client.69170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:25 smithi078 ceph-mon[99959]: pgmap v2295: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:27 smithi146 ceph-mon[103278]: pgmap v2296: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:27 smithi078 ceph-mon[99959]: pgmap v2296: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:27 smithi078 ceph-mon[91136]: pgmap v2296: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:29 smithi146 ceph-mon[103278]: pgmap v2297: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:29 smithi078 ceph-mon[91136]: pgmap v2297: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:29 smithi078 ceph-mon[99959]: pgmap v2297: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:36:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:36:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:30.416Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:30.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:30.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:31 smithi146 ceph-mon[103278]: pgmap v2298: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:31 smithi078 ceph-mon[91136]: pgmap v2298: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:31 smithi078 ceph-mon[99959]: pgmap v2298: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:33.650 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:33 smithi146 ceph-mon[103278]: pgmap v2299: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:33 smithi078 ceph-mon[91136]: pgmap v2299: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:33 smithi078 ceph-mon[99959]: pgmap v2299: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:33.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:33.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:33.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:34.593 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:35.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:35 smithi146 ceph-mon[103278]: pgmap v2300: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:35.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[91136]: pgmap v2300: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[99959]: pgmap v2300: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:37.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:37 smithi078 ceph-mon[91136]: pgmap v2301: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:37.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:37 smithi078 ceph-mon[99959]: pgmap v2301: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:37 smithi146 ceph-mon[103278]: pgmap v2301: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:39 smithi146 ceph-mon[103278]: pgmap v2302: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:39 smithi078 ceph-mon[91136]: pgmap v2302: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:39 smithi078 ceph-mon[99959]: pgmap v2302: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:36:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:36:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:40.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:40.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:40.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[91136]: pgmap v2303: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:40 smithi078 ceph-mon[99959]: pgmap v2303: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:40 smithi146 ceph-mon[103278]: pgmap v2303: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:43 smithi146 ceph-mon[103278]: pgmap v2304: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-mon[99959]: pgmap v2304: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-mon[91136]: pgmap v2304: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:43.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:43.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:43.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:45 smithi146 ceph-mon[103278]: pgmap v2305: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:45 smithi078 ceph-mon[99959]: pgmap v2305: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:45 smithi078 ceph-mon[91136]: pgmap v2305: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:47 smithi146 ceph-mon[103278]: pgmap v2306: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:47 smithi078 ceph-mon[91136]: pgmap v2306: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:47 smithi078 ceph-mon[99959]: pgmap v2306: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:49 smithi146 ceph-mon[103278]: pgmap v2307: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:49 smithi078 ceph-mon[91136]: pgmap v2307: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:49 smithi078 ceph-mon[99959]: pgmap v2307: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:50.736 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:36:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:36:50.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:50.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:50.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:50.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:50.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:50.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:51 smithi146 ceph-mon[103278]: pgmap v2308: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:51 smithi078 ceph-mon[91136]: pgmap v2308: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:51 smithi078 ceph-mon[99959]: pgmap v2308: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:53 smithi146 ceph-mon[103278]: pgmap v2309: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:53 smithi078 ceph-mon[91136]: pgmap v2309: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:53 smithi078 ceph-mon[99959]: pgmap v2309: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:54.193 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:53.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:54.193 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:53.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:36:54.194 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:36:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:36:53.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:36:54.194 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:36:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:54.723 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:36:54.723 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 87m - - 2024-09-20T21:36:54.723 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 87m - - 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (81m) 19s ago 82m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (82m) 14s ago 82m 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (75m) 19s ago 80m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (73m) 14s ago 86m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (75m) 19s ago 88m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (74m) 19s ago 88m 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (72m) 14s ago 87m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (73m) 19s ago 87m 121M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (82m) 19s ago 82m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (82m) 14s ago 82m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (71m) 19s ago 86m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (70m) 19s ago 86m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (70m) 19s ago 85m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (69m) 19s ago 85m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (69m) 14s ago 84m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:36:54.724 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (68m) 14s ago 84m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (67m) 14s ago 84m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (66m) 14s ago 83m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (75m) 14s ago 83m 67.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (66m) 19s ago 80m 87.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (65m) 14s ago 80m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (65m) 19s ago 80m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:36:54.725 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (65m) 14s ago 80m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:36:54.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:55.036 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:36:55.037 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:36:55.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:36:55.037 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:36:55.037 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:36:55.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:36:55.037 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:36:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:36:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:36:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:36:55.307 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:36:55.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:55 smithi146 ceph-mon[103278]: from='client.49263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:55.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:55 smithi146 ceph-mon[103278]: pgmap v2310: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:55.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:55 smithi146 ceph-mon[103278]: from='client.69188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:55.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/334348066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:55.624 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:36:55.624 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:55.624 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:36:55.624 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:55.625 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[99959]: from='client.49263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[99959]: pgmap v2310: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[99959]: from='client.69188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/334348066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[91136]: from='client.49263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[91136]: pgmap v2310: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:55.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[91136]: from='client.69188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:55.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/334348066' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:36:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:56 smithi146 ceph-mon[103278]: from='client.69194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:56 smithi146 ceph-mon[103278]: from='client.69206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/642368998' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[91136]: from='client.69194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[91136]: from='client.69206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/642368998' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[99959]: from='client.69194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[99959]: from='client.69206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:36:56.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/642368998' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:36:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:57 smithi078 ceph-mon[91136]: pgmap v2311: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:57 smithi078 ceph-mon[99959]: pgmap v2311: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:57 smithi146 ceph-mon[103278]: pgmap v2311: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:36:58.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:58.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:36:58 smithi078 ceph-mon[91136]: pgmap v2312: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:58.650 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:58.650 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:36:58 smithi078 ceph-mon[99959]: pgmap v2312: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:36:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:36:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:36:58 smithi146 ceph-mon[103278]: pgmap v2312: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:37:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:37:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:00.419Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:00.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:00.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:00 smithi146 ceph-mon[103278]: pgmap v2313: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[99959]: pgmap v2313: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:01.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:00 smithi078 ceph-mon[91136]: pgmap v2313: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:03 smithi146 ceph-mon[103278]: pgmap v2314: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:03 smithi078 ceph-mon[91136]: pgmap v2314: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:03 smithi078 ceph-mon[99959]: pgmap v2314: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:03.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:03.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:03.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:05 smithi146 ceph-mon[103278]: pgmap v2315: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:05 smithi078 ceph-mon[91136]: pgmap v2315: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:05 smithi078 ceph-mon[99959]: pgmap v2315: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:07 smithi146 ceph-mon[103278]: pgmap v2316: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:07 smithi078 ceph-mon[91136]: pgmap v2316: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:07 smithi078 ceph-mon[99959]: pgmap v2316: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:09 smithi146 ceph-mon[103278]: pgmap v2317: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:09 smithi078 ceph-mon[91136]: pgmap v2317: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:09 smithi078 ceph-mon[99959]: pgmap v2317: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:37:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:37:10.687 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:10.419Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:10.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:10.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:11.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:11 smithi078 ceph-mon[91136]: pgmap v2318: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:11.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:11 smithi078 ceph-mon[99959]: pgmap v2318: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:11 smithi146 ceph-mon[103278]: pgmap v2318: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:13 smithi146 ceph-mon[103278]: pgmap v2319: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-mon[91136]: pgmap v2319: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-mon[99959]: pgmap v2319: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:13.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:13.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:13.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:14.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:15.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:15 smithi146 ceph-mon[103278]: pgmap v2320: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:15 smithi078 ceph-mon[99959]: pgmap v2320: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:15 smithi078 ceph-mon[91136]: pgmap v2320: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:16.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:16.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:16.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:16.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:16 smithi146 ceph-mon[103278]: pgmap v2321: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[91136]: pgmap v2321: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:17.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:16 smithi078 ceph-mon[99959]: pgmap v2321: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:19.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:19 smithi078 ceph-mon[91136]: pgmap v2322: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:19.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:19 smithi078 ceph-mon[99959]: pgmap v2322: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:19 smithi146 ceph-mon[103278]: pgmap v2322: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:37:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:37:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:20.420Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:20.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:20.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:21 smithi146 ceph-mon[103278]: pgmap v2323: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[91136]: pgmap v2323: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:21 smithi078 ceph-mon[99959]: pgmap v2323: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:23 smithi146 ceph-mon[103278]: pgmap v2324: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:23 smithi078 ceph-mon[91136]: pgmap v2324: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:23 smithi078 ceph-mon[99959]: pgmap v2324: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:23.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:23.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:23.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:25 smithi146 ceph-mon[103278]: pgmap v2325: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:25 smithi078 ceph-mon[91136]: pgmap v2325: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:25 smithi078 ceph-mon[99959]: pgmap v2325: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:25.922 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:37:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:37:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 88m - - 2024-09-20T21:37:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 87m - - 2024-09-20T21:37:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (82m) 10s ago 83m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:37:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (83m) 6s ago 83m 87.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:37:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (75m) 10s ago 80m 62.2M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (74m) 6s ago 87m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (76m) 10s ago 88m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (74m) 10s ago 89m 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (73m) 6s ago 87m 112M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (73m) 10s ago 87m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (82m) 10s ago 82m 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (82m) 6s ago 82m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (72m) 10s ago 86m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (71m) 10s ago 86m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (70m) 10s ago 86m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (70m) 10s ago 85m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (69m) 6s ago 85m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:37:26.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (68m) 6s ago 84m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (67m) 6s ago 84m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (66m) 6s ago 84m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (75m) 6s ago 83m 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (66m) 10s ago 81m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (66m) 6s ago 81m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (66m) 10s ago 81m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:37:26.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (66m) 6s ago 81m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:26.759 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:37:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:37:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:37:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:37:26.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:37:26.760 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:37:27.030 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:37:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:37:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:37:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:37:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:37:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:37:27.136 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:37:27.339 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:37:27.339 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:27.339 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:27.339 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:27.339 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:27 smithi146 ceph-mon[103278]: from='client.69218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:27 smithi146 ceph-mon[103278]: from='client.69224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:27 smithi146 ceph-mon[103278]: pgmap v2326: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:27 smithi146 ceph-mon[103278]: from='client.49311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/878906909' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3985194049' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[99959]: from='client.69218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[99959]: from='client.69224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[99959]: pgmap v2326: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[99959]: from='client.49311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/878906909' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3985194049' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[91136]: from='client.69218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[91136]: from='client.69224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[91136]: pgmap v2326: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[91136]: from='client.49311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/878906909' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3985194049' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:28 smithi146 ceph-mon[103278]: from='client.49323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:28 smithi078 ceph-mon[99959]: from='client.49323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:28 smithi078 ceph-mon[91136]: from='client.49323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:29 smithi146 ceph-mon[103278]: pgmap v2327: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:29 smithi078 ceph-mon[99959]: pgmap v2327: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:29 smithi078 ceph-mon[91136]: pgmap v2327: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:37:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:37:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:30.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:30.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:30.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:31 smithi146 ceph-mon[103278]: pgmap v2328: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:31 smithi078 ceph-mon[99959]: pgmap v2328: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:31 smithi078 ceph-mon[91136]: pgmap v2328: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:33 smithi146 ceph-mon[103278]: pgmap v2329: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:33 smithi078 ceph-mon[99959]: pgmap v2329: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:33 smithi078 ceph-mon[91136]: pgmap v2329: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:33.857Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:33.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:33.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:35 smithi146 ceph-mon[103278]: pgmap v2330: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:35 smithi078 ceph-mon[91136]: pgmap v2330: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:35 smithi078 ceph-mon[99959]: pgmap v2330: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:36 smithi146 ceph-mon[103278]: pgmap v2331: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[91136]: pgmap v2331: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:37.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:36 smithi078 ceph-mon[99959]: pgmap v2331: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:39.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:39 smithi078 ceph-mon[91136]: pgmap v2332: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:39.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:39 smithi078 ceph-mon[99959]: pgmap v2332: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:39 smithi146 ceph-mon[103278]: pgmap v2332: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:40.546 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:40.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:40.546 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:40.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:40.546 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:40.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:41.001 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:37:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[99959]: pgmap v2333: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:41 smithi078 ceph-mon[91136]: pgmap v2333: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:41 smithi146 ceph-mon[103278]: pgmap v2333: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:43 smithi146 ceph-mon[103278]: pgmap v2334: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-mon[91136]: pgmap v2334: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-mon[99959]: pgmap v2334: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:43.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:43.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:43.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:45 smithi146 ceph-mon[103278]: pgmap v2335: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:45 smithi078 ceph-mon[91136]: pgmap v2335: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:45 smithi078 ceph-mon[99959]: pgmap v2335: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:47 smithi146 ceph-mon[103278]: pgmap v2336: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:47 smithi078 ceph-mon[99959]: pgmap v2336: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:47 smithi078 ceph-mon[91136]: pgmap v2336: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:49 smithi146 ceph-mon[103278]: pgmap v2337: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:49 smithi078 ceph-mon[91136]: pgmap v2337: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:49 smithi078 ceph-mon[99959]: pgmap v2337: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:37:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:37:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:50.422Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:50.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:50.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:51 smithi146 ceph-mon[103278]: pgmap v2338: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:51 smithi078 ceph-mon[91136]: pgmap v2338: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:51 smithi078 ceph-mon[99959]: pgmap v2338: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:53 smithi146 ceph-mon[103278]: pgmap v2339: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:53 smithi078 ceph-mon[91136]: pgmap v2339: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:53 smithi078 ceph-mon[99959]: pgmap v2339: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:53.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:53.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:37:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:37:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:37:53.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:37:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:37:55.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:55 smithi146 ceph-mon[103278]: pgmap v2340: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:55 smithi078 ceph-mon[91136]: pgmap v2340: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:55 smithi078 ceph-mon[99959]: pgmap v2340: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:56 smithi146 ceph-mon[103278]: pgmap v2341: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[91136]: pgmap v2341: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:37:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:56 smithi078 ceph-mon[99959]: pgmap v2341: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:37:57.633 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:37:58.159 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:37:58.159 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 88m - - 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 88m - - 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (82m) 2s ago 83m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (83m) 17s ago 83m 87.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (76m) 2s ago 81m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (74m) 17s ago 87m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (76m) 2s ago 89m 557M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (75m) 2s ago 89m 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (73m) 17s ago 88m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (74m) 2s ago 88m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (83m) 2s ago 83m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (83m) 17s ago 83m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (72m) 2s ago 87m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:37:58.160 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (71m) 2s ago 87m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (71m) 2s ago 86m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (70m) 2s ago 86m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (70m) 17s ago 85m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (69m) 17s ago 85m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (68m) 17s ago 85m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (67m) 17s ago 84m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (76m) 17s ago 84m 70.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (67m) 2s ago 81m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (66m) 17s ago 81m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (67m) 2s ago 81m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:37:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (66m) 17s ago 81m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:37:58.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:58.481 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:37:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:37:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:37:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:37:58.483 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:37:58.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:58.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:37:58.752 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:37:58.753 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:37:58.753 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:37:58.753 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:37:59.061 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:37:59.061 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:37:59.061 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:37:59.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[91136]: from='client.49335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[91136]: from='client.69260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[91136]: from='client.69266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[91136]: pgmap v2342: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:59.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1946942810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:59.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3584860761' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:59.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[99959]: from='client.49335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[99959]: from='client.69260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[99959]: from='client.69266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[99959]: pgmap v2342: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:59.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1946942810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:59.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:37:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3584860761' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:37:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:59 smithi146 ceph-mon[103278]: from='client.49335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:59 smithi146 ceph-mon[103278]: from='client.69260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:59 smithi146 ceph-mon[103278]: from='client.69266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:37:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:59 smithi146 ceph-mon[103278]: pgmap v2342: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:37:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1946942810' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:37:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:37:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3584860761' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:00.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:00 smithi078 ceph-mon[91136]: from='client.69278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:00.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:00 smithi078 ceph-mon[99959]: from='client.69278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:00 smithi146 ceph-mon[103278]: from='client.69278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:00.685 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:38:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:38:00.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:00.423Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:00.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:00.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:00.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:00.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:01.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[99959]: pgmap v2343: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:01.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:01 smithi078 ceph-mon[91136]: pgmap v2343: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:01 smithi146 ceph-mon[103278]: pgmap v2343: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:03 smithi146 ceph-mon[103278]: pgmap v2344: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:03 smithi078 ceph-mon[91136]: pgmap v2344: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:03 smithi078 ceph-mon[99959]: pgmap v2344: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:04.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:03.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:04.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:03.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:04.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:03.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:05.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:05 smithi078 ceph-mon[91136]: pgmap v2345: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:05.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:05 smithi078 ceph-mon[99959]: pgmap v2345: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:05 smithi146 ceph-mon[103278]: pgmap v2345: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:07 smithi146 ceph-mon[103278]: pgmap v2346: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:07 smithi078 ceph-mon[91136]: pgmap v2346: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:07 smithi078 ceph-mon[99959]: pgmap v2346: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:09 smithi146 ceph-mon[103278]: pgmap v2347: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:09 smithi078 ceph-mon[91136]: pgmap v2347: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:09 smithi078 ceph-mon[99959]: pgmap v2347: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:38:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:38:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:10.423Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:10.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:10.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:11.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:11 smithi078 ceph-mon[99959]: pgmap v2348: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:11.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:11 smithi078 ceph-mon[91136]: pgmap v2348: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:11 smithi146 ceph-mon[103278]: pgmap v2348: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:13 smithi146 ceph-mon[103278]: pgmap v2349: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-mon[91136]: pgmap v2349: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-mon[99959]: pgmap v2349: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:13.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:13.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:13.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:14.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:15 smithi146 ceph-mon[103278]: pgmap v2350: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:15 smithi078 ceph-mon[91136]: pgmap v2350: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:15 smithi078 ceph-mon[99959]: pgmap v2350: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:17.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:17 smithi146 ceph-mon[103278]: pgmap v2351: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[91136]: pgmap v2351: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:17.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:17 smithi078 ceph-mon[99959]: pgmap v2351: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:19 smithi146 ceph-mon[103278]: pgmap v2352: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:19 smithi078 ceph-mon[91136]: pgmap v2352: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:19 smithi078 ceph-mon[99959]: pgmap v2352: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:20.735 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:38:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:38:20.736 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:20.424Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:20.736 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:20.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:20.736 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:20.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:21.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[99959]: pgmap v2353: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:21.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[91136]: pgmap v2353: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:21.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:21 smithi146 ceph-mon[103278]: pgmap v2353: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:23 smithi146 ceph-mon[103278]: pgmap v2354: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:23 smithi078 ceph-mon[91136]: pgmap v2354: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:23 smithi078 ceph-mon[99959]: pgmap v2354: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:23.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:23.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:23.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:25 smithi146 ceph-mon[103278]: pgmap v2355: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:25 smithi078 ceph-mon[91136]: pgmap v2355: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:25 smithi078 ceph-mon[99959]: pgmap v2355: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:27 smithi146 ceph-mon[103278]: pgmap v2356: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:27 smithi078 ceph-mon[91136]: pgmap v2356: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:27 smithi078 ceph-mon[99959]: pgmap v2356: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:29.356 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:38:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:29 smithi146 ceph-mon[103278]: pgmap v2357: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:29 smithi078 ceph-mon[99959]: pgmap v2357: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:29 smithi078 ceph-mon[91136]: pgmap v2357: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 89m - - 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 88m - - 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (83m) 13s ago 84m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (84m) 9s ago 84m 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (76m) 13s ago 82m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:38:29.884 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (75m) 9s ago 88m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (77m) 13s ago 89m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (75m) 13s ago 90m 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (74m) 9s ago 88m 114M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (74m) 13s ago 88m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (83m) 13s ago 83m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (83m) 9s ago 83m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (73m) 13s ago 87m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (72m) 13s ago 87m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:38:29.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (71m) 13s ago 87m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (71m) 13s ago 86m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (70m) 9s ago 86m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (69m) 9s ago 86m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (68m) 9s ago 85m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (67m) 9s ago 85m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (76m) 9s ago 84m 68.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (67m) 13s ago 82m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (67m) 9s ago 82m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (67m) 13s ago 82m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:38:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (67m) 9s ago 82m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:38:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:38:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:38:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:38:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:38:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:38:30.200 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:38:30.469 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:38:30.470 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:38:30.471 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:38:30.471 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:38:30.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:30.425Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:30.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:30.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:30.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:30.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:30 smithi146 ceph-mon[103278]: from='client.69290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/668512608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:38:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:38:30.782 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:38:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:38:30.782 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:38:30.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-mon[99959]: from='client.69290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:30.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/668512608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:30.783 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:38:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:38:30.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-mon[91136]: from='client.69290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:30.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/668512608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:38:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:31 smithi146 ceph-mon[103278]: from='client.49377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:31 smithi146 ceph-mon[103278]: from='client.69302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:31 smithi146 ceph-mon[103278]: pgmap v2358: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:31 smithi146 ceph-mon[103278]: from='client.49395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/471360506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[91136]: from='client.49377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[91136]: from='client.69302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[91136]: pgmap v2358: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[91136]: from='client.49395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/471360506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:31.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[99959]: from='client.49377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[99959]: from='client.69302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[99959]: pgmap v2358: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:31.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[99959]: from='client.49395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:38:31.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/471360506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:38:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:33 smithi146 ceph-mon[103278]: pgmap v2359: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:33 smithi078 ceph-mon[91136]: pgmap v2359: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:33 smithi078 ceph-mon[99959]: pgmap v2359: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:33.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:33.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:33.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:35 smithi146 ceph-mon[103278]: pgmap v2360: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:35 smithi078 ceph-mon[91136]: pgmap v2360: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:35 smithi078 ceph-mon[99959]: pgmap v2360: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:37.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:37 smithi146 ceph-mon[103278]: pgmap v2361: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[91136]: pgmap v2361: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:37.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:37 smithi078 ceph-mon[99959]: pgmap v2361: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:39 smithi146 ceph-mon[103278]: pgmap v2362: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:39 smithi078 ceph-mon[91136]: pgmap v2362: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:39 smithi078 ceph-mon[99959]: pgmap v2362: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:38:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:38:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:40.425Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:40.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:40.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:41 smithi146 ceph-mon[103278]: pgmap v2363: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[91136]: pgmap v2363: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[99959]: pgmap v2363: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:43.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-mon[91136]: pgmap v2364: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:43.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:43.632 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-mon[99959]: pgmap v2364: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:43.632 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:43 smithi146 ceph-mon[103278]: pgmap v2364: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:43.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:43.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:43.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:45 smithi146 ceph-mon[103278]: pgmap v2365: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:45 smithi078 ceph-mon[91136]: pgmap v2365: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:45 smithi078 ceph-mon[99959]: pgmap v2365: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:47 smithi146 ceph-mon[103278]: pgmap v2366: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:47 smithi078 ceph-mon[91136]: pgmap v2366: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:47 smithi078 ceph-mon[99959]: pgmap v2366: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:49 smithi146 ceph-mon[103278]: pgmap v2367: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:49 smithi078 ceph-mon[91136]: pgmap v2367: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:49 smithi078 ceph-mon[99959]: pgmap v2367: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:38:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:38:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:50.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:50.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:50.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:51 smithi146 ceph-mon[103278]: pgmap v2368: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:51 smithi078 ceph-mon[91136]: pgmap v2368: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:51 smithi078 ceph-mon[99959]: pgmap v2368: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:53 smithi146 ceph-mon[103278]: pgmap v2369: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:53 smithi078 ceph-mon[91136]: pgmap v2369: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:53 smithi078 ceph-mon[99959]: pgmap v2369: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:38:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:53.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:53.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:38:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:38:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:38:53.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:38:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:38:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:55 smithi146 ceph-mon[103278]: pgmap v2370: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:55 smithi078 ceph-mon[91136]: pgmap v2370: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:55 smithi078 ceph-mon[99959]: pgmap v2370: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:57.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:57 smithi146 ceph-mon[103278]: pgmap v2371: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[91136]: pgmap v2371: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:38:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:57 smithi078 ceph-mon[99959]: pgmap v2371: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:38:58.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:38:59.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:38:59 smithi146 ceph-mon[103278]: pgmap v2372: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:38:59 smithi078 ceph-mon[91136]: pgmap v2372: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:38:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:38:59 smithi078 ceph-mon[99959]: pgmap v2372: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:39:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:39:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:00.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:00.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:00.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:01.077 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:39:01.609 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 89m - - 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 89m - - 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (83m) 5s ago 84m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (84m) 0s ago 84m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (77m) 5s ago 82m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (75m) 0s ago 88m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:39:01.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (77m) 5s ago 90m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (76m) 5s ago 90m 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (74m) 0s ago 89m 115M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (75m) 5s ago 89m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (84m) 5s ago 84m 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (84m) 0s ago 84m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (73m) 5s ago 88m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (72m) 5s ago 88m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (72m) 5s ago 87m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:39:01.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (71m) 5s ago 87m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (71m) 0s ago 86m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (70m) 0s ago 86m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (69m) 0s ago 86m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (68m) 0s ago 85m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (77m) 0s ago 85m 68.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (68m) 5s ago 82m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (68m) 0s ago 83m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (68m) 5s ago 82m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:39:01.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (67m) 0s ago 82m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:39:01.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[99959]: pgmap v2373: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:01.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[91136]: pgmap v2373: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:01.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:01 smithi146 ceph-mon[103278]: pgmap v2373: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:39:01.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:39:01.933 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:39:02.211 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:39:02.212 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:39:02.213 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:39:02.213 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:39:02.213 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:39:02.213 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:39:02.213 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:39:02.532 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:39:02.532 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:02.532 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:02 smithi146 ceph-mon[103278]: from='client.49407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:02 smithi146 ceph-mon[103278]: from='client.49413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2124945791' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:02.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:02 smithi078 ceph-mon[99959]: from='client.49407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:02.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:02 smithi078 ceph-mon[99959]: from='client.49413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:02.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2124945791' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:02.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:02 smithi078 ceph-mon[91136]: from='client.49407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:02.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:02 smithi078 ceph-mon[91136]: from='client.49413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:02.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2124945791' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[99959]: from='client.69338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[99959]: from='client.49431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[99959]: pgmap v2374: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2765939983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[91136]: from='client.69338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[91136]: from='client.49431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[91136]: pgmap v2374: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:03.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2765939983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:03 smithi146 ceph-mon[103278]: from='client.69338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:03 smithi146 ceph-mon[103278]: from='client.49431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:03 smithi146 ceph-mon[103278]: pgmap v2374: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2765939983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:03.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:03.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:03.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:03.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:03.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:03.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:05 smithi146 ceph-mon[103278]: pgmap v2375: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:05 smithi078 ceph-mon[91136]: pgmap v2375: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:05 smithi078 ceph-mon[99959]: pgmap v2375: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:07 smithi146 ceph-mon[103278]: pgmap v2376: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:07 smithi078 ceph-mon[91136]: pgmap v2376: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:07 smithi078 ceph-mon[99959]: pgmap v2376: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:09 smithi146 ceph-mon[103278]: pgmap v2377: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:09 smithi078 ceph-mon[91136]: pgmap v2377: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:09 smithi078 ceph-mon[99959]: pgmap v2377: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:10.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:39:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:39:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:10.427Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:10.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:10.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:10.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:11 smithi146 ceph-mon[103278]: pgmap v2378: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:11 smithi078 ceph-mon[99959]: pgmap v2378: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:11 smithi078 ceph-mon[91136]: pgmap v2378: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:13 smithi146 ceph-mon[103278]: pgmap v2379: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-mon[91136]: pgmap v2379: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-mon[99959]: pgmap v2379: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:13.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:13.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:13.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:15 smithi146 ceph-mon[103278]: pgmap v2380: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:15 smithi078 ceph-mon[91136]: pgmap v2380: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:15 smithi078 ceph-mon[99959]: pgmap v2380: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:17.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:17 smithi146 ceph-mon[103278]: pgmap v2381: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:17.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[91136]: pgmap v2381: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[99959]: pgmap v2381: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:17.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:19.716 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:19 smithi146 ceph-mon[103278]: pgmap v2382: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:19 smithi078 ceph-mon[99959]: pgmap v2382: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:19 smithi078 ceph-mon[91136]: pgmap v2382: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:39:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:39:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:20.427Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:20.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:20.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:21 smithi146 ceph-mon[103278]: pgmap v2383: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[91136]: pgmap v2383: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[99959]: pgmap v2383: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:23.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:23 smithi078 ceph-mon[99959]: pgmap v2384: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:23.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:23 smithi078 ceph-mon[91136]: pgmap v2384: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:23 smithi146 ceph-mon[103278]: pgmap v2384: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:23.983 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:23.864Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:23.983 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:23.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:23.984 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:23.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:24.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:24.637 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:25 smithi146 ceph-mon[103278]: pgmap v2385: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:25 smithi078 ceph-mon[91136]: pgmap v2385: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:25 smithi078 ceph-mon[99959]: pgmap v2385: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:27 smithi146 ceph-mon[103278]: pgmap v2386: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:27 smithi078 ceph-mon[91136]: pgmap v2386: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:27 smithi078 ceph-mon[99959]: pgmap v2386: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:29 smithi146 ceph-mon[103278]: pgmap v2387: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:29 smithi078 ceph-mon[91136]: pgmap v2387: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:29 smithi078 ceph-mon[99959]: pgmap v2387: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:39:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:39:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:30.429Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:30.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:30.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:31 smithi146 ceph-mon[103278]: pgmap v2388: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:31 smithi078 ceph-mon[91136]: pgmap v2388: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:31 smithi078 ceph-mon[99959]: pgmap v2388: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:32.828 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:39:33.358 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 90m - - 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 89m - - 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (84m) 16s ago 85m 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (85m) 12s ago 85m 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (77m) 16s ago 83m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (76m) 12s ago 89m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (78m) 16s ago 90m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (76m) 16s ago 91m 183M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (75m) 12s ago 89m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:39:33.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (75m) 16s ago 89m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (84m) 16s ago 84m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (84m) 12s ago 84m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (74m) 16s ago 89m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (73m) 16s ago 88m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (72m) 16s ago 88m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (72m) 16s ago 87m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (71m) 12s ago 87m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (70m) 12s ago 87m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (69m) 12s ago 86m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (68m) 12s ago 86m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (77m) 12s ago 85m 69.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (68m) 16s ago 83m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (68m) 12s ago 83m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (68m) 16s ago 83m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:39:33.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (68m) 12s ago 83m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:39:33.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:39:33.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:39:33.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:39:33.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:39:33.671 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:39:33.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:33 smithi078 ceph-mon[99959]: pgmap v2389: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:33.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:33 smithi078 ceph-mon[91136]: pgmap v2389: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:33 smithi146 ceph-mon[103278]: pgmap v2389: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:39:33.938 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:39:33.939 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:39:33.939 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:33.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:33.939 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:33.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:33.939 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:33.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:34.244 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:39:34.244 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:34.244 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:39:34.244 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:34.244 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:39:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[91136]: from='client.69362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[91136]: from='client.49449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[91136]: from='client.69374 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2653372888' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:34.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/481314781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:34.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:34.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[99959]: from='client.69362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[99959]: from='client.49449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[99959]: from='client.69374 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2653372888' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:34.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/481314781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:34 smithi146 ceph-mon[103278]: from='client.69362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:34 smithi146 ceph-mon[103278]: from='client.49449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:34 smithi146 ceph-mon[103278]: from='client.69374 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2653372888' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:39:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/481314781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:39:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:35 smithi146 ceph-mon[103278]: from='client.69386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:35 smithi146 ceph-mon[103278]: pgmap v2390: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:35 smithi078 ceph-mon[91136]: from='client.69386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:35 smithi078 ceph-mon[91136]: pgmap v2390: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:35 smithi078 ceph-mon[99959]: from='client.69386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:39:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:35 smithi078 ceph-mon[99959]: pgmap v2390: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:37 smithi146 ceph-mon[103278]: pgmap v2391: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[99959]: pgmap v2391: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[91136]: pgmap v2391: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:39.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:39 smithi146 ceph-mon[103278]: pgmap v2392: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:39 smithi078 ceph-mon[99959]: pgmap v2392: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:39 smithi078 ceph-mon[91136]: pgmap v2392: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:40.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:39:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:39:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:40.429Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:40.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:40.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:40.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:41 smithi146 ceph-mon[103278]: pgmap v2393: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[99959]: pgmap v2393: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[91136]: pgmap v2393: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:43 smithi146 ceph-mon[103278]: pgmap v2394: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-mon[91136]: pgmap v2394: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-mon[99959]: pgmap v2394: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:44.229 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:43.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:44.229 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:43.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:44.229 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:43.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:45.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:45 smithi078 ceph-mon[99959]: pgmap v2395: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:45.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:45 smithi078 ceph-mon[91136]: pgmap v2395: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:45 smithi146 ceph-mon[103278]: pgmap v2395: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:47 smithi078 ceph-mon[91136]: pgmap v2396: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:47 smithi078 ceph-mon[99959]: pgmap v2396: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:47 smithi146 ceph-mon[103278]: pgmap v2396: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:49 smithi078 ceph-mon[91136]: pgmap v2397: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:49 smithi078 ceph-mon[99959]: pgmap v2397: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:49 smithi146 ceph-mon[103278]: pgmap v2397: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:39:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:39:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:50.430Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:50.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:50.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:51 smithi078 ceph-mon[91136]: pgmap v2398: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:51 smithi078 ceph-mon[99959]: pgmap v2398: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:51 smithi146 ceph-mon[103278]: pgmap v2398: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:53 smithi078 ceph-mon[91136]: pgmap v2399: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:53 smithi078 ceph-mon[99959]: pgmap v2399: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:53 smithi146 ceph-mon[103278]: pgmap v2399: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:53.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:53.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:39:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:39:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:39:53.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:39:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:39:55.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:55 smithi146 ceph-mon[103278]: pgmap v2400: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:55 smithi078 ceph-mon[91136]: pgmap v2400: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:55 smithi078 ceph-mon[99959]: pgmap v2400: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:39:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:57 smithi146 ceph-mon[103278]: pgmap v2401: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[91136]: pgmap v2401: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[99959]: pgmap v2401: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:39:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:39:59.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:39:59 smithi146 ceph-mon[103278]: pgmap v2402: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:00.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:39:59 smithi078 ceph-mon[91136]: pgmap v2402: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:00.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:39:59 smithi078 ceph-mon[99959]: pgmap v2402: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T21:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:40:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:40:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:00.431Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:00.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:00.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:01.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:01 smithi146 ceph-mon[103278]: pgmap v2403: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:01.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:01.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:01.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[91136]: pgmap v2403: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[99959]: pgmap v2403: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:03.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:03 smithi078 ceph-mon[91136]: pgmap v2404: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:03.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:03 smithi078 ceph-mon[99959]: pgmap v2404: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:03 smithi146 ceph-mon[103278]: pgmap v2404: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:04.140 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:03.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:04.140 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:03.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:04.140 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:03.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:04.533 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:40:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:40:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 90m - - 2024-09-20T21:40:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 90m - - 2024-09-20T21:40:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (84m) 8s ago 85m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:40:05.071 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (85m) 3s ago 85m 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (78m) 8s ago 83m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (77m) 3s ago 89m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (78m) 8s ago 91m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (77m) 8s ago 91m 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (75m) 3s ago 90m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (76m) 8s ago 90m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (85m) 8s ago 85m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (85m) 3s ago 85m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (74m) 8s ago 89m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (73m) 8s ago 89m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (73m) 8s ago 88m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (72m) 8s ago 88m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (72m) 3s ago 88m 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:40:05.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (71m) 3s ago 87m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (70m) 3s ago 87m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (69m) 3s ago 86m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (78m) 3s ago 86m 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (69m) 8s ago 84m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (69m) 3s ago 84m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (69m) 8s ago 83m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:40:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (68m) 3s ago 83m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:40:05.391 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:40:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:40:05.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:40:05.392 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:40:05.673 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:40:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:40:05.673 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:40:05.674 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:40:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:05 smithi078 ceph-mon[99959]: pgmap v2405: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:05 smithi078 ceph-mon[99959]: from='client.49479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4024830046' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:05 smithi078 ceph-mon[91136]: pgmap v2405: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:05 smithi078 ceph-mon[91136]: from='client.49479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4024830046' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:06.000 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:06.001 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:40:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:40:06.001 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:05 smithi146 ceph-mon[103278]: pgmap v2405: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:05 smithi146 ceph-mon[103278]: from='client.49479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4024830046' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:06.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:06 smithi078 ceph-mon[99959]: from='client.69404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:06.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:06 smithi078 ceph-mon[99959]: from='client.49491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:06.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2889899902' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:06.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:06 smithi078 ceph-mon[91136]: from='client.69404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:06.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:06 smithi078 ceph-mon[91136]: from='client.49491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:06.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2889899902' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:06 smithi146 ceph-mon[103278]: from='client.69404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:06 smithi146 ceph-mon[103278]: from='client.49491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2889899902' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:07.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:07 smithi078 ceph-mon[91136]: from='client.69422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:07.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:07 smithi078 ceph-mon[91136]: pgmap v2406: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:07.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:07 smithi078 ceph-mon[99959]: from='client.69422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:07.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:07 smithi078 ceph-mon[99959]: pgmap v2406: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:07 smithi146 ceph-mon[103278]: from='client.69422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:07 smithi146 ceph-mon[103278]: pgmap v2406: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:10.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:09 smithi146 ceph-mon[103278]: pgmap v2407: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:09 smithi078 ceph-mon[99959]: pgmap v2407: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:09 smithi078 ceph-mon[91136]: pgmap v2407: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:40:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:40:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:10.432Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:10.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:10.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:11 smithi146 ceph-mon[103278]: pgmap v2408: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:11 smithi078 ceph-mon[91136]: pgmap v2408: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:11 smithi078 ceph-mon[99959]: pgmap v2408: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:13.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-mon[91136]: pgmap v2409: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:13.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:13.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-mon[99959]: pgmap v2409: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:13.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:13.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:13 smithi146 ceph-mon[103278]: pgmap v2409: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:40:13.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:13.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:13.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:13.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:14 smithi146 ceph-mon[103278]: pgmap v2410: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:14 smithi078 ceph-mon[99959]: pgmap v2410: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:14 smithi078 ceph-mon[91136]: pgmap v2410: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:17 smithi146 ceph-mon[103278]: pgmap v2411: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[99959]: pgmap v2411: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[91136]: pgmap v2411: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:19 smithi146 ceph-mon[103278]: pgmap v2412: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:19 smithi078 ceph-mon[99959]: pgmap v2412: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:19 smithi078 ceph-mon[91136]: pgmap v2412: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:40:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:40:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:20.433Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:20.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:20.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:21 smithi146 ceph-mon[103278]: pgmap v2413: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[99959]: pgmap v2413: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[91136]: pgmap v2413: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:23 smithi146 ceph-mon[103278]: pgmap v2414: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:23 smithi078 ceph-mon[99959]: pgmap v2414: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:23 smithi078 ceph-mon[91136]: pgmap v2414: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:23.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:23.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:23.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:24.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:24.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:24.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:25 smithi146 ceph-mon[103278]: pgmap v2415: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:25 smithi078 ceph-mon[91136]: pgmap v2415: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:25 smithi078 ceph-mon[99959]: pgmap v2415: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:27 smithi146 ceph-mon[103278]: pgmap v2416: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:27 smithi078 ceph-mon[91136]: pgmap v2416: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:27 smithi078 ceph-mon[99959]: pgmap v2416: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:29 smithi146 ceph-mon[103278]: pgmap v2417: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:29 smithi078 ceph-mon[99959]: pgmap v2417: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:29 smithi078 ceph-mon[91136]: pgmap v2417: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:40:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:40:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:30.433Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:30.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:30.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:31 smithi078 ceph-mon[99959]: pgmap v2418: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:31 smithi078 ceph-mon[91136]: pgmap v2418: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:31 smithi146 ceph-mon[103278]: pgmap v2418: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:33 smithi078 ceph-mon[91136]: pgmap v2419: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:33 smithi078 ceph-mon[99959]: pgmap v2419: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:33 smithi146 ceph-mon[103278]: pgmap v2419: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:33.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:33.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:33.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:35 smithi078 ceph-mon[99959]: pgmap v2420: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:35 smithi078 ceph-mon[91136]: pgmap v2420: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:35 smithi146 ceph-mon[103278]: pgmap v2420: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:36.295 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 91m - - 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 91m - - 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (85m) 19s ago 86m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (86m) 15s ago 86m 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (78m) 19s ago 84m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (77m) 15s ago 90m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (79m) 19s ago 92m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:40:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (77m) 19s ago 92m 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (76m) 15s ago 90m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (76m) 19s ago 90m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (85m) 19s ago 85m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (85m) 15s ago 85m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (75m) 19s ago 90m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (74m) 19s ago 89m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (73m) 19s ago 89m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (73m) 19s ago 88m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (72m) 15s ago 88m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (71m) 15s ago 88m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (70m) 15s ago 87m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (70m) 15s ago 87m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (78m) 15s ago 86m 69.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (69m) 19s ago 84m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:40:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (69m) 15s ago 84m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:40:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (69m) 19s ago 84m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:40:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (69m) 15s ago 84m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:40:37.183 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:40:37.184 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:40:37.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:40:37.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:40:37.185 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:40:37.458 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:40:37.458 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:40:37.458 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:40:37.459 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:40:37.770 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:40:37.770 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:40:37.770 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:40:37.770 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:37.770 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: from='client.49515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: pgmap v2421: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: from='client.69440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1717580001' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: from='client.49515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: pgmap v2421: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:37.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: from='client.69440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:37.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:37.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:37.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:37.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1717580001' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: from='client.49515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: pgmap v2421: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: from='client.69440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1717580001' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:40:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:38 smithi146 ceph-mon[103278]: from='client.49527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:38 smithi146 ceph-mon[103278]: from='client.49539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/278821030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:38 smithi078 ceph-mon[91136]: from='client.49527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:38 smithi078 ceph-mon[91136]: from='client.49539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/278821030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:38 smithi078 ceph-mon[99959]: from='client.49527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:38 smithi078 ceph-mon[99959]: from='client.49539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:40:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/278821030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:40:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:39 smithi146 ceph-mon[103278]: pgmap v2422: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:39 smithi078 ceph-mon[91136]: pgmap v2422: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:39 smithi078 ceph-mon[99959]: pgmap v2422: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:40:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:40:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:40.434Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:40.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:40.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[91136]: pgmap v2423: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[99959]: pgmap v2423: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:41.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:41 smithi146 ceph-mon[103278]: pgmap v2423: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:43.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-mon[91136]: pgmap v2424: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:43.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:43.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-mon[99959]: pgmap v2424: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:43.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:43 smithi146 ceph-mon[103278]: pgmap v2424: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:43.870Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:43.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:43.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:45 smithi078 ceph-mon[91136]: pgmap v2425: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:45 smithi078 ceph-mon[99959]: pgmap v2425: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:45 smithi146 ceph-mon[103278]: pgmap v2425: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:47.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:47 smithi078 ceph-mon[91136]: pgmap v2426: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:47.768 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:47 smithi078 ceph-mon[99959]: pgmap v2426: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:47 smithi146 ceph-mon[103278]: pgmap v2426: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:49 smithi146 ceph-mon[103278]: pgmap v2427: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:49 smithi078 ceph-mon[99959]: pgmap v2427: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:49 smithi078 ceph-mon[91136]: pgmap v2427: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:40:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:40:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:40:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:50.435Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:50.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:50.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:51 smithi146 ceph-mon[103278]: pgmap v2428: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:51 smithi078 ceph-mon[99959]: pgmap v2428: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:51 smithi078 ceph-mon[91136]: pgmap v2428: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:53.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:53 smithi078 ceph-mon[99959]: pgmap v2429: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:53.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:53 smithi078 ceph-mon[91136]: pgmap v2429: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:53 smithi146 ceph-mon[103278]: pgmap v2429: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:40:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:53.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:53.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:40:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:40:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:40:53.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:40:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:40:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:55 smithi146 ceph-mon[103278]: pgmap v2430: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:40:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:55 smithi078 ceph-mon[99959]: pgmap v2430: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:40:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:55 smithi078 ceph-mon[91136]: pgmap v2430: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:40:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:57 smithi146 ceph-mon[103278]: pgmap v2431: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:40:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[99959]: pgmap v2431: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[91136]: pgmap v2431: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:40:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:40:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:40:59 smithi146 ceph-mon[103278]: pgmap v2432: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:41:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:40:59 smithi078 ceph-mon[91136]: pgmap v2432: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:41:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:40:59 smithi078 ceph-mon[99959]: pgmap v2432: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:41:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:41:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:41:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:00.436Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:00.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:00.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:01 smithi146 ceph-mon[103278]: pgmap v2433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T21:41:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[91136]: pgmap v2433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[99959]: pgmap v2433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:02 smithi146 ceph-mon[103278]: pgmap v2434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:41:02.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:02 smithi078 ceph-mon[91136]: pgmap v2434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:41:02.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:02 smithi078 ceph-mon[99959]: pgmap v2434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:41:03.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:03.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:03.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:03.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:03.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:04 smithi078 ceph-mon[91136]: pgmap v2435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:41:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:04 smithi078 ceph-mon[99959]: pgmap v2435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:41:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:04 smithi146 ceph-mon[103278]: pgmap v2435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:41:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:07 smithi078 ceph-mon[91136]: pgmap v2436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T21:41:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:07 smithi078 ceph-mon[99959]: pgmap v2436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T21:41:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:07 smithi146 ceph-mon[103278]: pgmap v2436: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T21:41:08.061 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 91m - - 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 91m - - 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (85m) 11s ago 86m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (86m) 7s ago 86m 88.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (79m) 11s ago 84m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (78m) 7s ago 90m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:41:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (80m) 11s ago 92m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (78m) 11s ago 92m 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (76m) 7s ago 91m 116M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (77m) 11s ago 91m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (86m) 11s ago 86m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (86m) 7s ago 86m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (75m) 11s ago 90m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (75m) 11s ago 90m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (74m) 11s ago 89m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (73m) 11s ago 89m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (73m) 7s ago 89m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (72m) 7s ago 88m 62.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (71m) 7s ago 88m 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (70m) 7s ago 87m 62.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:41:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (79m) 7s ago 87m 69.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:41:08.610 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (70m) 11s ago 85m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:41:08.610 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (70m) 7s ago 85m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:41:08.610 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (70m) 11s ago 85m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:41:08.610 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (70m) 7s ago 84m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:41:08.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:41:08.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:08.919 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:41:08.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:41:08.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:41:08.919 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:41:09.187 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:41:09.188 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:41:09.188 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:41:09.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:41:09.496 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:41:09.496 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:09.496 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:09 smithi146 ceph-mon[103278]: from='client.49551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:09 smithi146 ceph-mon[103278]: from='client.49557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:09 smithi146 ceph-mon[103278]: pgmap v2437: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:41:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1125919291' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/203508361' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[91136]: from='client.49551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[91136]: from='client.49557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[91136]: pgmap v2437: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:41:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1125919291' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/203508361' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[99959]: from='client.49551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[99959]: from='client.49557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[99959]: pgmap v2437: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:41:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1125919291' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/203508361' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:10.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-mon[99959]: from='client.69482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:10.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-mon[99959]: from='client.49575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:10.701 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:41:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:41:10.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-mon[91136]: from='client.69482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:10.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-mon[91136]: from='client.49575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:10.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:10.437Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:10.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:10.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:10.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:10.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:10 smithi146 ceph-mon[103278]: from='client.69482 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:10 smithi146 ceph-mon[103278]: from='client.49575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:11 smithi146 ceph-mon[103278]: pgmap v2438: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:41:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:11 smithi078 ceph-mon[91136]: pgmap v2438: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:41:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:11 smithi078 ceph-mon[99959]: pgmap v2438: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T21:41:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:13 smithi146 ceph-mon[103278]: pgmap v2439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:41:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-mon[99959]: pgmap v2439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:41:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-mon[91136]: pgmap v2439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-20T21:41:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:13.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:13.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:13.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:14.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:15 smithi078 ceph-mon[91136]: pgmap v2440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T21:41:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:15 smithi078 ceph-mon[99959]: pgmap v2440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T21:41:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:15 smithi146 ceph-mon[103278]: pgmap v2440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[91136]: pgmap v2441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[99959]: pgmap v2441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:17 smithi146 ceph-mon[103278]: pgmap v2441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:41:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:19 smithi078 ceph-mon[99959]: pgmap v2442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:41:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:19 smithi078 ceph-mon[91136]: pgmap v2442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:41:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:19 smithi146 ceph-mon[103278]: pgmap v2442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:41:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:41:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:41:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:20.437Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:20.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:20.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:21 smithi078 ceph-mon[99959]: pgmap v2443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:41:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:21 smithi078 ceph-mon[91136]: pgmap v2443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:41:21.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:21 smithi146 ceph-mon[103278]: pgmap v2443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:41:22.903 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[91136]: pgmap v2444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:22 smithi078 ceph-mon[99959]: pgmap v2444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T21:41:22.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:22.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:22 smithi146 ceph-mon[103278]: pgmap v2444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T21:41:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:23.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:23.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:24.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:23.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:24 smithi146 ceph-mon[103278]: pgmap v2445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:41:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:24 smithi078 ceph-mon[91136]: pgmap v2445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:41:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:24 smithi078 ceph-mon[99959]: pgmap v2445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-20T21:41:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:27 smithi146 ceph-mon[103278]: pgmap v2446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T21:41:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:27 smithi078 ceph-mon[91136]: pgmap v2446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T21:41:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:27 smithi078 ceph-mon[99959]: pgmap v2446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-20T21:41:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:29.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:29 smithi078 ceph-mon[91136]: pgmap v2447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:29.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:29 smithi078 ceph-mon[99959]: pgmap v2447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:29 smithi146 ceph-mon[103278]: pgmap v2447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:41:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T21:41:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:30.438Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:30.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:30.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:31 smithi146 ceph-mon[103278]: pgmap v2448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:31 smithi078 ceph-mon[91136]: pgmap v2448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:31 smithi078 ceph-mon[99959]: pgmap v2448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:33 smithi146 ceph-mon[103278]: pgmap v2449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:33 smithi078 ceph-mon[99959]: pgmap v2449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:33 smithi078 ceph-mon[91136]: pgmap v2449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:33.874Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:33.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:33.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:35 smithi078 ceph-mon[91136]: pgmap v2450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:35 smithi078 ceph-mon[99959]: pgmap v2450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:35 smithi146 ceph-mon[103278]: pgmap v2450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[91136]: pgmap v2451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[99959]: pgmap v2451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:37.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:37 smithi146 ceph-mon[103278]: pgmap v2451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:39 smithi078 ceph-mon[99959]: pgmap v2452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:39 smithi078 ceph-mon[91136]: pgmap v2452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:39.801 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:41:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:39 smithi146 ceph-mon[103278]: pgmap v2452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 92m - - 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 92m - - 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (86m) 2s ago 87m 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (87m) 18s ago 87m 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (79m) 2s ago 85m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (78m) 18s ago 91m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (80m) 2s ago 93m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (78m) 2s ago 93m 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (77m) 18s ago 91m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (77m) 2s ago 91m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:41:40.325 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (87m) 2s ago 86m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (86m) 18s ago 86m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (76m) 2s ago 91m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (75m) 2s ago 90m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (74m) 2s ago 90m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (74m) 2s ago 89m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (73m) 18s ago 89m 75.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (72m) 18s ago 89m 62.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (72m) 18s ago 88m 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (71m) 18s ago 88m 63.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (80m) 18s ago 87m 69.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (70m) 2s ago 85m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (70m) 18s ago 85m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (70m) 2s ago 85m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:41:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (70m) 18s ago 85m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:41:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:40.438Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:40.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:40.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:41:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:41:40.648 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:41:40.935 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:41:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:41:40.936 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:41:40.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:41:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-20T21:41:41.256 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:41:41.256 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:41.256 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:41:41.256 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:41.256 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[91136]: from='client.69506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[91136]: from='client.49593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[91136]: from='client.69518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[91136]: pgmap v2453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/500456534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1495655229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[99959]: from='client.69506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[99959]: from='client.49593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[99959]: from='client.69518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[99959]: pgmap v2453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/500456534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1495655229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:41 smithi146 ceph-mon[103278]: from='client.69506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:41 smithi146 ceph-mon[103278]: from='client.49593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:41 smithi146 ceph-mon[103278]: from='client.69518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:41 smithi146 ceph-mon[103278]: pgmap v2453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/500456534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:41:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1495655229' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:41:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:42 smithi146 ceph-mon[103278]: from='client.69530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[99959]: from='client.69530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[91136]: from='client.69530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:42.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:43.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-mon[91136]: pgmap v2454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:43.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:43.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-mon[99959]: pgmap v2454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:43.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:43.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:43 smithi146 ceph-mon[103278]: pgmap v2454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:43.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:43.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:43.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:43.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:44.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:45 smithi146 ceph-mon[103278]: pgmap v2455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:45 smithi078 ceph-mon[91136]: pgmap v2455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:45 smithi078 ceph-mon[99959]: pgmap v2455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:47.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:47 smithi078 ceph-mon[91136]: pgmap v2456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:47.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:47 smithi078 ceph-mon[99959]: pgmap v2456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:47 smithi146 ceph-mon[103278]: pgmap v2456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:49.639 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:49 smithi078 ceph-mon[99959]: pgmap v2457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:49.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:49 smithi078 ceph-mon[91136]: pgmap v2457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:49 smithi146 ceph-mon[103278]: pgmap v2457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:41:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:41:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:50.439Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:50.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:50.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:51 smithi146 ceph-mon[103278]: pgmap v2458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:51 smithi078 ceph-mon[99959]: pgmap v2458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:51 smithi078 ceph-mon[91136]: pgmap v2458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:53.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:53 smithi078 ceph-mon[99959]: pgmap v2459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:53.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:53 smithi078 ceph-mon[91136]: pgmap v2459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:53 smithi146 ceph-mon[103278]: pgmap v2459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:53.876Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:53.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:41:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:41:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:41:53.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:41:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:41:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:55 smithi146 ceph-mon[103278]: pgmap v2460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:55 smithi078 ceph-mon[91136]: pgmap v2460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:55 smithi078 ceph-mon[99959]: pgmap v2460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:41:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:57 smithi146 ceph-mon[103278]: pgmap v2461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[99959]: pgmap v2461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[91136]: pgmap v2461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:41:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:41:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:41:59 smithi146 ceph-mon[103278]: pgmap v2462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:41:59 smithi078 ceph-mon[99959]: pgmap v2462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:41:59 smithi078 ceph-mon[91136]: pgmap v2462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:42:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:42:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:00.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:00.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:00.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:01 smithi146 ceph-mon[103278]: pgmap v2463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:01 smithi078 ceph-mon[99959]: pgmap v2463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:01 smithi078 ceph-mon[91136]: pgmap v2463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:02.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:02 smithi146 ceph-mon[103278]: pgmap v2464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[91136]: pgmap v2464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:02 smithi078 ceph-mon[99959]: pgmap v2464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:04.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:03.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:03.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:03.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:05.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:04 smithi146 ceph-mon[103278]: pgmap v2465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:04 smithi078 ceph-mon[91136]: pgmap v2465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:04 smithi078 ceph-mon[99959]: pgmap v2465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:07 smithi078 ceph-mon[91136]: pgmap v2466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:07 smithi078 ceph-mon[99959]: pgmap v2466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:07 smithi146 ceph-mon[103278]: pgmap v2466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:09 smithi078 ceph-mon[91136]: pgmap v2467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:09 smithi078 ceph-mon[99959]: pgmap v2467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:09 smithi146 ceph-mon[103278]: pgmap v2467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:10.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:10.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:10.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:10.991 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:42:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:42:11.569 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:42:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:11 smithi078 ceph-mon[99959]: pgmap v2468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:11 smithi078 ceph-mon[91136]: pgmap v2468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:11 smithi146 ceph-mon[103278]: pgmap v2468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 92m - - 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 92m - - 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (86m) 14s ago 88m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (87m) 10s ago 87m 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (80m) 14s ago 85m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:42:12.116 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (79m) 10s ago 91m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (81m) 14s ago 93m 558M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (79m) 14s ago 93m 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (78m) 10s ago 92m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (78m) 14s ago 92m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (87m) 14s ago 87m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (87m) 10s ago 87m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (77m) 14s ago 91m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (76m) 14s ago 91m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (75m) 14s ago 90m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (74m) 14s ago 90m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (74m) 10s ago 90m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (73m) 10s ago 89m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (72m) 10s ago 89m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:42:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (71m) 10s ago 88m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:42:12.118 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (80m) 10s ago 88m 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:42:12.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (71m) 14s ago 86m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:42:12.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (71m) 10s ago 86m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:42:12.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (71m) 14s ago 86m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:42:12.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (71m) 10s ago 86m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:42:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:42:12.429 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:42:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:42:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:42:12.699 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:42:12.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:12 smithi078 ceph-mon[91136]: from='client.49623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:12.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1956845334' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:12.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:12 smithi078 ceph-mon[99959]: from='client.49623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:12.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1956845334' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:13.007 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:42:13.008 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:13.008 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:12 smithi146 ceph-mon[103278]: from='client.49623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1956845334' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[91136]: from='client.69548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[91136]: from='client.49635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[91136]: pgmap v2469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1225087570' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[99959]: from='client.69548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[99959]: from='client.49635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[99959]: pgmap v2469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1225087570' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:13 smithi146 ceph-mon[103278]: from='client.69548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:13 smithi146 ceph-mon[103278]: from='client.49635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:13 smithi146 ceph-mon[103278]: pgmap v2469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1225087570' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:13.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:13.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:13.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:13.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:14 smithi078 ceph-mon[91136]: from='client.69566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:14 smithi078 ceph-mon[99959]: from='client.69566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:14 smithi146 ceph-mon[103278]: from='client.69566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:15 smithi146 ceph-mon[103278]: pgmap v2470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:15 smithi078 ceph-mon[91136]: pgmap v2470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:15 smithi078 ceph-mon[99959]: pgmap v2470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:17 smithi146 ceph-mon[103278]: pgmap v2471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:17 smithi078 ceph-mon[91136]: pgmap v2471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:17 smithi078 ceph-mon[99959]: pgmap v2471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:18 smithi146 ceph-mon[103278]: pgmap v2472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[91136]: pgmap v2472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:18 smithi078 ceph-mon[99959]: pgmap v2472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:42:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:42:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:20.441Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:20.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:20.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:21 smithi078 ceph-mon[91136]: pgmap v2473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:21 smithi078 ceph-mon[99959]: pgmap v2473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:21 smithi146 ceph-mon[103278]: pgmap v2473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:23.270 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:23 smithi146 ceph-mon[103278]: pgmap v2474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[99959]: pgmap v2474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-mon[91136]: pgmap v2474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:24.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:23.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:23.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:24.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:23.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:25.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:25 smithi146 ceph-mon[103278]: pgmap v2475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:25.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:25 smithi078 ceph-mon[91136]: pgmap v2475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:25.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:25 smithi078 ceph-mon[99959]: pgmap v2475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:27 smithi078 ceph-mon[91136]: pgmap v2476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:27 smithi078 ceph-mon[99959]: pgmap v2476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:27 smithi146 ceph-mon[103278]: pgmap v2476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:29 smithi078 ceph-mon[91136]: pgmap v2477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:29 smithi078 ceph-mon[99959]: pgmap v2477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:29 smithi146 ceph-mon[103278]: pgmap v2477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:30.442Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:30.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:30.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:42:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:42:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:31 smithi078 ceph-mon[99959]: pgmap v2478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:31 smithi078 ceph-mon[91136]: pgmap v2478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:31 smithi146 ceph-mon[103278]: pgmap v2478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:33.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:33 smithi078 ceph-mon[91136]: pgmap v2479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:33.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:33 smithi078 ceph-mon[99959]: pgmap v2479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:33 smithi146 ceph-mon[103278]: pgmap v2479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:33.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:33.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:33.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:35 smithi146 ceph-mon[103278]: pgmap v2480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:35 smithi078 ceph-mon[91136]: pgmap v2480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:35 smithi078 ceph-mon[99959]: pgmap v2480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:37.804 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:37 smithi078 ceph-mon[99959]: pgmap v2481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:37.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:37 smithi078 ceph-mon[91136]: pgmap v2481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:37 smithi146 ceph-mon[103278]: pgmap v2481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:38 smithi146 ceph-mon[103278]: pgmap v2482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[91136]: pgmap v2482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:38 smithi078 ceph-mon[99959]: pgmap v2482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:42:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:42:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:40.443Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:40.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:40.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:41 smithi078 ceph-mon[91136]: pgmap v2483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:41 smithi078 ceph-mon[99959]: pgmap v2483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:41 smithi146 ceph-mon[103278]: pgmap v2483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[99959]: pgmap v2484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-mon[91136]: pgmap v2484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:43.294 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:42:43.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:43.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:43 smithi146 ceph-mon[103278]: pgmap v2484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:43.823 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:42:43.823 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 93m - - 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 93m - - 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (87m) 6s ago 88m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (88m) 1s ago 88m 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (80m) 6s ago 86m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (79m) 1s ago 92m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (81m) 6s ago 94m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (79m) 6s ago 94m 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (78m) 1s ago 92m 117M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (78m) 6s ago 92m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (88m) 6s ago 88m 15.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (87m) 1s ago 87m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (77m) 6s ago 92m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (76m) 6s ago 91m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:42:43.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (75m) 6s ago 91m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (75m) 6s ago 91m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (74m) 1s ago 90m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (73m) 1s ago 90m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (73m) 1s ago 89m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (72m) 1s ago 89m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (81m) 1s ago 88m 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (71m) 6s ago 86m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (71m) 1s ago 86m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (71m) 6s ago 86m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:42:43.825 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (71m) 1s ago 86m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:42:44.137 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:42:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:42:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:42:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:42:44.139 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:42:44.139 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:43.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:44.139 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:43.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:44.139 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:43.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:44 smithi146 ceph-mon[103278]: from='client.49659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:44 smithi146 ceph-mon[103278]: from='client.69584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1812641467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:44.407 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:42:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:42:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:42:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:42:44.408 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:42:44.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[91136]: from='client.49659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[91136]: from='client.69584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1812641467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[99959]: from='client.49659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[99959]: from='client.69584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:44.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1812641467' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:42:44.714 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:42:44.714 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:42:44.714 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:42:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:45 smithi146 ceph-mon[103278]: from='client.69590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:45 smithi146 ceph-mon[103278]: from='client.49683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:45 smithi146 ceph-mon[103278]: pgmap v2485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1388675344' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[99959]: from='client.69590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[99959]: from='client.49683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[99959]: pgmap v2485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1388675344' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[91136]: from='client.69590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[91136]: from='client.49683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[91136]: pgmap v2485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1388675344' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:42:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:47 smithi146 ceph-mon[103278]: pgmap v2486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:47 smithi078 ceph-mon[91136]: pgmap v2486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:47 smithi078 ceph-mon[99959]: pgmap v2486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:49 smithi078 ceph-mon[91136]: pgmap v2487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:49 smithi078 ceph-mon[99959]: pgmap v2487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:49 smithi146 ceph-mon[103278]: pgmap v2487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:50.741 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:42:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:42:50.742 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:50.444Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:50.742 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:50.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:50.742 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:50.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:51.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:51 smithi078 ceph-mon[91136]: pgmap v2488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:51.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:51 smithi078 ceph-mon[99959]: pgmap v2488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:51 smithi146 ceph-mon[103278]: pgmap v2488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:53.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:53 smithi078 ceph-mon[99959]: pgmap v2489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:53.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:53 smithi078 ceph-mon[91136]: pgmap v2489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:53 smithi146 ceph-mon[103278]: pgmap v2489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:53.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:53.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:42:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:42:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:42:53.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:42:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:42:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:55 smithi146 ceph-mon[103278]: pgmap v2490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:55 smithi078 ceph-mon[91136]: pgmap v2490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:55 smithi078 ceph-mon[99959]: pgmap v2490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:57.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:57 smithi078 ceph-mon[91136]: pgmap v2491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:57.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:57 smithi078 ceph-mon[99959]: pgmap v2491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:57 smithi146 ceph-mon[103278]: pgmap v2491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:42:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:42:58 smithi146 ceph-mon[103278]: pgmap v2492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[91136]: pgmap v2492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:42:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:42:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:42:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:42:58 smithi078 ceph-mon[99959]: pgmap v2492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:43:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:43:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:00.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:00.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:00.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:01 smithi146 ceph-mon[103278]: pgmap v2493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:01 smithi078 ceph-mon[99959]: pgmap v2493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:01 smithi078 ceph-mon[91136]: pgmap v2493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[91136]: pgmap v2494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-mon[99959]: pgmap v2494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:03 smithi146 ceph-mon[103278]: pgmap v2494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:03.880Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:03.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:03.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:05 smithi146 ceph-mon[103278]: pgmap v2495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:05 smithi078 ceph-mon[91136]: pgmap v2495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:05 smithi078 ceph-mon[99959]: pgmap v2495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:07 smithi146 ceph-mon[103278]: pgmap v2496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:07 smithi078 ceph-mon[91136]: pgmap v2496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:07 smithi078 ceph-mon[99959]: pgmap v2496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:09 smithi146 ceph-mon[103278]: pgmap v2497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:09 smithi078 ceph-mon[91136]: pgmap v2497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:09 smithi078 ceph-mon[99959]: pgmap v2497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:43:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:43:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:10.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:10.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:10.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:11.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:11 smithi078 ceph-mon[99959]: pgmap v2498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:11.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:11 smithi078 ceph-mon[91136]: pgmap v2498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:11 smithi146 ceph-mon[103278]: pgmap v2498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:13.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:13 smithi146 ceph-mon[103278]: pgmap v2499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:13.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:13.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-mon[91136]: pgmap v2499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:13.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:13.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-mon[99959]: pgmap v2499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:13.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:13.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:13.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:13.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:14.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:15.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:15.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:15.016 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 93m - - 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 93m - - 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (88m) 17s ago 89m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (88m) 13s ago 88m 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (81m) 17s ago 86m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (80m) 13s ago 93m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (82m) 17s ago 94m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (80m) 17s ago 94m 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (79m) 13s ago 93m 118M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:43:15.542 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (79m) 17s ago 93m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (88m) 17s ago 88m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (88m) 13s ago 88m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (78m) 17s ago 92m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (77m) 17s ago 92m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (76m) 17s ago 91m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (75m) 17s ago 91m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (75m) 13s ago 91m 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (74m) 13s ago 90m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (73m) 13s ago 90m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (72m) 13s ago 90m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (81m) 13s ago 89m 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (72m) 17s ago 87m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (72m) 13s ago 87m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (72m) 17s ago 87m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:43:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (72m) 13s ago 87m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:43:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:43:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:43:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:43:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:43:15.856 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:15 smithi078 ceph-mon[99959]: pgmap v2500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:15.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:15 smithi078 ceph-mon[91136]: pgmap v2500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:16.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:15 smithi146 ceph-mon[103278]: pgmap v2500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:16.123 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:43:16.124 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:43:16.433 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:43:16.433 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:16.433 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[91136]: from='client.69614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[91136]: from='client.49701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[91136]: from='client.69626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1769661048' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/842732530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[99959]: from='client.69614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[99959]: from='client.49701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[99959]: from='client.69626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1769661048' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:16.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/842732530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:16 smithi146 ceph-mon[103278]: from='client.69614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:16 smithi146 ceph-mon[103278]: from='client.49701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:16 smithi146 ceph-mon[103278]: from='client.69626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1769661048' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/842732530' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:17 smithi146 ceph-mon[103278]: from='client.49719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:17 smithi146 ceph-mon[103278]: pgmap v2501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:17 smithi078 ceph-mon[99959]: from='client.49719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:17 smithi078 ceph-mon[99959]: pgmap v2501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:17 smithi078 ceph-mon[91136]: from='client.49719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:17 smithi078 ceph-mon[91136]: pgmap v2501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:19 smithi146 ceph-mon[103278]: pgmap v2502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[91136]: pgmap v2502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:19 smithi078 ceph-mon[99959]: pgmap v2502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:43:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:43:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:20.446Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:20.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:20.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:21 smithi146 ceph-mon[103278]: pgmap v2503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:21 smithi078 ceph-mon[91136]: pgmap v2503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:21 smithi078 ceph-mon[99959]: pgmap v2503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[91136]: pgmap v2504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-mon[99959]: pgmap v2504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:23 smithi146 ceph-mon[103278]: pgmap v2504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:23.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:23.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:23.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:25 smithi146 ceph-mon[103278]: pgmap v2505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:25 smithi078 ceph-mon[99959]: pgmap v2505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:25 smithi078 ceph-mon[91136]: pgmap v2505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:27.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:27 smithi146 ceph-mon[103278]: pgmap v2506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:27 smithi078 ceph-mon[91136]: pgmap v2506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:28.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:27 smithi078 ceph-mon[99959]: pgmap v2506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:28.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:29 smithi146 ceph-mon[103278]: pgmap v2507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:29 smithi078 ceph-mon[91136]: pgmap v2507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:29 smithi078 ceph-mon[99959]: pgmap v2507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:43:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:43:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:30.446Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:30.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:30.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:31 smithi078 ceph-mon[91136]: pgmap v2508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:31 smithi078 ceph-mon[99959]: pgmap v2508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:31 smithi146 ceph-mon[103278]: pgmap v2508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:33 smithi078 ceph-mon[91136]: pgmap v2509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:33 smithi078 ceph-mon[99959]: pgmap v2509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:33 smithi146 ceph-mon[103278]: pgmap v2509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:34.134 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:33.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:34.134 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:33.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:34.134 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:33.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:34.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:34.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:35 smithi146 ceph-mon[103278]: pgmap v2510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:35 smithi078 ceph-mon[99959]: pgmap v2510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:35 smithi078 ceph-mon[91136]: pgmap v2510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:37 smithi146 ceph-mon[103278]: pgmap v2511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:37 smithi078 ceph-mon[99959]: pgmap v2511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:37 smithi078 ceph-mon[91136]: pgmap v2511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:39 smithi146 ceph-mon[103278]: pgmap v2512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[99959]: pgmap v2512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:39 smithi078 ceph-mon[91136]: pgmap v2512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:43:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:43:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:40.447Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:40.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:40.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:41.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:41 smithi078 ceph-mon[91136]: pgmap v2513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:41.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:41 smithi078 ceph-mon[99959]: pgmap v2513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:41 smithi146 ceph-mon[103278]: pgmap v2513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:43.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[91136]: pgmap v2514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:43.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:43.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[99959]: pgmap v2514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:43.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:43.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:43 smithi146 ceph-mon[103278]: pgmap v2514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:43.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:43.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:43.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:43.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:43.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:44.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:45 smithi146 ceph-mon[103278]: pgmap v2515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:45 smithi078 ceph-mon[91136]: pgmap v2515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:45 smithi078 ceph-mon[99959]: pgmap v2515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:46.736 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:43:47.261 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:43:47.261 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 94m - - 2024-09-20T21:43:47.261 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 94m - - 2024-09-20T21:43:47.261 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (88m) 9s ago 89m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:43:47.261 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (89m) 4s ago 89m 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (81m) 9s ago 87m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (80m) 4s ago 93m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (82m) 9s ago 95m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (80m) 9s ago 95m 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (79m) 4s ago 94m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (79m) 9s ago 93m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (89m) 9s ago 89m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (89m) 4s ago 89m 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (78m) 9s ago 93m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (77m) 9s ago 92m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (77m) 9s ago 92m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (76m) 9s ago 92m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (75m) 4s ago 91m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:43:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (74m) 4s ago 91m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (74m) 4s ago 90m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (73m) 4s ago 90m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (82m) 4s ago 89m 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (72m) 9s ago 87m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (72m) 4s ago 87m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (72m) 9s ago 87m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:43:47.263 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (72m) 4s ago 87m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:43:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:43:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:43:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:43:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:43:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:43:47.573 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:43:47.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:47 smithi146 ceph-mon[103278]: pgmap v2516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:43:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:43:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:43:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:43:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:43:47.845 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:43:47.845 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:47 smithi078 ceph-mon[91136]: pgmap v2516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:47.845 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:47 smithi078 ceph-mon[99959]: pgmap v2516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:48.155 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:43:48.155 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:48.155 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:43:48.155 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:48.155 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:43:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:48 smithi146 ceph-mon[103278]: from='client.69650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:48 smithi146 ceph-mon[103278]: from='client.69656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:48 smithi146 ceph-mon[103278]: from='client.69662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3735489568' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1326544839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[99959]: from='client.69650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[99959]: from='client.69656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[99959]: from='client.69662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3735489568' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1326544839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[91136]: from='client.69650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[91136]: from='client.69656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[91136]: from='client.69662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3735489568' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:43:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1326544839' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:43:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:49 smithi146 ceph-mon[103278]: from='client.49755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:49 smithi146 ceph-mon[103278]: pgmap v2517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:49 smithi078 ceph-mon[91136]: from='client.49755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:49 smithi078 ceph-mon[91136]: pgmap v2517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:49 smithi078 ceph-mon[99959]: from='client.49755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:43:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:49 smithi078 ceph-mon[99959]: pgmap v2517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:43:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:43:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:50.448Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:50.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:50.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:51.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:51 smithi078 ceph-mon[91136]: pgmap v2518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:51.837 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:51 smithi078 ceph-mon[99959]: pgmap v2518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:51 smithi146 ceph-mon[103278]: pgmap v2518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:53 smithi078 ceph-mon[91136]: pgmap v2519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:53 smithi078 ceph-mon[99959]: pgmap v2519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:53 smithi146 ceph-mon[103278]: pgmap v2519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:54.207 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:53.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:54.207 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:53.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:43:54.207 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:43:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:43:53.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:43:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:43:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:55 smithi146 ceph-mon[103278]: pgmap v2520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:55 smithi078 ceph-mon[91136]: pgmap v2520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:55 smithi078 ceph-mon[99959]: pgmap v2520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:43:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:57 smithi146 ceph-mon[103278]: pgmap v2521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:57 smithi078 ceph-mon[91136]: pgmap v2521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:57 smithi078 ceph-mon[99959]: pgmap v2521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:43:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:43:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:43:59 smithi146 ceph-mon[103278]: pgmap v2522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:43:59 smithi078 ceph-mon[91136]: pgmap v2522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:43:59 smithi078 ceph-mon[99959]: pgmap v2522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:44:00] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:44:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:00.449Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:00.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:00.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:01 smithi146 ceph-mon[103278]: pgmap v2523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:01 smithi078 ceph-mon[99959]: pgmap v2523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:01 smithi078 ceph-mon[91136]: pgmap v2523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:03.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[99959]: pgmap v2524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:03.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:03.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:03.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:03.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[91136]: pgmap v2524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:03.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:03.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:03.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:03 smithi146 ceph-mon[103278]: pgmap v2524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:03.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:03.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:03.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:05.894 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:05 smithi146 ceph-mon[103278]: pgmap v2525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:05 smithi078 ceph-mon[99959]: pgmap v2525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:05 smithi078 ceph-mon[91136]: pgmap v2525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:07.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:07 smithi146 ceph-mon[103278]: pgmap v2526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:07 smithi078 ceph-mon[99959]: pgmap v2526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:07 smithi078 ceph-mon[91136]: pgmap v2526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:09 smithi146 ceph-mon[103278]: pgmap v2527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:09 smithi078 ceph-mon[91136]: pgmap v2527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:09 smithi078 ceph-mon[99959]: pgmap v2527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:44:10] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:44:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:10.450Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:10.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:10.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:11.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:11 smithi078 ceph-mon[99959]: pgmap v2528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:11.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:11 smithi078 ceph-mon[91136]: pgmap v2528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:11 smithi146 ceph-mon[103278]: pgmap v2528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:13.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-mon[91136]: pgmap v2529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:13.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:13.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-mon[99959]: pgmap v2529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:13.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:13 smithi146 ceph-mon[103278]: pgmap v2529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:13.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:13.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:13.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:14.656 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:14.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:15 smithi146 ceph-mon[103278]: pgmap v2530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:15 smithi078 ceph-mon[91136]: pgmap v2530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:15 smithi078 ceph-mon[99959]: pgmap v2530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:17 smithi146 ceph-mon[103278]: pgmap v2531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:17 smithi078 ceph-mon[99959]: pgmap v2531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:17 smithi078 ceph-mon[91136]: pgmap v2531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:18.456 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:44:18.977 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 94m - - 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 94m - - 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (89m) 0s ago 90m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (89m) 16s ago 89m 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (82m) 0s ago 87m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (81m) 16s ago 94m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (83m) 0s ago 95m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (81m) 0s ago 95m 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (80m) 16s ago 94m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (80m) 0s ago 94m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:44:18.978 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (89m) 0s ago 89m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (89m) 16s ago 89m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (79m) 0s ago 93m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (78m) 0s ago 93m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (77m) 0s ago 93m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (77m) 0s ago 92m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (76m) 16s ago 92m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (75m) 16s ago 91m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (74m) 16s ago 91m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (73m) 16s ago 91m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (82m) 16s ago 90m 69.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:44:18.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (73m) 0s ago 88m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:44:18.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (73m) 16s ago 88m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:44:18.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (73m) 0s ago 88m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:44:18.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (73m) 16s ago 88m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:44:19.285 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:44:19.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:44:19.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:44:19.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:44:19.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:44:19.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:44:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:19 smithi146 ceph-mon[103278]: from='client.69686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:19 smithi146 ceph-mon[103278]: pgmap v2532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2520270126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:19.556 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:44:19.556 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:44:19.556 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:44:19.556 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:44:19.557 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:44:19.557 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[99959]: from='client.69686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[99959]: pgmap v2532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2520270126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[91136]: from='client.69686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[91136]: pgmap v2532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:19.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2520270126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:19.867 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:44:19.867 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:19.867 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:20 smithi146 ceph-mon[103278]: from='client.69692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:20 smithi146 ceph-mon[103278]: from='client.69698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:20 smithi146 ceph-mon[103278]: from='client.49791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1349040276' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:44:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:44:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[99959]: from='client.69692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[99959]: from='client.69698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[99959]: from='client.49791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1349040276' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:20.450Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:20.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:20.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[91136]: from='client.69692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[91136]: from='client.69698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[91136]: from='client.49791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:20.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1349040276' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:21 smithi146 ceph-mon[103278]: pgmap v2533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:21 smithi078 ceph-mon[91136]: pgmap v2533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:21 smithi078 ceph-mon[99959]: pgmap v2533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[99959]: pgmap v2534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[91136]: pgmap v2534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:23.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:23.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:23 smithi146 ceph-mon[103278]: pgmap v2534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:24.146 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:23.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:24.146 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:23.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:24.146 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:23.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:25.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:25 smithi146 ceph-mon[103278]: pgmap v2535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:25 smithi078 ceph-mon[91136]: pgmap v2535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:25 smithi078 ceph-mon[99959]: pgmap v2535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:26.237 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 21:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T21:44:25.847551225Z level=info msg="Completed cleanup jobs" duration=12.041807ms 2024-09-20T21:44:27.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:27 smithi146 ceph-mon[103278]: pgmap v2536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:27 smithi078 ceph-mon[91136]: pgmap v2536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:27 smithi078 ceph-mon[99959]: pgmap v2536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:29 smithi146 ceph-mon[103278]: pgmap v2537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:29 smithi078 ceph-mon[99959]: pgmap v2537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:29 smithi078 ceph-mon[91136]: pgmap v2537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:44:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:44:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:30.451Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:30.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:30.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:31 smithi146 ceph-mon[103278]: pgmap v2538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:31 smithi078 ceph-mon[91136]: pgmap v2538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:31 smithi078 ceph-mon[99959]: pgmap v2538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:33 smithi078 ceph-mon[91136]: pgmap v2539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:33 smithi078 ceph-mon[99959]: pgmap v2539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:33 smithi146 ceph-mon[103278]: pgmap v2539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:34.118 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:33.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:34.118 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:33.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:34.119 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:33.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:34.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:34.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:35 smithi146 ceph-mon[103278]: pgmap v2540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:35 smithi078 ceph-mon[91136]: pgmap v2540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:35 smithi078 ceph-mon[99959]: pgmap v2540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:37 smithi146 ceph-mon[103278]: pgmap v2541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:37 smithi078 ceph-mon[99959]: pgmap v2541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:37 smithi078 ceph-mon[91136]: pgmap v2541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:39 smithi146 ceph-mon[103278]: pgmap v2542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[99959]: pgmap v2542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[91136]: pgmap v2542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:44:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:44:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:40.451Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:40.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:40.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:41 smithi146 ceph-mon[103278]: pgmap v2543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:42.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:41 smithi078 ceph-mon[91136]: pgmap v2543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:42.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:41 smithi078 ceph-mon[99959]: pgmap v2543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:43.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[91136]: pgmap v2544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[99959]: pgmap v2544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:43.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:43 smithi146 ceph-mon[103278]: pgmap v2544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:44:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:43.886Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:43.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:43.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:45 smithi146 ceph-mon[103278]: pgmap v2545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:45 smithi078 ceph-mon[99959]: pgmap v2545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:45 smithi078 ceph-mon[91136]: pgmap v2545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:47.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:47 smithi146 ceph-mon[103278]: pgmap v2546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:47 smithi078 ceph-mon[91136]: pgmap v2546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:47 smithi078 ceph-mon[99959]: pgmap v2546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:49 smithi146 ceph-mon[103278]: pgmap v2547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:49 smithi078 ceph-mon[99959]: pgmap v2547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:49 smithi078 ceph-mon[91136]: pgmap v2547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:50.159 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:44:50.690 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:44:50.690 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 95m - - 2024-09-20T21:44:50.690 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 95m - - 2024-09-20T21:44:50.690 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (89m) 12s ago 90m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:44:50.690 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (90m) 7s ago 90m 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (82m) 12s ago 88m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (81m) 7s ago 94m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (83m) 12s ago 96m 559M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (81m) 12s ago 96m 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (80m) 7s ago 95m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (81m) 12s ago 94m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (90m) 12s ago 90m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (90m) 7s ago 90m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (79m) 12s ago 94m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (78m) 12s ago 93m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (78m) 12s ago 93m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (77m) 12s ago 93m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (77m) 7s ago 92m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:44:50.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (75m) 7s ago 92m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (75m) 7s ago 91m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (74m) 7s ago 91m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (83m) 7s ago 91m 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (73m) 12s ago 88m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (73m) 7s ago 88m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (73m) 12s ago 88m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:44:50.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (73m) 7s ago 88m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:44:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:44:50] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:44:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:50.452Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:50.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:50.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:51.000 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:44:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:44:51.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:44:51.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:44:51.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:44:51.267 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:44:51.267 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:44:51.267 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:44:51.267 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:44:51.268 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:44:51.578 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:44:51.578 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:44:51.578 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:44:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:51 smithi146 ceph-mon[103278]: from='client.69722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:51 smithi146 ceph-mon[103278]: from='client.49809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:51 smithi146 ceph-mon[103278]: pgmap v2548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2556044849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[91136]: from='client.69722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[91136]: from='client.49809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[91136]: pgmap v2548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2556044849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[99959]: from='client.69722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[99959]: from='client.49809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[99959]: pgmap v2548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2556044849' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:44:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:52 smithi078 ceph-mon[91136]: from='client.69734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:52 smithi078 ceph-mon[91136]: from='client.69746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1298255331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:52 smithi078 ceph-mon[99959]: from='client.69734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:52 smithi078 ceph-mon[99959]: from='client.69746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1298255331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:52 smithi146 ceph-mon[103278]: from='client.69734 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:52 smithi146 ceph-mon[103278]: from='client.69746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:44:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1298255331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:44:53.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:53 smithi078 ceph-mon[91136]: pgmap v2549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:53.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:53 smithi078 ceph-mon[99959]: pgmap v2549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:53 smithi146 ceph-mon[103278]: pgmap v2549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:53.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:53.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:44:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:44:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:44:53.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:44:54.682 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:54.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:44:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:55 smithi078 ceph-mon[91136]: pgmap v2550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:55 smithi078 ceph-mon[99959]: pgmap v2550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:55 smithi146 ceph-mon[103278]: pgmap v2550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:44:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:57 smithi146 ceph-mon[103278]: pgmap v2551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:57 smithi078 ceph-mon[91136]: pgmap v2551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:57 smithi078 ceph-mon[99959]: pgmap v2551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:44:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:44:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:59 smithi146 ceph-mon[103278]: pgmap v2552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:44:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[91136]: pgmap v2552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[99959]: pgmap v2552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:44:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:45:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:45:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:00.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:00.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:00.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:01 smithi078 ceph-mon[91136]: pgmap v2553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:01 smithi078 ceph-mon[99959]: pgmap v2553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:01 smithi146 ceph-mon[103278]: pgmap v2553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:03.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[99959]: pgmap v2554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:03.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:03.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:03.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:03.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[91136]: pgmap v2554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:03.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:03.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:03.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:03 smithi146 ceph-mon[103278]: pgmap v2554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:03.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:03.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:03.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:05 smithi146 ceph-mon[103278]: pgmap v2555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:05 smithi078 ceph-mon[91136]: pgmap v2555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:05 smithi078 ceph-mon[99959]: pgmap v2555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:07.716 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:07 smithi146 ceph-mon[103278]: pgmap v2556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:07 smithi078 ceph-mon[91136]: pgmap v2556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:07 smithi078 ceph-mon[99959]: pgmap v2556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:09.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:09 smithi146 ceph-mon[103278]: pgmap v2557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:09 smithi078 ceph-mon[99959]: pgmap v2557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:09 smithi078 ceph-mon[91136]: pgmap v2557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:45:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:45:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:10.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:10.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:10.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:11 smithi078 ceph-mon[99959]: pgmap v2558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:11 smithi078 ceph-mon[91136]: pgmap v2558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:11 smithi146 ceph-mon[103278]: pgmap v2558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:13.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-mon[91136]: pgmap v2559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:13.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:13.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-mon[99959]: pgmap v2559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:13.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:13 smithi146 ceph-mon[103278]: pgmap v2559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:14.256 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:13.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:14.257 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:13.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:14.257 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:13.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:15.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:15 smithi078 ceph-mon[91136]: pgmap v2560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:15 smithi078 ceph-mon[99959]: pgmap v2560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:15 smithi146 ceph-mon[103278]: pgmap v2560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:16.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:16 smithi078 ceph-mon[99959]: pgmap v2561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:16.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:16 smithi078 ceph-mon[91136]: pgmap v2561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:16 smithi146 ceph-mon[103278]: pgmap v2561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:19 smithi146 ceph-mon[103278]: pgmap v2562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[91136]: pgmap v2562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[99959]: pgmap v2562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:45:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:45:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:20.454Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:20.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:20.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:21.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:21 smithi078 ceph-mon[99959]: pgmap v2563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:21.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:21 smithi078 ceph-mon[91136]: pgmap v2563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:21.873 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:45:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:21 smithi146 ceph-mon[103278]: pgmap v2563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:22.406 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 96m - - 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 95m - - 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (90m) 3s ago 91m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (90m) 19s ago 90m 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (83m) 3s ago 88m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (82m) 19s ago 95m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (84m) 3s ago 96m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (82m) 3s ago 97m 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (81m) 19s ago 95m 119M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:45:22.407 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (81m) 3s ago 95m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (90m) 3s ago 90m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (90m) 19s ago 90m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (80m) 3s ago 94m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (79m) 3s ago 94m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (78m) 3s ago 94m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (78m) 3s ago 93m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (77m) 19s ago 93m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (76m) 19s ago 92m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:45:22.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (75m) 19s ago 92m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:45:22.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (74m) 19s ago 92m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:45:22.409 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (83m) 19s ago 91m 68.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:45:22.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (74m) 3s ago 89m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:45:22.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (74m) 19s ago 89m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:45:22.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (74m) 3s ago 89m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:45:22.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (74m) 19s ago 89m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:45:22.719 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:45:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:45:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:45:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:22.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:45:22.720 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:45:22.989 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:45:22.990 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:45:22.991 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:45:22.991 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:45:22.991 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:45:22.991 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:45:22.991 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:45:22.991 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:45:23.306 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:45:23.306 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:23.306 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:23.306 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:23.306 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='client.49839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='client.69764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='client.49851 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: pgmap v2564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/323425640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1135738699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='client.49839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='client.69764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='client.49851 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: pgmap v2564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/323425640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:23.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1135738699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='client.49839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='client.69764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='client.49851 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: pgmap v2564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/323425640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1135738699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:23.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:23.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:23.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:24 smithi146 ceph-mon[103278]: from='client.49863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:24 smithi078 ceph-mon[91136]: from='client.49863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:24 smithi078 ceph-mon[99959]: from='client.49863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:25 smithi146 ceph-mon[103278]: pgmap v2565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:25 smithi078 ceph-mon[91136]: pgmap v2565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:25 smithi078 ceph-mon[99959]: pgmap v2565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:27 smithi146 ceph-mon[103278]: pgmap v2566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:27 smithi078 ceph-mon[91136]: pgmap v2566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:27 smithi078 ceph-mon[99959]: pgmap v2566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:30.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:29 smithi146 ceph-mon[103278]: pgmap v2567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:29 smithi078 ceph-mon[91136]: pgmap v2567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:29 smithi078 ceph-mon[99959]: pgmap v2567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:45:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:45:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:30.454Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:30.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:30.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:32.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:31 smithi146 ceph-mon[103278]: pgmap v2568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:31 smithi078 ceph-mon[91136]: pgmap v2568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:31 smithi078 ceph-mon[99959]: pgmap v2568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:33.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:33 smithi078 ceph-mon[99959]: pgmap v2569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:33.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:33 smithi078 ceph-mon[91136]: pgmap v2569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:33 smithi146 ceph-mon[103278]: pgmap v2569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:33.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:33.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:33.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:34.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:34.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:35.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:35 smithi078 ceph-mon[99959]: pgmap v2570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:35.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:35 smithi078 ceph-mon[91136]: pgmap v2570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:35 smithi146 ceph-mon[103278]: pgmap v2570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:37.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:37 smithi078 ceph-mon[91136]: pgmap v2571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:37.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:37 smithi078 ceph-mon[99959]: pgmap v2571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:37 smithi146 ceph-mon[103278]: pgmap v2571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:39 smithi146 ceph-mon[103278]: pgmap v2572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[91136]: pgmap v2572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[99959]: pgmap v2572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:45:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:45:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:40.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:40.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:40.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:41 smithi146 ceph-mon[103278]: pgmap v2573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:41 smithi078 ceph-mon[91136]: pgmap v2573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:41 smithi078 ceph-mon[99959]: pgmap v2573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:43.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[99959]: pgmap v2574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:43.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:43.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:43.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:43.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:43.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[91136]: pgmap v2574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:43.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:43.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:43.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:43.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:43 smithi146 ceph-mon[103278]: pgmap v2574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:45:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:43.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:43.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:43.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:45 smithi146 ceph-mon[103278]: pgmap v2575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:45 smithi078 ceph-mon[91136]: pgmap v2575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:45 smithi078 ceph-mon[99959]: pgmap v2575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:46 smithi146 ceph-mon[103278]: pgmap v2576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:46 smithi078 ceph-mon[91136]: pgmap v2576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:46 smithi078 ceph-mon[99959]: pgmap v2576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:49.868 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:49 smithi146 ceph-mon[103278]: pgmap v2577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:49 smithi078 ceph-mon[91136]: pgmap v2577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:49 smithi078 ceph-mon[99959]: pgmap v2577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:45:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:45:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:50.456Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:50.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:50.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:51 smithi146 ceph-mon[103278]: pgmap v2578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:51 smithi078 ceph-mon[91136]: pgmap v2578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:51 smithi078 ceph-mon[99959]: pgmap v2578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:53.606 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:45:53.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:53 smithi078 ceph-mon[99959]: pgmap v2579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:53.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:53 smithi078 ceph-mon[91136]: pgmap v2579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:53 smithi146 ceph-mon[103278]: pgmap v2579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:54.133 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:45:54.133 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 96m - - 2024-09-20T21:45:54.133 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 96m - - 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (90m) 15s ago 91m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (91m) 10s ago 91m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (84m) 15s ago 89m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (82m) 10s ago 95m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (84m) 15s ago 97m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (83m) 15s ago 97m 192M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (81m) 10s ago 96m 120M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:45:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (82m) 15s ago 95m 126M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (91m) 15s ago 91m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (91m) 10s ago 91m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (80m) 15s ago 95m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (79m) 15s ago 95m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (79m) 15s ago 94m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (78m) 15s ago 94m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (78m) 10s ago 93m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (77m) 10s ago 93m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (76m) 10s ago 93m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (75m) 10s ago 92m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (84m) 10s ago 92m 68.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:45:54.135 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (75m) 15s ago 89m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:45:54.136 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (74m) 10s ago 89m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:45:54.136 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (74m) 15s ago 89m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:45:54.136 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (74m) 10s ago 89m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:45:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:53.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:53.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:45:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:45:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:45:53.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:45:54.449 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:45:54.449 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:45:54.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:45:54.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:54.449 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:45:54.450 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:45:54.718 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:45:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:45:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:45:54.719 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:45:54.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:54.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:54 smithi078 ceph-mon[91136]: from='client.69794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:54.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4088782721' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:54.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:54.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:54 smithi078 ceph-mon[99959]: from='client.69794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:54.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4088782721' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:45:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:54 smithi146 ceph-mon[103278]: from='client.69794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4088782721' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:45:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:45:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:55.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:45:55.029 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:55.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:45:55.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[91136]: from='client.69800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:55.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[91136]: from='client.69806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:55.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[91136]: pgmap v2580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:55.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3280419203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:55.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[99959]: from='client.69800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:55.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[99959]: from='client.69806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:55.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[99959]: pgmap v2580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:55.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3280419203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:55 smithi146 ceph-mon[103278]: from='client.69800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:55 smithi146 ceph-mon[103278]: from='client.69806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:55 smithi146 ceph-mon[103278]: pgmap v2580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:45:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3280419203' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:45:56.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:56 smithi078 ceph-mon[91136]: from='client.69818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:56.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:56 smithi078 ceph-mon[99959]: from='client.69818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:56 smithi146 ceph-mon[103278]: from='client.69818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:45:57.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:57 smithi078 ceph-mon[99959]: pgmap v2581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:57.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:57 smithi078 ceph-mon[91136]: pgmap v2581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:57 smithi146 ceph-mon[103278]: pgmap v2581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:45:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:45:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:59 smithi146 ceph-mon[103278]: pgmap v2582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[91136]: pgmap v2582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[99959]: pgmap v2582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:46:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:46:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:00.457Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:00.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:00.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:01 smithi146 ceph-mon[103278]: pgmap v2583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:01 smithi078 ceph-mon[91136]: pgmap v2583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:01 smithi078 ceph-mon[99959]: pgmap v2583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:03 smithi146 ceph-mon[103278]: pgmap v2584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[99959]: pgmap v2584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[91136]: pgmap v2584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:03.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:03.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:03.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:03.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:05 smithi146 ceph-mon[103278]: pgmap v2585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:05 smithi078 ceph-mon[99959]: pgmap v2585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:05 smithi078 ceph-mon[91136]: pgmap v2585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:06 smithi146 ceph-mon[103278]: pgmap v2586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:06 smithi078 ceph-mon[91136]: pgmap v2586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:06 smithi078 ceph-mon[99959]: pgmap v2586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:09 smithi146 ceph-mon[103278]: pgmap v2587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:09 smithi078 ceph-mon[99959]: pgmap v2587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:09 smithi078 ceph-mon[91136]: pgmap v2587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:46:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:46:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:10.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:10.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:10.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:11 smithi146 ceph-mon[103278]: pgmap v2588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:11 smithi078 ceph-mon[91136]: pgmap v2588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:11 smithi078 ceph-mon[99959]: pgmap v2588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:13.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-mon[91136]: pgmap v2589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:13.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:13.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-mon[99959]: pgmap v2589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:13.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:13 smithi146 ceph-mon[103278]: pgmap v2589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:13.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:13.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:13.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:15 smithi146 ceph-mon[103278]: pgmap v2590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:15 smithi078 ceph-mon[99959]: pgmap v2590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:15 smithi078 ceph-mon[91136]: pgmap v2590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:17.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:17 smithi078 ceph-mon[91136]: pgmap v2591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:17.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:17 smithi078 ceph-mon[99959]: pgmap v2591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:17 smithi146 ceph-mon[103278]: pgmap v2591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:19 smithi146 ceph-mon[103278]: pgmap v2592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[91136]: pgmap v2592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[99959]: pgmap v2592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:46:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:46:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:20.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:20.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:20.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:21 smithi146 ceph-mon[103278]: pgmap v2593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:21 smithi078 ceph-mon[91136]: pgmap v2593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:21 smithi078 ceph-mon[99959]: pgmap v2593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:23.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[99959]: pgmap v2594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[91136]: pgmap v2594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:23.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:23 smithi146 ceph-mon[103278]: pgmap v2594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:23.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": context deadline exceeded" 2024-09-20T21:46:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:23.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:23.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:25.322 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:46:25.850 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:46:25.850 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 97m - - 2024-09-20T21:46:25.850 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 96m - - 2024-09-20T21:46:25.850 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (91m) 6s ago 92m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:46:25.850 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (92m) 2s ago 92m 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:46:25.850 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (84m) 6s ago 89m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (83m) 2s ago 96m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (85m) 6s ago 97m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (83m) 6s ago 98m 193M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (82m) 2s ago 96m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (82m) 6s ago 96m 132M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (91m) 6s ago 91m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (91m) 2s ago 91m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (81m) 6s ago 95m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (80m) 6s ago 95m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (79m) 6s ago 95m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (79m) 6s ago 94m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (78m) 2s ago 94m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (77m) 2s ago 93m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:46:25.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (76m) 2s ago 93m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:46:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (75m) 2s ago 93m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:46:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (84m) 2s ago 92m 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:46:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (75m) 6s ago 90m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:46:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (75m) 2s ago 90m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:46:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (75m) 6s ago 90m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:46:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (75m) 2s ago 90m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:46:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:25 smithi146 ceph-mon[103278]: pgmap v2595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:25 smithi078 ceph-mon[99959]: pgmap v2595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:25 smithi078 ceph-mon[91136]: pgmap v2595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:26.158 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:46:26.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:46:26.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:46:26.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:26.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:46:26.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:46:26.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:26.160 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:46:26.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:46:26.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:46:26.161 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:46:26.430 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:46:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:46:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:46:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:46:26.432 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:46:26.743 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:46:26.744 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:26.744 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:26.744 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:26.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:26 smithi078 ceph-mon[99959]: from='client.49911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:26.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:26 smithi078 ceph-mon[99959]: from='client.69836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:26.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1081715249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:26.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:26 smithi078 ceph-mon[91136]: from='client.49911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:26.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:26 smithi078 ceph-mon[91136]: from='client.69836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:26.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1081715249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:26 smithi146 ceph-mon[103278]: from='client.49911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:26 smithi146 ceph-mon[103278]: from='client.69836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1081715249' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:27 smithi146 ceph-mon[103278]: from='client.69842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:27 smithi146 ceph-mon[103278]: from='client.49935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:27 smithi146 ceph-mon[103278]: pgmap v2596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3168895914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[99959]: from='client.69842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[99959]: from='client.49935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[99959]: pgmap v2596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3168895914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[91136]: from='client.69842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[91136]: from='client.49935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[91136]: pgmap v2596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3168895914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:28 smithi078 ceph-mon[91136]: pgmap v2597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:28 smithi078 ceph-mon[99959]: pgmap v2597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:28 smithi146 ceph-mon[103278]: pgmap v2597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:46:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:46:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:30.459Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:30.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:30.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:31.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:31 smithi146 ceph-mon[103278]: pgmap v2598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:31 smithi078 ceph-mon[91136]: pgmap v2598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:31 smithi078 ceph-mon[99959]: pgmap v2598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:33.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:33 smithi078 ceph-mon[99959]: pgmap v2599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:33.896 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:33 smithi078 ceph-mon[91136]: pgmap v2599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:33 smithi146 ceph-mon[103278]: pgmap v2599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:33.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:33.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:33.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:35 smithi146 ceph-mon[103278]: pgmap v2600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:35 smithi078 ceph-mon[91136]: pgmap v2600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:35 smithi078 ceph-mon[99959]: pgmap v2600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:37.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:37 smithi078 ceph-mon[99959]: pgmap v2601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:37.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:37 smithi078 ceph-mon[91136]: pgmap v2601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:37 smithi146 ceph-mon[103278]: pgmap v2601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:39 smithi146 ceph-mon[103278]: pgmap v2602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[91136]: pgmap v2602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[99959]: pgmap v2602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:46:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T21:46:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:40.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:40.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:40.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:41.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:41 smithi078 ceph-mon[91136]: pgmap v2603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:41.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:41 smithi078 ceph-mon[99959]: pgmap v2603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:41 smithi146 ceph-mon[103278]: pgmap v2603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:43.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-mon[99959]: pgmap v2604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:43.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:43.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-mon[91136]: pgmap v2604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:43.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:43 smithi146 ceph-mon[103278]: pgmap v2604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:43.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:43.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:43.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:44.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:44.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:44.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:44.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:45.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:46:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:45 smithi146 ceph-mon[103278]: pgmap v2605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:45 smithi078 ceph-mon[91136]: pgmap v2605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:45 smithi078 ceph-mon[99959]: pgmap v2605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:47 smithi146 ceph-mon[103278]: pgmap v2606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:47 smithi078 ceph-mon[91136]: pgmap v2606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:47 smithi078 ceph-mon[99959]: pgmap v2606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:49 smithi146 ceph-mon[103278]: pgmap v2607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:49 smithi078 ceph-mon[91136]: pgmap v2607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:49 smithi078 ceph-mon[99959]: pgmap v2607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:46:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:46:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:50.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:50.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:50.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:51 smithi078 ceph-mon[91136]: pgmap v2608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:51 smithi078 ceph-mon[99959]: pgmap v2608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:51.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:51 smithi146 ceph-mon[103278]: pgmap v2608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:52 smithi146 ceph-mon[103278]: pgmap v2609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:52 smithi078 ceph-mon[91136]: pgmap v2609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:52 smithi078 ceph-mon[99959]: pgmap v2609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:46:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:53.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:53.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:46:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:46:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:46:53.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:46:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:54 smithi146 ceph-mon[103278]: pgmap v2610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:54 smithi078 ceph-mon[91136]: pgmap v2610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:54 smithi078 ceph-mon[99959]: pgmap v2610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:46:57.043 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:46:57.576 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:46:57.576 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 97m - - 2024-09-20T21:46:57.576 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 97m - - 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (91m) 18s ago 92m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (92m) 13s ago 92m 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (85m) 18s ago 90m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (83m) 13s ago 96m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (85m) 18s ago 98m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (84m) 18s ago 98m 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (82m) 13s ago 97m 122M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (83m) 18s ago 97m 128M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (92m) 18s ago 92m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (92m) 13s ago 92m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:46:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (81m) 18s ago 96m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (80m) 18s ago 96m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (80m) 18s ago 95m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (79m) 18s ago 95m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (79m) 13s ago 94m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (78m) 13s ago 94m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (77m) 13s ago 94m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (76m) 13s ago 93m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (85m) 13s ago 93m 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (76m) 18s ago 90m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (75m) 13s ago 90m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:46:57.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (76m) 18s ago 90m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:46:57.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (75m) 13s ago 90m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:46:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:57 smithi078 ceph-mon[91136]: pgmap v2611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:57 smithi078 ceph-mon[99959]: pgmap v2611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:57.897 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:46:57.897 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:46:57.897 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:46:57.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:57.897 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:46:57.898 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:46:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:57 smithi146 ceph-mon[103278]: pgmap v2611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:46:58.168 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:46:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:46:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:46:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:46:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:46:58.169 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:46:58.486 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:46:58.486 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:46:58.486 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[91136]: from='client.69866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[91136]: from='client.49953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[91136]: from='client.49959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3413244547' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2567734302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[99959]: from='client.69866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[99959]: from='client.49953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[99959]: from='client.49959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3413244547' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2567734302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:58 smithi146 ceph-mon[103278]: from='client.69866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:58 smithi146 ceph-mon[103278]: from='client.49953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:58 smithi146 ceph-mon[103278]: from='client.49959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3413244547' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:46:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:46:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2567734302' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:46:59.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[99959]: from='client.49971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:46:59.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[91136]: from='client.49971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:59 smithi146 ceph-mon[103278]: from='client.49971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:59 smithi146 ceph-mon[103278]: pgmap v2612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[91136]: pgmap v2612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[99959]: pgmap v2612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:47:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:47:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:00.461Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:00.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:00.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:01 smithi146 ceph-mon[103278]: pgmap v2613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:01 smithi078 ceph-mon[91136]: pgmap v2613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:01 smithi078 ceph-mon[99959]: pgmap v2613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:03.898 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:03 smithi078 ceph-mon[91136]: pgmap v2614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:03.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:03 smithi078 ceph-mon[99959]: pgmap v2614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:03 smithi146 ceph-mon[103278]: pgmap v2614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:03.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:03.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:03.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:06.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:05 smithi146 ceph-mon[103278]: pgmap v2615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:05 smithi078 ceph-mon[91136]: pgmap v2615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:05 smithi078 ceph-mon[99959]: pgmap v2615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:07 smithi146 ceph-mon[103278]: pgmap v2616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:07 smithi078 ceph-mon[91136]: pgmap v2616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:07 smithi078 ceph-mon[99959]: pgmap v2616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:09 smithi146 ceph-mon[103278]: pgmap v2617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:09 smithi078 ceph-mon[99959]: pgmap v2617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:09 smithi078 ceph-mon[91136]: pgmap v2617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:10.773 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:47:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:47:10.774 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:10.461Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:10.774 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:10.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:10.774 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:10.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:11.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:11 smithi146 ceph-mon[103278]: pgmap v2618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:11 smithi078 ceph-mon[91136]: pgmap v2618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:11 smithi078 ceph-mon[99959]: pgmap v2618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:13.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-mon[99959]: pgmap v2619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:13.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:13.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-mon[91136]: pgmap v2619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:13.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:13 smithi146 ceph-mon[103278]: pgmap v2619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:13.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:13.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:13.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:15 smithi146 ceph-mon[103278]: pgmap v2620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:15 smithi078 ceph-mon[91136]: pgmap v2620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:15 smithi078 ceph-mon[99959]: pgmap v2620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:16 smithi146 ceph-mon[103278]: pgmap v2621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:16 smithi078 ceph-mon[91136]: pgmap v2621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:16 smithi078 ceph-mon[99959]: pgmap v2621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:19 smithi146 ceph-mon[103278]: pgmap v2622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[99959]: pgmap v2622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[91136]: pgmap v2622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:20.462Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:20.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:20.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:47:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:47:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:21 smithi146 ceph-mon[103278]: pgmap v2623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:21 smithi078 ceph-mon[91136]: pgmap v2623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:21 smithi078 ceph-mon[99959]: pgmap v2623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:23 smithi078 ceph-mon[91136]: pgmap v2624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:23 smithi078 ceph-mon[99959]: pgmap v2624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:23 smithi146 ceph-mon[103278]: pgmap v2624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:23.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:23.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:23.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:24.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:24.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:25 smithi146 ceph-mon[103278]: pgmap v2625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:25 smithi078 ceph-mon[91136]: pgmap v2625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:25 smithi078 ceph-mon[99959]: pgmap v2625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:27 smithi146 ceph-mon[103278]: pgmap v2626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:27 smithi078 ceph-mon[91136]: pgmap v2626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:27 smithi078 ceph-mon[99959]: pgmap v2626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:28.791 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:47:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:29.363 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:47:29.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 98m - - 2024-09-20T21:47:29.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 97m - - 2024-09-20T21:47:29.363 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (92m) 9s ago 93m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (93m) 5s ago 93m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (85m) 9s ago 91m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (84m) 5s ago 97m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (86m) 9s ago 98m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (84m) 9s ago 99m 195M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (83m) 5s ago 97m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (83m) 9s ago 97m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (92m) 9s ago 92m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (92m) 5s ago 92m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (82m) 9s ago 96m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (81m) 9s ago 96m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:47:29.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (80m) 9s ago 96m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (80m) 9s ago 95m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (79m) 5s ago 95m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (78m) 5s ago 95m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (77m) 5s ago 94m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (76m) 5s ago 94m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (85m) 5s ago 93m 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (76m) 9s ago 91m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (76m) 5s ago 91m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (76m) 9s ago 91m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:47:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (76m) 5s ago 91m 88.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:47:29.678 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:47:29.678 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:47:29.679 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:47:29.946 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:47:29.947 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:47:29.947 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:47:29.947 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:47:29.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:29 smithi078 ceph-mon[99959]: pgmap v2627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:29.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:29 smithi078 ceph-mon[91136]: pgmap v2627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:29 smithi146 ceph-mon[103278]: pgmap v2627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:30.254 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:47:30.254 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:47:30.254 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:47:30.254 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:47:30.254 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:47:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:30.463Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:30.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:30.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:30 smithi146 ceph-mon[103278]: from='client.69902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:30 smithi146 ceph-mon[103278]: from='client.69908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:30 smithi146 ceph-mon[103278]: from='client.69914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2081230724' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3340773782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:47:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[91136]: from='client.69902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[91136]: from='client.69908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[91136]: from='client.69914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2081230724' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3340773782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[99959]: from='client.69902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[99959]: from='client.69908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[99959]: from='client.69914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2081230724' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:47:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3340773782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:47:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:31 smithi146 ceph-mon[103278]: from='client.50007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:31 smithi146 ceph-mon[103278]: pgmap v2628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:31 smithi078 ceph-mon[91136]: from='client.50007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:31 smithi078 ceph-mon[91136]: pgmap v2628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:31 smithi078 ceph-mon[99959]: from='client.50007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:47:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:31 smithi078 ceph-mon[99959]: pgmap v2628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:33.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:33 smithi078 ceph-mon[99959]: pgmap v2629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:33.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:33 smithi078 ceph-mon[91136]: pgmap v2629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:33 smithi146 ceph-mon[103278]: pgmap v2629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:33.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:33.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:33.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:35 smithi146 ceph-mon[103278]: pgmap v2630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:35 smithi078 ceph-mon[91136]: pgmap v2630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:35 smithi078 ceph-mon[99959]: pgmap v2630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:36 smithi146 ceph-mon[103278]: pgmap v2631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:36 smithi078 ceph-mon[91136]: pgmap v2631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:36 smithi078 ceph-mon[99959]: pgmap v2631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:39.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:39 smithi078 ceph-mon[99959]: pgmap v2632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:39.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:39 smithi078 ceph-mon[91136]: pgmap v2632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:39 smithi146 ceph-mon[103278]: pgmap v2632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:47:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:47:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[91136]: pgmap v2633: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-mon[99959]: pgmap v2633: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:40.464Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:40.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:40.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:40 smithi146 ceph-mon[103278]: pgmap v2633: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:43.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-mon[91136]: pgmap v2634: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:43.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:43.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-mon[99959]: pgmap v2634: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:43.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:43 smithi146 ceph-mon[103278]: pgmap v2634: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:43.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:43.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:43.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:47:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:45 smithi146 ceph-mon[103278]: pgmap v2635: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:45 smithi078 ceph-mon[91136]: pgmap v2635: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:45 smithi078 ceph-mon[99959]: pgmap v2635: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:47 smithi146 ceph-mon[103278]: pgmap v2636: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:47 smithi078 ceph-mon[91136]: pgmap v2636: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:47 smithi078 ceph-mon[99959]: pgmap v2636: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:49 smithi078 ceph-mon[91136]: pgmap v2637: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:49 smithi078 ceph-mon[99959]: pgmap v2637: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:49 smithi146 ceph-mon[103278]: pgmap v2637: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:47:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:47:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:50.465Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:50.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:50.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:51.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:51 smithi146 ceph-mon[103278]: pgmap v2638: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:51 smithi078 ceph-mon[91136]: pgmap v2638: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:51 smithi078 ceph-mon[99959]: pgmap v2638: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:53.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:53 smithi078 ceph-mon[91136]: pgmap v2639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:53.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:53 smithi078 ceph-mon[99959]: pgmap v2639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:53 smithi146 ceph-mon[103278]: pgmap v2639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:53.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:53.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:47:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:47:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:47:53.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:47:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:47:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:55 smithi146 ceph-mon[103278]: pgmap v2640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:55 smithi078 ceph-mon[91136]: pgmap v2640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:55 smithi078 ceph-mon[99959]: pgmap v2640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:56 smithi146 ceph-mon[103278]: pgmap v2641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:56 smithi078 ceph-mon[91136]: pgmap v2641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:56 smithi078 ceph-mon[99959]: pgmap v2641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:47:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:47:59.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:47:59 smithi078 ceph-mon[99959]: pgmap v2642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:59.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:47:59 smithi078 ceph-mon[91136]: pgmap v2642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:47:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:47:59 smithi146 ceph-mon[103278]: pgmap v2642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:00.555 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:48:00.665 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:48:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:48:00.665 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:00.466Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:00.665 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:00.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:00.666 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:00.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[99959]: from='client.50019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[99959]: pgmap v2643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[91136]: from='client.50019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:00 smithi078 ceph-mon[91136]: pgmap v2643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 98m - - 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 98m - - 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (92m) 1s ago 93m 31.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (93m) 17s ago 93m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (86m) 1s ago 91m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (84m) 17s ago 97m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (86m) 1s ago 99m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (85m) 1s ago 99m 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (83m) 17s ago 98m 123M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (84m) 1s ago 98m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (93m) 1s ago 93m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (93m) 17s ago 93m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:48:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (82m) 1s ago 97m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (81m) 1s ago 97m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (81m) 1s ago 96m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (80m) 1s ago 96m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (80m) 17s ago 95m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (79m) 17s ago 95m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (78m) 17s ago 95m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (77m) 17s ago 94m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (86m) 17s ago 94m 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (77m) 1s ago 91m 87.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (76m) 17s ago 92m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (77m) 1s ago 91m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:48:01.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (76m) 17s ago 91m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:48:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:00 smithi146 ceph-mon[103278]: from='client.50019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:00 smithi146 ceph-mon[103278]: pgmap v2643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:48:01.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:48:01.434 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:48:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:48:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:48:01.709 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:48:01.710 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:48:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:48:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:02.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:02.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:02.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:01 smithi078 ceph-mon[99959]: from='client.69944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:02.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:01 smithi078 ceph-mon[99959]: from='client.69950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:02.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/921850342' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:02.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:01 smithi078 ceph-mon[91136]: from='client.69944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:02.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:01 smithi078 ceph-mon[91136]: from='client.69950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:02.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/921850342' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:01 smithi146 ceph-mon[103278]: from='client.69944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:01 smithi146 ceph-mon[103278]: from='client.69950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/921850342' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:03.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:02 smithi078 ceph-mon[91136]: from='client.50043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:03.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/906228745' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:03.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:02 smithi078 ceph-mon[91136]: pgmap v2644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:03.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:02 smithi078 ceph-mon[99959]: from='client.50043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:03.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/906228745' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:03.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:02 smithi078 ceph-mon[99959]: pgmap v2644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:02 smithi146 ceph-mon[103278]: from='client.50043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/906228745' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:02 smithi146 ceph-mon[103278]: pgmap v2644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:04.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:03.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:03.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:04.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:03.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:05 smithi146 ceph-mon[103278]: pgmap v2645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[91136]: pgmap v2645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:05.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:05 smithi078 ceph-mon[99959]: pgmap v2645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:07 smithi146 ceph-mon[103278]: pgmap v2646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:07 smithi078 ceph-mon[91136]: pgmap v2646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:07 smithi078 ceph-mon[99959]: pgmap v2646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:09 smithi146 ceph-mon[103278]: pgmap v2647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:09 smithi078 ceph-mon[91136]: pgmap v2647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:09 smithi078 ceph-mon[99959]: pgmap v2647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:48:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:48:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:10.466Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:10.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:10.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:11.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:11 smithi078 ceph-mon[91136]: pgmap v2648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:11.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:11 smithi078 ceph-mon[99959]: pgmap v2648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:11.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:11 smithi146 ceph-mon[103278]: pgmap v2648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:13.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-mon[99959]: pgmap v2649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:13.903 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:13.903 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-mon[91136]: pgmap v2649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:13.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:13.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:13 smithi146 ceph-mon[103278]: pgmap v2649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:13.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:13.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:13.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:13.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:14.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:15 smithi146 ceph-mon[103278]: pgmap v2650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:15 smithi078 ceph-mon[91136]: pgmap v2650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:15 smithi078 ceph-mon[99959]: pgmap v2650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:17 smithi146 ceph-mon[103278]: pgmap v2651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:17 smithi078 ceph-mon[99959]: pgmap v2651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:17 smithi078 ceph-mon[91136]: pgmap v2651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:18 smithi146 ceph-mon[103278]: pgmap v2652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:18 smithi078 ceph-mon[91136]: pgmap v2652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:18 smithi078 ceph-mon[99959]: pgmap v2652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:20.467Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:20.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:20.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:20.948 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:48:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[91136]: pgmap v2653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:20 smithi078 ceph-mon[99959]: pgmap v2653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:20 smithi146 ceph-mon[103278]: pgmap v2653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:23.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:23 smithi078 ceph-mon[91136]: pgmap v2654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:23.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:23 smithi078 ceph-mon[99959]: pgmap v2654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:23 smithi146 ceph-mon[103278]: pgmap v2654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:23.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:23.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:23.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:25.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:25 smithi146 ceph-mon[103278]: pgmap v2655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:25 smithi078 ceph-mon[91136]: pgmap v2655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:25 smithi078 ceph-mon[99959]: pgmap v2655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:27 smithi146 ceph-mon[103278]: pgmap v2656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:27 smithi078 ceph-mon[91136]: pgmap v2656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:27 smithi078 ceph-mon[99959]: pgmap v2656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:30.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:29 smithi078 ceph-mon[99959]: pgmap v2657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:30.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:29 smithi078 ceph-mon[91136]: pgmap v2657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:29 smithi146 ceph-mon[103278]: pgmap v2657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:30 smithi078 ceph-mon[91136]: pgmap v2658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:30 smithi078 ceph-mon[99959]: pgmap v2658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:48:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:48:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:30.468Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:30.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:30.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:30 smithi146 ceph-mon[103278]: pgmap v2658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:32.316 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:48:32.847 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:48:32.847 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 99m - - 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 98m - - 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (93m) 12s ago 94m 31.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (94m) 8s ago 94m 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (86m) 12s ago 92m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (85m) 8s ago 98m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (87m) 12s ago 99m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (85m) 12s ago 100m 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (84m) 8s ago 98m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (84m) 12s ago 98m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (93m) 12s ago 93m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (93m) 8s ago 93m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (83m) 12s ago 98m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:48:32.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (82m) 12s ago 97m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (81m) 12s ago 97m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (81m) 12s ago 96m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (80m) 8s ago 96m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (79m) 8s ago 96m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (78m) 8s ago 95m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (77m) 8s ago 95m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (86m) 8s ago 94m 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (77m) 12s ago 92m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (77m) 8s ago 92m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (77m) 12s ago 92m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:48:32.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (77m) 8s ago 92m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:48:33.155 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:48:33.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:48:33.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:48:33.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:33.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:48:33.156 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:48:33.428 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:48:33.429 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:48:33.738 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:48:33.739 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:33.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:48:33.739 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:33.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:48:33.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[99959]: from='client.50055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:33.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[99959]: from='client.69980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:33.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[99959]: pgmap v2659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:33.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4015542498' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:33.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[91136]: from='client.50055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:33.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[91136]: from='client.69980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:33.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[91136]: pgmap v2659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:33.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4015542498' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:33.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:33 smithi146 ceph-mon[103278]: from='client.50055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:33.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:33 smithi146 ceph-mon[103278]: from='client.69980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:33.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:33 smithi146 ceph-mon[103278]: pgmap v2659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:33.826 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4015542498' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:48:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:33.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:33.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:33.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:34 smithi146 ceph-mon[103278]: from='client.69986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:34 smithi146 ceph-mon[103278]: from='client.69998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/192646695' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[91136]: from='client.69986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[91136]: from='client.69998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/192646695' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[99959]: from='client.69986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[99959]: from='client.69998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:48:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/192646695' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:48:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:35 smithi146 ceph-mon[103278]: pgmap v2660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:35 smithi078 ceph-mon[91136]: pgmap v2660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:35 smithi078 ceph-mon[99959]: pgmap v2660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:37 smithi146 ceph-mon[103278]: pgmap v2661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:37 smithi078 ceph-mon[99959]: pgmap v2661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:37 smithi078 ceph-mon[91136]: pgmap v2661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:38 smithi146 ceph-mon[103278]: pgmap v2662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:38 smithi078 ceph-mon[99959]: pgmap v2662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:38 smithi078 ceph-mon[91136]: pgmap v2662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:40.758 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:48:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:48:40.758 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:40.468Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:40.758 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:40.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:40.758 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:40.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[99959]: pgmap v2663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:41 smithi078 ceph-mon[91136]: pgmap v2663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:41 smithi146 ceph-mon[103278]: pgmap v2663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:43.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-mon[91136]: pgmap v2664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:43.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:43.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-mon[99959]: pgmap v2664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:43.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:43 smithi146 ceph-mon[103278]: pgmap v2664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:43.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:43.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:43.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:48:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:45 smithi146 ceph-mon[103278]: pgmap v2665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:45 smithi078 ceph-mon[91136]: pgmap v2665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:45 smithi078 ceph-mon[99959]: pgmap v2665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:46 smithi146 ceph-mon[103278]: pgmap v2666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:46 smithi078 ceph-mon[91136]: pgmap v2666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:46 smithi078 ceph-mon[99959]: pgmap v2666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:49 smithi146 ceph-mon[103278]: pgmap v2667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:49 smithi078 ceph-mon[91136]: pgmap v2667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:49 smithi078 ceph-mon[99959]: pgmap v2667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:50.469Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:50.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:50.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:50.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:48:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:48:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:51 smithi146 ceph-mon[103278]: pgmap v2668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:51 smithi078 ceph-mon[91136]: pgmap v2668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:51 smithi078 ceph-mon[99959]: pgmap v2668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:54.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:53 smithi146 ceph-mon[103278]: pgmap v2669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:53 smithi078 ceph-mon[91136]: pgmap v2669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:53 smithi078 ceph-mon[99959]: pgmap v2669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:53.905Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:53.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:48:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:48:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:48:53.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:48:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:48:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:55 smithi146 ceph-mon[103278]: pgmap v2670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:55 smithi078 ceph-mon[91136]: pgmap v2670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:55 smithi078 ceph-mon[99959]: pgmap v2670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:56 smithi146 ceph-mon[103278]: pgmap v2671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:56 smithi078 ceph-mon[91136]: pgmap v2671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:56 smithi078 ceph-mon[99959]: pgmap v2671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:48:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:48:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:48:59 smithi146 ceph-mon[103278]: pgmap v2672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:48:59 smithi078 ceph-mon[91136]: pgmap v2672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:48:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:48:59 smithi078 ceph-mon[99959]: pgmap v2672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:00.470Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:00.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:00.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:00.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:49:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:49:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:01 smithi146 ceph-mon[103278]: pgmap v2673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[91136]: pgmap v2673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:01 smithi078 ceph-mon[99959]: pgmap v2673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:03.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:03 smithi078 ceph-mon[91136]: pgmap v2674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:03.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:03 smithi078 ceph-mon[99959]: pgmap v2674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:03 smithi146 ceph-mon[103278]: pgmap v2674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:04.050 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:49:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:03.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:03.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:03.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:04.603 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:49:04.603 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 99m - - 2024-09-20T21:49:04.603 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 99m - - 2024-09-20T21:49:04.603 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (93m) 4s ago 94m 31.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (94m) 20s ago 94m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (87m) 4s ago 92m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (86m) 20s ago 98m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (87m) 4s ago 100m 560M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (86m) 4s ago 100m 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (84m) 20s ago 99m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (85m) 4s ago 99m 132M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (94m) 4s ago 94m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (94m) 20s ago 94m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (83m) 4s ago 98m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (82m) 4s ago 98m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (82m) 4s ago 97m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (81m) 4s ago 97m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (81m) 20s ago 97m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:49:04.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (80m) 20s ago 96m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (79m) 20s ago 96m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (78m) 20s ago 95m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (87m) 20s ago 95m 74.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (78m) 4s ago 93m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (78m) 20s ago 93m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (78m) 4s ago 92m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:49:04.605 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (77m) 20s ago 92m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:49:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:04.914 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:49:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:49:04.915 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:49:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:05.184 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:49:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:49:05.185 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:49:05.492 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:49:05.492 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:49:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:49:05.493 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:49:05.493 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: from='client.70010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: from='client.70016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: from='client.70022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: pgmap v2675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/551944949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: from='client.50115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3727069491' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: from='client.70010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: from='client.70016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: from='client.70022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: pgmap v2675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/551944949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: from='client.50115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3727069491' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: from='client.70010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: from='client.70016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: from='client.70022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: pgmap v2675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/551944949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:05.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: from='client.50115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:05.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3727069491' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:06 smithi146 ceph-mon[103278]: pgmap v2676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:06 smithi078 ceph-mon[91136]: pgmap v2676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:06 smithi078 ceph-mon[99959]: pgmap v2676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:09 smithi146 ceph-mon[103278]: pgmap v2677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:09 smithi078 ceph-mon[91136]: pgmap v2677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:09 smithi078 ceph-mon[99959]: pgmap v2677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:49:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:49:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:10.471Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:10.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:10.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:11 smithi146 ceph-mon[103278]: pgmap v2678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:11 smithi078 ceph-mon[91136]: pgmap v2678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:11 smithi078 ceph-mon[99959]: pgmap v2678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:12 smithi146 ceph-mon[103278]: pgmap v2679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:12 smithi078 ceph-mon[91136]: pgmap v2679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:12 smithi078 ceph-mon[99959]: pgmap v2679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:13.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:13.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:13.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:14.794 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:14 smithi146 ceph-mon[103278]: pgmap v2680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:14 smithi078 ceph-mon[91136]: pgmap v2680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:14 smithi078 ceph-mon[99959]: pgmap v2680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:17 smithi146 ceph-mon[103278]: pgmap v2681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:17 smithi078 ceph-mon[91136]: pgmap v2681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:17 smithi078 ceph-mon[99959]: pgmap v2681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:18 smithi146 ceph-mon[103278]: pgmap v2682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:18 smithi078 ceph-mon[99959]: pgmap v2682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:18 smithi078 ceph-mon[91136]: pgmap v2682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:20.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:20.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:20.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:20.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:49:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:49:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:21 smithi146 ceph-mon[103278]: pgmap v2683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[91136]: pgmap v2683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:21 smithi078 ceph-mon[99959]: pgmap v2683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:23.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:23 smithi078 ceph-mon[91136]: pgmap v2684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:23.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:23 smithi078 ceph-mon[99959]: pgmap v2684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:23 smithi146 ceph-mon[103278]: pgmap v2684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:23.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:23.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:23.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:24 smithi078 ceph-mon[91136]: pgmap v2685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:24 smithi078 ceph-mon[99959]: pgmap v2685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:24 smithi146 ceph-mon[103278]: pgmap v2685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:26 smithi146 ceph-mon[103278]: pgmap v2686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:26 smithi078 ceph-mon[99959]: pgmap v2686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:26 smithi078 ceph-mon[91136]: pgmap v2686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:29 smithi146 ceph-mon[103278]: pgmap v2687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:29 smithi078 ceph-mon[91136]: pgmap v2687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:29 smithi078 ceph-mon[99959]: pgmap v2687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:30.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:30 smithi078 ceph-mon[99959]: pgmap v2688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:30.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:30 smithi078 ceph-mon[91136]: pgmap v2688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:30.773 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:49:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:49:30.773 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:30.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:30.773 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:30.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:30.773 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:30.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:30 smithi146 ceph-mon[103278]: pgmap v2688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:32 smithi146 ceph-mon[103278]: pgmap v2689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:32 smithi078 ceph-mon[99959]: pgmap v2689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:32 smithi078 ceph-mon[91136]: pgmap v2689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:33.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:33.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:33.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:34.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:34 smithi146 ceph-mon[103278]: pgmap v2690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:34 smithi078 ceph-mon[99959]: pgmap v2690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:34 smithi078 ceph-mon[91136]: pgmap v2690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:35.785 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 100m - - 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 99m - - 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (94m) 15s ago 95m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (95m) 11s ago 95m 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (87m) 15s ago 93m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (86m) 11s ago 99m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (88m) 15s ago 101m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (86m) 15s ago 101m 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:49:36.313 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (85m) 11s ago 99m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (85m) 15s ago 99m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (94m) 15s ago 94m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (94m) 11s ago 94m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (84m) 15s ago 99m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (83m) 15s ago 98m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (82m) 15s ago 98m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (82m) 15s ago 97m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (81m) 11s ago 97m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (80m) 11s ago 97m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (79m) 11s ago 96m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (79m) 11s ago 96m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (87m) 11s ago 95m 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (78m) 15s ago 93m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:49:36.314 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (78m) 11s ago 93m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:49:36.315 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (78m) 15s ago 93m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:49:36.315 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (78m) 11s ago 93m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:49:36.623 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:49:36.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:49:36.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:49:36.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:36.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:49:36.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:49:36.895 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:49:36.895 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:49:36.895 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:49:36.895 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:49:36.896 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:49:36.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[99959]: from='client.50127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[99959]: from='client.50133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[99959]: from='client.70058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[99959]: pgmap v2691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1925006057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[91136]: from='client.50127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[91136]: from='client.50133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[91136]: from='client.70058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[91136]: pgmap v2691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:36.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1925006057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:36 smithi146 ceph-mon[103278]: from='client.50127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:36 smithi146 ceph-mon[103278]: from='client.50133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:36 smithi146 ceph-mon[103278]: from='client.70058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:36 smithi146 ceph-mon[103278]: pgmap v2691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1925006057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:49:37.206 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:49:37.206 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:49:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:49:37.206 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:49:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:49:37.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:37 smithi146 ceph-mon[103278]: from='client.50151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:37.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1033479773' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:37 smithi078 ceph-mon[91136]: from='client.50151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1033479773' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:37 smithi078 ceph-mon[99959]: from='client.50151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:49:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1033479773' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:49:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:38 smithi146 ceph-mon[103278]: pgmap v2692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:38 smithi078 ceph-mon[91136]: pgmap v2692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:38 smithi078 ceph-mon[99959]: pgmap v2692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:49:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:49:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:40.473Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:40.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:40.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[91136]: pgmap v2693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:41 smithi078 ceph-mon[99959]: pgmap v2693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:41 smithi146 ceph-mon[103278]: pgmap v2693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:42 smithi146 ceph-mon[103278]: pgmap v2694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:43.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:42 smithi078 ceph-mon[91136]: pgmap v2694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:43.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:42 smithi078 ceph-mon[99959]: pgmap v2694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:43.911 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:43.911 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:43.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:43.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:44.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:43.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:44.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:43.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:44.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:43.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:44 smithi146 ceph-mon[103278]: pgmap v2695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:44 smithi078 ceph-mon[91136]: pgmap v2695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:44 smithi078 ceph-mon[99959]: pgmap v2695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:49:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:46 smithi146 ceph-mon[103278]: pgmap v2696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:46 smithi078 ceph-mon[91136]: pgmap v2696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:46 smithi078 ceph-mon[99959]: pgmap v2696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:48 smithi146 ceph-mon[103278]: pgmap v2697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:48 smithi078 ceph-mon[91136]: pgmap v2697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:48 smithi078 ceph-mon[99959]: pgmap v2697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:50 smithi078 ceph-mon[91136]: pgmap v2698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:50 smithi078 ceph-mon[99959]: pgmap v2698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:49:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:49:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:50.474Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:50.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:50.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:50 smithi146 ceph-mon[103278]: pgmap v2698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:52 smithi146 ceph-mon[103278]: pgmap v2699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:52 smithi078 ceph-mon[91136]: pgmap v2699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:52 smithi078 ceph-mon[99959]: pgmap v2699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:49:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:53.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:53.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:49:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:49:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:49:53.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:49:54.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:54 smithi146 ceph-mon[103278]: pgmap v2700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:54 smithi078 ceph-mon[99959]: pgmap v2700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:54 smithi078 ceph-mon[91136]: pgmap v2700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:56 smithi146 ceph-mon[103278]: pgmap v2701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:56 smithi078 ceph-mon[91136]: pgmap v2701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:56 smithi078 ceph-mon[99959]: pgmap v2701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:49:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:49:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:49:59 smithi146 ceph-mon[103278]: pgmap v2702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:49:59 smithi078 ceph-mon[91136]: pgmap v2702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:49:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:49:59 smithi078 ceph-mon[99959]: pgmap v2702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T21:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:50:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:50:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:00.475Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:00.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:00.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:01 smithi146 ceph-mon[103278]: pgmap v2703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[91136]: pgmap v2703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:01 smithi078 ceph-mon[99959]: pgmap v2703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:02 smithi146 ceph-mon[103278]: pgmap v2704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:02 smithi078 ceph-mon[91136]: pgmap v2704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:02 smithi078 ceph-mon[99959]: pgmap v2704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:03.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:03.911 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:03.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:03.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:03.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:04.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:04 smithi078 ceph-mon[91136]: pgmap v2705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:04.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:04 smithi078 ceph-mon[99959]: pgmap v2705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:04 smithi146 ceph-mon[103278]: pgmap v2705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:07 smithi078 ceph-mon[91136]: pgmap v2706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:07 smithi078 ceph-mon[99959]: pgmap v2706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:07.506 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:50:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:07 smithi146 ceph-mon[103278]: pgmap v2706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:08.030 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:50:08.030 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 100m - - 2024-09-20T21:50:08.030 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 100m - - 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (94m) 7s ago 95m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (95m) 3s ago 95m 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (88m) 7s ago 93m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (87m) 3s ago 99m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (88m) 7s ago 101m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (87m) 7s ago 101m 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (85m) 3s ago 100m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (86m) 7s ago 100m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (95m) 7s ago 95m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (95m) 3s ago 95m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (84m) 7s ago 99m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (83m) 7s ago 99m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (83m) 7s ago 98m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (82m) 7s ago 98m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:50:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (82m) 3s ago 98m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (81m) 3s ago 97m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (80m) 3s ago 97m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (79m) 3s ago 96m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (88m) 3s ago 96m 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (79m) 7s ago 94m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (79m) 3s ago 94m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (79m) 7s ago 94m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (79m) 3s ago 93m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:50:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:08 smithi078 ceph-mon[91136]: from='client.70082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:08 smithi078 ceph-mon[99959]: from='client.70082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:50:08.339 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:50:08.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:50:08.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:50:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:50:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:08 smithi146 ceph-mon[103278]: from='client.70082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:08.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:50:08.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:50:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:08.918 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:08.918 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[91136]: from='client.70088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[91136]: from='client.50175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3209026261' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[91136]: from='client.50187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[91136]: pgmap v2707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/161356281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[99959]: from='client.70088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[99959]: from='client.50175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3209026261' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[99959]: from='client.50187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[99959]: pgmap v2707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:09.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/161356281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:09 smithi146 ceph-mon[103278]: from='client.70088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:09 smithi146 ceph-mon[103278]: from='client.50175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3209026261' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:09 smithi146 ceph-mon[103278]: from='client.50187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:09 smithi146 ceph-mon[103278]: pgmap v2707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/161356281' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:10 smithi078 ceph-mon[91136]: pgmap v2708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:10 smithi078 ceph-mon[99959]: pgmap v2708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:50:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:50:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:10.476Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:10.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:10.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:10 smithi146 ceph-mon[103278]: pgmap v2708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:12 smithi146 ceph-mon[103278]: pgmap v2709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:12 smithi078 ceph-mon[91136]: pgmap v2709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:12 smithi078 ceph-mon[99959]: pgmap v2709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:13.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:13.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:13.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:14.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:14 smithi146 ceph-mon[103278]: pgmap v2710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:14 smithi078 ceph-mon[91136]: pgmap v2710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:14 smithi078 ceph-mon[99959]: pgmap v2710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:16 smithi146 ceph-mon[103278]: pgmap v2711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:16 smithi078 ceph-mon[91136]: pgmap v2711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:16 smithi078 ceph-mon[99959]: pgmap v2711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:18 smithi146 ceph-mon[103278]: pgmap v2712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:18 smithi078 ceph-mon[91136]: pgmap v2712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:18 smithi078 ceph-mon[99959]: pgmap v2712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:20.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:20 smithi078 ceph-mon[99959]: pgmap v2713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:20.745 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:50:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:50:20.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:20 smithi078 ceph-mon[91136]: pgmap v2713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:20.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:20.476Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:20.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:20.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:20.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:20.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:20 smithi146 ceph-mon[103278]: pgmap v2713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:22 smithi146 ceph-mon[103278]: pgmap v2714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:22 smithi078 ceph-mon[99959]: pgmap v2714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:22 smithi078 ceph-mon[91136]: pgmap v2714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:23.911Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:23.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:23.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:24 smithi146 ceph-mon[103278]: pgmap v2715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:24 smithi078 ceph-mon[91136]: pgmap v2715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:24 smithi078 ceph-mon[99959]: pgmap v2715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:26.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:27 smithi146 ceph-mon[103278]: pgmap v2716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:27 smithi078 ceph-mon[91136]: pgmap v2716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:27 smithi078 ceph-mon[99959]: pgmap v2716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:29 smithi146 ceph-mon[103278]: pgmap v2717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:29 smithi078 ceph-mon[91136]: pgmap v2717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:29 smithi078 ceph-mon[99959]: pgmap v2717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:30.730 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:50:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:50:30.731 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:30.477Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:30.731 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:30.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:30.731 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:30.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:30 smithi146 ceph-mon[103278]: pgmap v2718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:30 smithi078 ceph-mon[99959]: pgmap v2718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:30 smithi078 ceph-mon[91136]: pgmap v2718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:32 smithi146 ceph-mon[103278]: pgmap v2719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:32 smithi078 ceph-mon[91136]: pgmap v2719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:32 smithi078 ceph-mon[99959]: pgmap v2719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:33.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:33.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:33.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:34 smithi146 ceph-mon[103278]: pgmap v2720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:34 smithi078 ceph-mon[91136]: pgmap v2720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:34 smithi078 ceph-mon[99959]: pgmap v2720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:36.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:36 smithi146 ceph-mon[103278]: pgmap v2721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:36 smithi078 ceph-mon[91136]: pgmap v2721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:36 smithi078 ceph-mon[99959]: pgmap v2721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:38 smithi146 ceph-mon[103278]: pgmap v2722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:38 smithi078 ceph-mon[99959]: pgmap v2722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:38 smithi078 ceph-mon[91136]: pgmap v2722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:39.213 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 101m - - 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 101m - - 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (95m) 19s ago 96m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (96m) 14s ago 96m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (88m) 19s ago 94m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (87m) 14s ago 100m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:50:39.741 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (89m) 19s ago 102m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (87m) 19s ago 102m 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (86m) 14s ago 100m 124M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (86m) 19s ago 100m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (96m) 19s ago 95m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (95m) 14s ago 95m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (85m) 19s ago 100m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (84m) 19s ago 99m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (83m) 19s ago 99m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (83m) 19s ago 98m 71.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (82m) 14s ago 98m 77.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (81m) 14s ago 98m 63.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (81m) 14s ago 97m 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (80m) 14s ago 97m 63.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:50:39.742 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (89m) 14s ago 96m 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:50:39.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (79m) 19s ago 94m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:50:39.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (79m) 14s ago 94m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:50:39.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (79m) 19s ago 94m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:50:39.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (79m) 14s ago 94m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:50:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:39 smithi146 ceph-mon[103278]: from='client.70118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:39 smithi146 ceph-mon[103278]: from='client.50205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:39 smithi078 ceph-mon[91136]: from='client.70118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:39 smithi078 ceph-mon[91136]: from='client.50205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:39 smithi078 ceph-mon[99959]: from='client.70118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:39 smithi078 ceph-mon[99959]: from='client.50205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.051 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:50:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:50:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:50:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:50:40.052 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:50:40.319 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:50:40.320 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:50:40.320 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:50:40.320 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:50:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:40.478Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:40.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:40.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:40.631 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:50:40.631 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:40.631 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:50:40.631 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:40.631 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:50:40.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:50:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:50:40.861 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[99959]: from='client.70130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/546530838' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[99959]: from='client.70142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2129926271' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[99959]: pgmap v2723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[91136]: from='client.70130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/546530838' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[91136]: from='client.70142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2129926271' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:40.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:40 smithi078 ceph-mon[91136]: pgmap v2723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:40 smithi146 ceph-mon[103278]: from='client.70130 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/546530838' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:50:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:40 smithi146 ceph-mon[103278]: from='client.70142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:50:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2129926271' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:50:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:40 smithi146 ceph-mon[103278]: pgmap v2723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:42.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:43 smithi146 ceph-mon[103278]: pgmap v2724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-mon[91136]: pgmap v2724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-mon[99959]: pgmap v2724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:43.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:43.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:43.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:44.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:44.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:45.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:45 smithi146 ceph-mon[103278]: pgmap v2725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:45.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[91136]: pgmap v2725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[99959]: pgmap v2725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:45.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:50:46.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:46 smithi078 ceph-mon[91136]: pgmap v2726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:46.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:46 smithi078 ceph-mon[99959]: pgmap v2726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:46 smithi146 ceph-mon[103278]: pgmap v2726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T21:50:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:48 smithi146 ceph-mon[103278]: pgmap v2727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:48 smithi078 ceph-mon[91136]: pgmap v2727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:48 smithi078 ceph-mon[99959]: pgmap v2727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:50:50.731 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:50:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:50:50.732 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:50.479Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:50.732 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:50.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:50.732 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:50.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:50 smithi146 ceph-mon[103278]: pgmap v2728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:50 smithi078 ceph-mon[91136]: pgmap v2728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:50 smithi078 ceph-mon[99959]: pgmap v2728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:52 smithi146 ceph-mon[103278]: pgmap v2729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:52 smithi078 ceph-mon[91136]: pgmap v2729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:52 smithi078 ceph-mon[99959]: pgmap v2729: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:50:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:50:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:53.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:53.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:50:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:50:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:50:53.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:50:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:54 smithi146 ceph-mon[103278]: pgmap v2730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:50:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:54 smithi078 ceph-mon[91136]: pgmap v2730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:50:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:54 smithi078 ceph-mon[99959]: pgmap v2730: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T21:50:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:56 smithi146 ceph-mon[103278]: pgmap v2731: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:50:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:56 smithi078 ceph-mon[91136]: pgmap v2731: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:50:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:56 smithi078 ceph-mon[99959]: pgmap v2731: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T21:50:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:50:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:50:59 smithi146 ceph-mon[103278]: pgmap v2732: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:50:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:50:59 smithi078 ceph-mon[99959]: pgmap v2732: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:50:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:50:59 smithi078 ceph-mon[91136]: pgmap v2732: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T21:51:00.735 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:51:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:51:00.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:00.479Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:00.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:00.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:00.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:00.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:00.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:00 smithi078 ceph-mon[91136]: pgmap v2733: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:00.998 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:00 smithi078 ceph-mon[99959]: pgmap v2733: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:00 smithi146 ceph-mon[103278]: pgmap v2733: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:02 smithi146 ceph-mon[103278]: pgmap v2734: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:02 smithi078 ceph-mon[91136]: pgmap v2734: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:02 smithi078 ceph-mon[99959]: pgmap v2734: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:03.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:03.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:03.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:04 smithi146 ceph-mon[103278]: pgmap v2735: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:51:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:04 smithi078 ceph-mon[91136]: pgmap v2735: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:51:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:04 smithi078 ceph-mon[99959]: pgmap v2735: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:51:06.633 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.633 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.634 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:07 smithi078 ceph-mon[91136]: pgmap v2736: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T21:51:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:07 smithi078 ceph-mon[99959]: pgmap v2736: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T21:51:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:07 smithi146 ceph-mon[103278]: pgmap v2736: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T21:51:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:08 smithi146 ceph-mon[103278]: pgmap v2737: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:51:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:08 smithi078 ceph-mon[91136]: pgmap v2737: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:51:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:08 smithi078 ceph-mon[99959]: pgmap v2737: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T21:51:10.740 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:51:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:51:10.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:10.480Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:10.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:10.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:10.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:10.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:10.945 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:51:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:10 smithi146 ceph-mon[103278]: pgmap v2738: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:51:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:10 smithi078 ceph-mon[99959]: pgmap v2738: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:51:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:10 smithi078 ceph-mon[91136]: pgmap v2738: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 101m - - 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 101m - - 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (95m) 10s ago 97m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (96m) 6s ago 96m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (89m) 10s ago 94m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (88m) 6s ago 100m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (90m) 10s ago 102m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (88m) 10s ago 102m 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:51:11.471 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (87m) 6s ago 101m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (87m) 10s ago 101m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (96m) 10s ago 96m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (96m) 6s ago 96m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (86m) 10s ago 100m 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (85m) 10s ago 100m 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (84m) 10s ago 99m 63.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (83m) 10s ago 99m 71.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (83m) 6s ago 99m 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (82m) 6s ago 98m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (81m) 6s ago 98m 68.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (80m) 6s ago 97m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (89m) 6s ago 97m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (80m) 10s ago 95m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (80m) 6s ago 95m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:51:11.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (80m) 10s ago 95m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:51:11.473 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (80m) 6s ago 95m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:51:11.785 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:51:11.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:11.787 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:51:11.787 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:51:11.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:51:11.787 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:51:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:11 smithi146 ceph-mon[103278]: from='client.50235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:11 smithi146 ceph-mon[103278]: from='client.50241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:11 smithi146 ceph-mon[103278]: from='client.70166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:11 smithi078 ceph-mon[99959]: from='client.50235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:11 smithi078 ceph-mon[99959]: from='client.50241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:11 smithi078 ceph-mon[99959]: from='client.70166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:11 smithi078 ceph-mon[91136]: from='client.50235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:11 smithi078 ceph-mon[91136]: from='client.50241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:11 smithi078 ceph-mon[91136]: from='client.70166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:51:12.078 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:51:12.079 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:51:12.403 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:51:12.403 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:12.403 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:12.403 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:12.403 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2736463572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:12 smithi146 ceph-mon[103278]: from='client.70178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1255372743' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:12 smithi146 ceph-mon[103278]: pgmap v2739: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T21:51:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2736463572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[91136]: from='client.70178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1255372743' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[91136]: pgmap v2739: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T21:51:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2736463572' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[99959]: from='client.70178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1255372743' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:12 smithi078 ceph-mon[99959]: pgmap v2739: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T21:51:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:13.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:13.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:13.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:14 smithi146 ceph-mon[103278]: pgmap v2740: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:14 smithi078 ceph-mon[91136]: pgmap v2740: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:14 smithi078 ceph-mon[99959]: pgmap v2740: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T21:51:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:16 smithi146 ceph-mon[103278]: pgmap v2741: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:51:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:16 smithi078 ceph-mon[99959]: pgmap v2741: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:51:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:16 smithi078 ceph-mon[91136]: pgmap v2741: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T21:51:18.882 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:18 smithi146 ceph-mon[103278]: pgmap v2742: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:51:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:18 smithi078 ceph-mon[91136]: pgmap v2742: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:51:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:18 smithi078 ceph-mon[99959]: pgmap v2742: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T21:51:20.735 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:51:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:51:20.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:20.481Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:20.736 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:20.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:20.736 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:20.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:20 smithi146 ceph-mon[103278]: pgmap v2743: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:51:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:20 smithi078 ceph-mon[91136]: pgmap v2743: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:51:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:20 smithi078 ceph-mon[99959]: pgmap v2743: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T21:51:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:23.397 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:23 smithi078 ceph-mon[91136]: pgmap v2744: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T21:51:23.397 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:23 smithi078 ceph-mon[99959]: pgmap v2744: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T21:51:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:23 smithi146 ceph-mon[103278]: pgmap v2744: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T21:51:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:23.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:23.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:23.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:25 smithi146 ceph-mon[103278]: pgmap v2745: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:51:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:25 smithi078 ceph-mon[91136]: pgmap v2745: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:51:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:25 smithi078 ceph-mon[99959]: pgmap v2745: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T21:51:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:26.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:26.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:26.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:27.550 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:27 smithi078 ceph-mon[99959]: pgmap v2746: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T21:51:27.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:27 smithi078 ceph-mon[91136]: pgmap v2746: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T21:51:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:27 smithi146 ceph-mon[103278]: pgmap v2746: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T21:51:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:29 smithi146 ceph-mon[103278]: pgmap v2747: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:29 smithi078 ceph-mon[91136]: pgmap v2747: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:29 smithi078 ceph-mon[99959]: pgmap v2747: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:30.737 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:51:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:51:30.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:30.481Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:30.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:30.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:30.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:30.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:30 smithi146 ceph-mon[103278]: pgmap v2748: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:30 smithi078 ceph-mon[91136]: pgmap v2748: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:30 smithi078 ceph-mon[99959]: pgmap v2748: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:32 smithi146 ceph-mon[103278]: pgmap v2749: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:32 smithi078 ceph-mon[91136]: pgmap v2749: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:32 smithi078 ceph-mon[99959]: pgmap v2749: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:33.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:33.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:33.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:34 smithi146 ceph-mon[103278]: pgmap v2750: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:34 smithi078 ceph-mon[91136]: pgmap v2750: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:34 smithi078 ceph-mon[99959]: pgmap v2750: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:36 smithi146 ceph-mon[103278]: pgmap v2751: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:36 smithi078 ceph-mon[91136]: pgmap v2751: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:36 smithi078 ceph-mon[99959]: pgmap v2751: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:38 smithi146 ceph-mon[103278]: pgmap v2752: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:38 smithi078 ceph-mon[91136]: pgmap v2752: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:38 smithi078 ceph-mon[99959]: pgmap v2752: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:40.738 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:51:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:51:40.739 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:40.482Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:40.739 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:40.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:40.739 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:40.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:40.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:40 smithi078 ceph-mon[91136]: pgmap v2753: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:40.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:40 smithi078 ceph-mon[99959]: pgmap v2753: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:40 smithi146 ceph-mon[103278]: pgmap v2753: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:42.690 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:51:43.213 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:51:43.213 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 102m - - 2024-09-20T21:51:43.213 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 102m - - 2024-09-20T21:51:43.213 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (96m) 1s ago 97m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:51:43.213 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (97m) 17s ago 97m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:51:43.213 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (89m) 1s ago 95m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (88m) 17s ago 101m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (90m) 1s ago 103m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (88m) 1s ago 103m 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (87m) 17s ago 101m 125M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (87m) 1s ago 101m 132M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (97m) 1s ago 97m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (96m) 17s ago 96m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (86m) 1s ago 101m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (85m) 1s ago 100m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (84m) 1s ago 100m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (84m) 1s ago 100m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (83m) 17s ago 99m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (82m) 17s ago 99m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:51:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (82m) 17s ago 98m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:51:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (81m) 17s ago 98m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:51:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (90m) 17s ago 97m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:51:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (80m) 1s ago 95m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:51:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (80m) 17s ago 95m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:51:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (80m) 1s ago 95m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:51:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (80m) 17s ago 95m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:51:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:43 smithi146 ceph-mon[103278]: pgmap v2754: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:43 smithi146 ceph-mon[103278]: from='client.70190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:43.525 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:51:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:51:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:51:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:51:43.526 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:51:43.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-mon[99959]: pgmap v2754: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:43.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-mon[99959]: from='client.70190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:43.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:43.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-mon[91136]: pgmap v2754: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:43.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-mon[91136]: from='client.70190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:43.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:51:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:51:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:51:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:51:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:51:43.796 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:51:44.105 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:51:44.105 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:51:44.105 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:51:44.105 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:43.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:44.105 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:43.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:44.106 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:43.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:44 smithi146 ceph-mon[103278]: from='client.70196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:44 smithi146 ceph-mon[103278]: from='client.70202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3352756325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1266067130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[91136]: from='client.70196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[91136]: from='client.70202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3352756325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1266067130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[99959]: from='client.70196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[99959]: from='client.70202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3352756325' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:51:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1266067130' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:51:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:45 smithi146 ceph-mon[103278]: from='client.70214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:45 smithi146 ceph-mon[103278]: pgmap v2755: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:45 smithi078 ceph-mon[91136]: from='client.70214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:45 smithi078 ceph-mon[91136]: pgmap v2755: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:45 smithi078 ceph-mon[99959]: from='client.70214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:51:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:45 smithi078 ceph-mon[99959]: pgmap v2755: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:46.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:46.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:46.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:46.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:46.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:46.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:51:47.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:47 smithi078 ceph-mon[91136]: pgmap v2756: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:47.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:47 smithi078 ceph-mon[99959]: pgmap v2756: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:47 smithi146 ceph-mon[103278]: pgmap v2756: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:48.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:48 smithi078 ceph-mon[91136]: pgmap v2757: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:48.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:48 smithi078 ceph-mon[99959]: pgmap v2757: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:48 smithi146 ceph-mon[103278]: pgmap v2757: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:50.740 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:51:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:51:50.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:50.482Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:50.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:50.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:50.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:50.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:50 smithi146 ceph-mon[103278]: pgmap v2758: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:50 smithi078 ceph-mon[91136]: pgmap v2758: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:50 smithi078 ceph-mon[99959]: pgmap v2758: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:52 smithi146 ceph-mon[103278]: pgmap v2759: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:52 smithi078 ceph-mon[91136]: pgmap v2759: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:52 smithi078 ceph-mon[99959]: pgmap v2759: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:51:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:53.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:53.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:51:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:51:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:51:53.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:51:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:54 smithi146 ceph-mon[103278]: pgmap v2760: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:54 smithi078 ceph-mon[91136]: pgmap v2760: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:54 smithi078 ceph-mon[99959]: pgmap v2760: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:56 smithi146 ceph-mon[103278]: pgmap v2761: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:56 smithi078 ceph-mon[91136]: pgmap v2761: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:56 smithi078 ceph-mon[99959]: pgmap v2761: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:51:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:51:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:51:59 smithi146 ceph-mon[103278]: pgmap v2762: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:51:59 smithi078 ceph-mon[91136]: pgmap v2762: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:51:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:51:59 smithi078 ceph-mon[99959]: pgmap v2762: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:00.743 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:52:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:52:00.743 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:00.483Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:00.744 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:00.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:00.744 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:00.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:00 smithi146 ceph-mon[103278]: pgmap v2763: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:00 smithi078 ceph-mon[91136]: pgmap v2763: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:00 smithi078 ceph-mon[99959]: pgmap v2763: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:03 smithi146 ceph-mon[103278]: pgmap v2764: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:03 smithi078 ceph-mon[91136]: pgmap v2764: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:03 smithi078 ceph-mon[99959]: pgmap v2764: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:03.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:03.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:03.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:05 smithi146 ceph-mon[103278]: pgmap v2765: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:05 smithi078 ceph-mon[91136]: pgmap v2765: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:05 smithi078 ceph-mon[99959]: pgmap v2765: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[91136]: pgmap v2766: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:06 smithi078 ceph-mon[99959]: pgmap v2766: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:06 smithi146 ceph-mon[103278]: pgmap v2766: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:08 smithi146 ceph-mon[103278]: pgmap v2767: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:08 smithi078 ceph-mon[99959]: pgmap v2767: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:08 smithi078 ceph-mon[91136]: pgmap v2767: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:10.743 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:52:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T21:52:10.743 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:10.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:10.744 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:10.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:10.744 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:10.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:10 smithi146 ceph-mon[103278]: pgmap v2768: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:10 smithi078 ceph-mon[91136]: pgmap v2768: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:10 smithi078 ceph-mon[99959]: pgmap v2768: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:12 smithi146 ceph-mon[103278]: pgmap v2769: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:12 smithi078 ceph-mon[91136]: pgmap v2769: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:12 smithi078 ceph-mon[99959]: pgmap v2769: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:13.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:13.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:13.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:14.397 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 102m - - 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 102m - - 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (97m) 13s ago 98m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (97m) 9s ago 97m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (90m) 13s ago 95m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (89m) 9s ago 102m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:52:14.922 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (91m) 13s ago 103m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (89m) 13s ago 103m 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (88m) 9s ago 102m 127M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (88m) 13s ago 102m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (97m) 13s ago 97m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (97m) 9s ago 97m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (87m) 13s ago 101m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (86m) 13s ago 101m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (85m) 13s ago 100m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (84m) 13s ago 100m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (84m) 9s ago 100m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (83m) 9s ago 99m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (82m) 9s ago 99m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (81m) 9s ago 98m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:52:14.923 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (90m) 9s ago 98m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:52:14.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (81m) 13s ago 96m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:52:14.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (81m) 9s ago 96m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:52:14.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (81m) 13s ago 96m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:52:14.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (81m) 9s ago 96m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:52:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:14 smithi146 ceph-mon[103278]: from='client.50307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:14 smithi146 ceph-mon[103278]: from='client.50313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:14 smithi146 ceph-mon[103278]: pgmap v2770: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:14 smithi078 ceph-mon[91136]: from='client.50307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:14 smithi078 ceph-mon[91136]: from='client.50313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:14 smithi078 ceph-mon[91136]: pgmap v2770: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:14 smithi078 ceph-mon[99959]: from='client.50307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:14 smithi078 ceph-mon[99959]: from='client.50313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:14 smithi078 ceph-mon[99959]: pgmap v2770: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:52:15.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:52:15.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:52:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:52:15.503 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:52:15.503 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:52:15.503 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:52:15.503 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:52:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:15 smithi078 ceph-mon[91136]: from='client.70238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2795307525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:15 smithi078 ceph-mon[91136]: from='client.70250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:15 smithi078 ceph-mon[99959]: from='client.70238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2795307525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:15 smithi078 ceph-mon[99959]: from='client.70250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:15.812 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:52:15.813 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:15.813 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:15.813 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:15.813 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:15 smithi146 ceph-mon[103278]: from='client.70238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2795307525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:15 smithi146 ceph-mon[103278]: from='client.70250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1421508457' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:16 smithi146 ceph-mon[103278]: pgmap v2771: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1421508457' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:16 smithi078 ceph-mon[91136]: pgmap v2771: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1421508457' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:16 smithi078 ceph-mon[99959]: pgmap v2771: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:18 smithi146 ceph-mon[103278]: pgmap v2772: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:18 smithi078 ceph-mon[91136]: pgmap v2772: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:18 smithi078 ceph-mon[99959]: pgmap v2772: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:20.745 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:52:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:52:20.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:20.485Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:20.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:20.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:20.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:20.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:20 smithi146 ceph-mon[103278]: pgmap v2773: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:20 smithi078 ceph-mon[91136]: pgmap v2773: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:20 smithi078 ceph-mon[99959]: pgmap v2773: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:23 smithi078 ceph-mon[91136]: pgmap v2774: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:23 smithi078 ceph-mon[99959]: pgmap v2774: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:23 smithi146 ceph-mon[103278]: pgmap v2774: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:23.920Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:23.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:23.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:24.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:25 smithi078 ceph-mon[91136]: pgmap v2775: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:25 smithi078 ceph-mon[99959]: pgmap v2775: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:25.931 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:25 smithi146 ceph-mon[103278]: pgmap v2775: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:26 smithi146 ceph-mon[103278]: pgmap v2776: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[91136]: pgmap v2776: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:26 smithi078 ceph-mon[99959]: pgmap v2776: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:29 smithi146 ceph-mon[103278]: pgmap v2777: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:29 smithi078 ceph-mon[99959]: pgmap v2777: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:29 smithi078 ceph-mon[91136]: pgmap v2777: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:30.746 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:52:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:52:30.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:30.485Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:30.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:30.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:30.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:30.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:30 smithi146 ceph-mon[103278]: pgmap v2778: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:30 smithi078 ceph-mon[91136]: pgmap v2778: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:30 smithi078 ceph-mon[99959]: pgmap v2778: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:32 smithi146 ceph-mon[103278]: pgmap v2779: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:32 smithi078 ceph-mon[91136]: pgmap v2779: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:32 smithi078 ceph-mon[99959]: pgmap v2779: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:33.920Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:33.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:33.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:34 smithi146 ceph-mon[103278]: pgmap v2780: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:34 smithi078 ceph-mon[99959]: pgmap v2780: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:34 smithi078 ceph-mon[91136]: pgmap v2780: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:36 smithi146 ceph-mon[103278]: pgmap v2781: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:36 smithi078 ceph-mon[91136]: pgmap v2781: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:36 smithi078 ceph-mon[99959]: pgmap v2781: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:38 smithi146 ceph-mon[103278]: pgmap v2782: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:38 smithi078 ceph-mon[99959]: pgmap v2782: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:38 smithi078 ceph-mon[91136]: pgmap v2782: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:40.747 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:52:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:52:40.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:40.486Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:40.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:40.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:40.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:40.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:40.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:40 smithi146 ceph-mon[103278]: pgmap v2783: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:40 smithi078 ceph-mon[91136]: pgmap v2783: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:40 smithi078 ceph-mon[99959]: pgmap v2783: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-mon[91136]: pgmap v2784: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-mon[99959]: pgmap v2784: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:43 smithi146 ceph-mon[103278]: pgmap v2784: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:43.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:43.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:43.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:44.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:45 smithi146 ceph-mon[103278]: pgmap v2785: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:45 smithi078 ceph-mon[99959]: pgmap v2785: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:45 smithi078 ceph-mon[91136]: pgmap v2785: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:46.118 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:52:46.640 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 103m - - 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 103m - - 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (97m) 4s ago 98m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (98m) 0s ago 98m 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (90m) 4s ago 96m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (89m) 0s ago 102m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (91m) 4s ago 104m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (89m) 4s ago 104m 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (88m) 0s ago 103m 128M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (88m) 4s ago 102m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (98m) 4s ago 98m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (98m) 0s ago 98m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (87m) 4s ago 102m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (86m) 4s ago 101m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (86m) 4s ago 101m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (85m) 4s ago 101m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (84m) 0s ago 100m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (83m) 0s ago 100m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (83m) 0s ago 99m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (82m) 0s ago 99m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (91m) 0s ago 98m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (81m) 4s ago 96m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (81m) 0s ago 96m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (81m) 4s ago 96m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (81m) 0s ago 96m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:52:46.951 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:52:46.951 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:52:46.953 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:52:47.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='client.70262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='client.50349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='client.50355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: pgmap v2786: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3166767913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='client.70262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='client.50349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='client.50355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: pgmap v2786: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:47.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3166767913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='client.70262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='client.50349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='client.50355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: pgmap v2786: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3166767913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:52:47.533 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:52:47.533 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:47.533 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:52:47.534 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:47.534 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:52:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:48 smithi146 ceph-mon[103278]: from='client.70286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/386684893' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:48 smithi078 ceph-mon[99959]: from='client.70286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/386684893' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:48 smithi078 ceph-mon[91136]: from='client.70286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:52:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/386684893' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:52:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:49 smithi078 ceph-mon[91136]: pgmap v2787: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:49 smithi078 ceph-mon[99959]: pgmap v2787: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:49 smithi146 ceph-mon[103278]: pgmap v2787: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:50.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:50.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:50.528 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:50.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:50.528 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:50.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:50 smithi078 ceph-mon[91136]: pgmap v2788: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:52:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:52:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:50 smithi078 ceph-mon[99959]: pgmap v2788: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:50 smithi146 ceph-mon[103278]: pgmap v2788: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:52 smithi146 ceph-mon[103278]: pgmap v2789: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:52 smithi078 ceph-mon[91136]: pgmap v2789: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:52 smithi078 ceph-mon[99959]: pgmap v2789: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:52:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:53.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:53.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:52:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:52:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:52:53.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:52:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:54 smithi146 ceph-mon[103278]: pgmap v2790: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:54 smithi078 ceph-mon[91136]: pgmap v2790: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:54 smithi078 ceph-mon[99959]: pgmap v2790: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:56 smithi146 ceph-mon[103278]: pgmap v2791: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:56 smithi078 ceph-mon[91136]: pgmap v2791: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:56 smithi078 ceph-mon[99959]: pgmap v2791: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:52:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:52:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:52:59 smithi146 ceph-mon[103278]: pgmap v2792: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:52:59 smithi078 ceph-mon[91136]: pgmap v2792: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:52:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:52:59 smithi078 ceph-mon[99959]: pgmap v2792: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:00.751 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:53:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:53:00.751 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:00.488Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:00.752 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:00.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:00.752 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:00.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:00 smithi146 ceph-mon[103278]: pgmap v2793: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:00 smithi078 ceph-mon[99959]: pgmap v2793: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:00 smithi078 ceph-mon[91136]: pgmap v2793: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[99959]: pgmap v2794: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:02 smithi078 ceph-mon[91136]: pgmap v2794: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:02 smithi146 ceph-mon[103278]: pgmap v2794: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:03.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:03.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:03.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:04 smithi078 ceph-mon[99959]: pgmap v2795: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:04 smithi078 ceph-mon[91136]: pgmap v2795: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:04 smithi146 ceph-mon[103278]: pgmap v2795: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:07 smithi146 ceph-mon[103278]: pgmap v2796: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[91136]: pgmap v2796: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:07 smithi078 ceph-mon[99959]: pgmap v2796: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:08 smithi146 ceph-mon[103278]: pgmap v2797: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:08 smithi078 ceph-mon[91136]: pgmap v2797: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:08 smithi078 ceph-mon[99959]: pgmap v2797: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:10.489Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:10.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:10.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:10.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:10 smithi078 ceph-mon[91136]: pgmap v2798: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:10.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:10 smithi078 ceph-mon[99959]: pgmap v2798: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:10.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:53:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:53:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:10 smithi146 ceph-mon[103278]: pgmap v2798: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:12 smithi146 ceph-mon[103278]: pgmap v2799: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:12 smithi078 ceph-mon[91136]: pgmap v2799: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:12 smithi078 ceph-mon[99959]: pgmap v2799: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:13.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:13.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:13.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:14 smithi146 ceph-mon[103278]: pgmap v2800: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:14 smithi078 ceph-mon[91136]: pgmap v2800: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:14 smithi078 ceph-mon[99959]: pgmap v2800: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:16 smithi146 ceph-mon[103278]: pgmap v2801: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:16 smithi078 ceph-mon[91136]: pgmap v2801: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:16 smithi078 ceph-mon[99959]: pgmap v2801: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:17.833 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 103m - - 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 103m - - 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (98m) 16s ago 99m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (98m) 12s ago 98m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (91m) 16s ago 96m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (90m) 12s ago 103m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (92m) 16s ago 104m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (90m) 16s ago 104m 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (89m) 12s ago 103m 129M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:53:18.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (89m) 16s ago 103m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (98m) 16s ago 98m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (98m) 12s ago 98m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (88m) 16s ago 102m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (87m) 16s ago 102m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (86m) 16s ago 102m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (86m) 16s ago 101m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (85m) 12s ago 101m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (84m) 12s ago 100m 64.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (83m) 12s ago 100m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (82m) 12s ago 100m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (91m) 12s ago 99m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (82m) 16s ago 97m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (82m) 12s ago 97m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (82m) 16s ago 97m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:53:18.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (82m) 12s ago 97m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:53:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:53:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:53:18.668 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:53:18.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:53:18.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[91136]: from='client.50379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[91136]: from='client.50385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[91136]: from='client.50391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/369033329' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[91136]: pgmap v2802: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[99959]: from='client.50379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[99959]: from='client.50385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[99959]: from='client.50391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:18.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/369033329' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:18.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:18 smithi078 ceph-mon[99959]: pgmap v2802: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:18 smithi146 ceph-mon[103278]: from='client.50379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:18 smithi146 ceph-mon[103278]: from='client.50385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:18 smithi146 ceph-mon[103278]: from='client.50391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/369033329' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:18 smithi146 ceph-mon[103278]: pgmap v2802: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:19.241 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:53:19.242 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:19.242 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:20.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:19 smithi146 ceph-mon[103278]: from='client.70322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:20.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3937300704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:19 smithi078 ceph-mon[91136]: from='client.70322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3937300704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:19 smithi078 ceph-mon[99959]: from='client.70322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3937300704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:20.750 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:53:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:53:20.751 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:20.489Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:20.751 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:20.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:20.751 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:20.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:20 smithi146 ceph-mon[103278]: pgmap v2803: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:20 smithi078 ceph-mon[91136]: pgmap v2803: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:20 smithi078 ceph-mon[99959]: pgmap v2803: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:22 smithi146 ceph-mon[103278]: pgmap v2804: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[91136]: pgmap v2804: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:22 smithi078 ceph-mon[99959]: pgmap v2804: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:23.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:23.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:23.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:24 smithi146 ceph-mon[103278]: pgmap v2805: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:24 smithi078 ceph-mon[91136]: pgmap v2805: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:24 smithi078 ceph-mon[99959]: pgmap v2805: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:27 smithi146 ceph-mon[103278]: pgmap v2806: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[91136]: pgmap v2806: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:27 smithi078 ceph-mon[99959]: pgmap v2806: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:29 smithi146 ceph-mon[103278]: pgmap v2807: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:29 smithi078 ceph-mon[91136]: pgmap v2807: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:29 smithi078 ceph-mon[99959]: pgmap v2807: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:30.490Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:30.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:30.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:30.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:30.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:30 smithi078 ceph-mon[99959]: pgmap v2808: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:30.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:30 smithi078 ceph-mon[91136]: pgmap v2808: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:30.906 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:53:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:53:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:30 smithi146 ceph-mon[103278]: pgmap v2808: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:32 smithi146 ceph-mon[103278]: pgmap v2809: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:32 smithi078 ceph-mon[91136]: pgmap v2809: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:32 smithi078 ceph-mon[99959]: pgmap v2809: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:33.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:33.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:33.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:34 smithi146 ceph-mon[103278]: pgmap v2810: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:34 smithi078 ceph-mon[91136]: pgmap v2810: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:34 smithi078 ceph-mon[99959]: pgmap v2810: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:36 smithi146 ceph-mon[103278]: pgmap v2811: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:36 smithi078 ceph-mon[91136]: pgmap v2811: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:36 smithi078 ceph-mon[99959]: pgmap v2811: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:38 smithi146 ceph-mon[103278]: pgmap v2812: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:38 smithi078 ceph-mon[91136]: pgmap v2812: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:38 smithi078 ceph-mon[99959]: pgmap v2812: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:40.757 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:53:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:53:40.757 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:40.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:40.757 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:40.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:40.757 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:40.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:40 smithi146 ceph-mon[103278]: pgmap v2813: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:40 smithi078 ceph-mon[91136]: pgmap v2813: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:40 smithi078 ceph-mon[99959]: pgmap v2813: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:43.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:43 smithi146 ceph-mon[103278]: pgmap v2814: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[91136]: pgmap v2814: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-mon[99959]: pgmap v2814: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:43.925Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:43.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:43.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:45 smithi078 ceph-mon[99959]: pgmap v2815: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:45 smithi078 ceph-mon[91136]: pgmap v2815: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:45 smithi146 ceph-mon[103278]: pgmap v2815: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:47 smithi146 ceph-mon[103278]: pgmap v2816: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[91136]: pgmap v2816: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:53:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:47 smithi078 ceph-mon[99959]: pgmap v2816: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:48 smithi146 ceph-mon[103278]: pgmap v2817: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:48 smithi078 ceph-mon[99959]: pgmap v2817: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:48 smithi078 ceph-mon[91136]: pgmap v2817: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:49.539 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:53:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:49 smithi146 ceph-mon[103278]: from='client.70334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:49 smithi078 ceph-mon[99959]: from='client.70334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:49 smithi078 ceph-mon[91136]: from='client.70334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 104m - - 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 104m - - 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (98m) 8s ago 99m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (99m) 3s ago 99m 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (91m) 8s ago 97m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (90m) 3s ago 103m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (92m) 8s ago 105m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (90m) 8s ago 105m 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (89m) 3s ago 104m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (90m) 8s ago 103m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:53:50.066 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (99m) 8s ago 99m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (99m) 3s ago 99m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (88m) 8s ago 103m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (87m) 8s ago 102m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (87m) 8s ago 102m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (86m) 8s ago 102m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (86m) 3s ago 101m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (84m) 3s ago 101m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (84m) 3s ago 100m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (83m) 3s ago 100m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (92m) 3s ago 99m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (82m) 8s ago 97m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (82m) 3s ago 97m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (82m) 8s ago 97m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:53:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (82m) 3s ago 97m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:53:50.389 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:53:50.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:53:50.390 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:53:50.390 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:53:50.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:53:50.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:53:50.661 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:53:50.662 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:53:50.663 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:53:50.663 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:53:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:53:50.663 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:50.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:50.663 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:50.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:50.663 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:50.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:53:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:53:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:50.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:53:50.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[99959]: from='client.70340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[99959]: from='client.50427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/373072654' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:50.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[99959]: from='client.70358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[99959]: pgmap v2818: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:50.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[91136]: from='client.70340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[91136]: from='client.50427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/373072654' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:50.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[91136]: from='client.70358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:50.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:50 smithi078 ceph-mon[91136]: pgmap v2818: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:50 smithi146 ceph-mon[103278]: from='client.70340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:50 smithi146 ceph-mon[103278]: from='client.50427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/373072654' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:53:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:50 smithi146 ceph-mon[103278]: from='client.70358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:53:51.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:50 smithi146 ceph-mon[103278]: pgmap v2818: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:52.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/440220553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:52.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/440220553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/440220553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:53:53.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:52 smithi078 ceph-mon[91136]: pgmap v2819: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:53.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:52 smithi078 ceph-mon[99959]: pgmap v2819: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:52 smithi146 ceph-mon[103278]: pgmap v2819: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:53.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:53.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:54.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:53:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:53.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:53.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:53:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:53:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:53:53.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:53:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:54 smithi078 ceph-mon[91136]: pgmap v2820: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:54 smithi078 ceph-mon[99959]: pgmap v2820: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:54 smithi146 ceph-mon[103278]: pgmap v2820: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:56 smithi146 ceph-mon[103278]: pgmap v2821: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:56 smithi078 ceph-mon[99959]: pgmap v2821: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:56 smithi078 ceph-mon[91136]: pgmap v2821: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:53:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:53:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:53:59 smithi078 ceph-mon[91136]: pgmap v2822: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:53:59 smithi078 ceph-mon[99959]: pgmap v2822: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:53:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:53:59 smithi146 ceph-mon[103278]: pgmap v2822: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:00.763 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:54:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:54:00.763 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:00.492Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:00.763 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:00.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:00.763 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:00.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:00 smithi146 ceph-mon[103278]: pgmap v2823: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:00 smithi078 ceph-mon[99959]: pgmap v2823: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:00 smithi078 ceph-mon[91136]: pgmap v2823: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:03.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:03 smithi146 ceph-mon[103278]: pgmap v2824: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[91136]: pgmap v2824: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-mon[99959]: pgmap v2824: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:03.927Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:03.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:03.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:05 smithi146 ceph-mon[103278]: pgmap v2825: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:05 smithi078 ceph-mon[91136]: pgmap v2825: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:05 smithi078 ceph-mon[99959]: pgmap v2825: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:07 smithi146 ceph-mon[103278]: pgmap v2826: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[91136]: pgmap v2826: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:07 smithi078 ceph-mon[99959]: pgmap v2826: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:08 smithi078 ceph-mon[91136]: pgmap v2827: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:08 smithi078 ceph-mon[99959]: pgmap v2827: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:08 smithi146 ceph-mon[103278]: pgmap v2827: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:54:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T21:54:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:10.492Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:10.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:10.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:11 smithi146 ceph-mon[103278]: pgmap v2828: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:11 smithi078 ceph-mon[91136]: pgmap v2828: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:11 smithi078 ceph-mon[99959]: pgmap v2828: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-mon[91136]: pgmap v2829: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-mon[99959]: pgmap v2829: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:13.928Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:13.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:14.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:13.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:13 smithi146 ceph-mon[103278]: pgmap v2829: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:15 smithi078 ceph-mon[99959]: pgmap v2830: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:15 smithi078 ceph-mon[91136]: pgmap v2830: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:16.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:15 smithi146 ceph-mon[103278]: pgmap v2830: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:17 smithi078 ceph-mon[99959]: pgmap v2831: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:17 smithi078 ceph-mon[91136]: pgmap v2831: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:18.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:17 smithi146 ceph-mon[103278]: pgmap v2831: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:19 smithi078 ceph-mon[99959]: pgmap v2832: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:19 smithi078 ceph-mon[91136]: pgmap v2832: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:19 smithi146 ceph-mon[103278]: pgmap v2832: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:54:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:54:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:20.493Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:20.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:20.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:21.272 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:54:21.808 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:54:21.808 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 105m - - 2024-09-20T21:54:21.808 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 104m - - 2024-09-20T21:54:21.808 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (99m) 19s ago 100m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (99m) 15s ago 99m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (92m) 19s ago 97m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (91m) 15s ago 104m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (93m) 19s ago 105m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (91m) 19s ago 106m 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (90m) 15s ago 104m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (90m) 19s ago 104m 137M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (99m) 19s ago 99m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (99m) 15s ago 99m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (89m) 19s ago 103m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (88m) 19s ago 103m 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (87m) 19s ago 103m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (87m) 19s ago 102m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (86m) 15s ago 102m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:54:21.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (85m) 15s ago 101m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (84m) 15s ago 101m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (83m) 15s ago 101m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (92m) 15s ago 100m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (83m) 19s ago 98m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (83m) 15s ago 98m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (83m) 19s ago 98m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:54:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (83m) 15s ago 98m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:54:21.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:21 smithi146 ceph-mon[103278]: pgmap v2833: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:21.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:21 smithi146 ceph-mon[103278]: from='client.50451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:21.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:21 smithi146 ceph-mon[103278]: from='client.50457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:21 smithi078 ceph-mon[99959]: pgmap v2833: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:21 smithi078 ceph-mon[99959]: from='client.50451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:21 smithi078 ceph-mon[99959]: from='client.50457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:21 smithi078 ceph-mon[91136]: pgmap v2833: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:21 smithi078 ceph-mon[91136]: from='client.50451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:21 smithi078 ceph-mon[91136]: from='client.50457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:22.120 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:22.121 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:54:22.122 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:54:22.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:54:22.122 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:54:22.412 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:54:22.413 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:54:22.722 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:54:22.722 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:22.722 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:22.722 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='client.70382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1333530040' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='client.50475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2349341410' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='client.70382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1333530040' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='client.50475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2349341410' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:23.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='client.70382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:23.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1333530040' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:23.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:23.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='client.50475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:23.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2349341410' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:23.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:23.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:23.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:24.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:24 smithi146 ceph-mon[103278]: pgmap v2834: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:24.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:24 smithi078 ceph-mon[91136]: pgmap v2834: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:24 smithi078 ceph-mon[99959]: pgmap v2834: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 21:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T21:54:25.847111141Z level=info msg="Completed cleanup jobs" duration=11.993767ms 2024-09-20T21:54:26.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:26 smithi146 ceph-mon[103278]: pgmap v2835: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:26 smithi078 ceph-mon[91136]: pgmap v2835: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:26 smithi078 ceph-mon[99959]: pgmap v2835: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:28.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:28 smithi146 ceph-mon[103278]: pgmap v2836: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:28 smithi078 ceph-mon[99959]: pgmap v2836: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:28 smithi078 ceph-mon[91136]: pgmap v2836: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:30 smithi078 ceph-mon[91136]: pgmap v2837: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:30 smithi078 ceph-mon[99959]: pgmap v2837: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:54:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:54:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:30.494Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:30.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:30.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:30 smithi146 ceph-mon[103278]: pgmap v2837: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:32.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:32 smithi078 ceph-mon[91136]: pgmap v2838: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:32.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:32 smithi078 ceph-mon[99959]: pgmap v2838: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:32 smithi146 ceph-mon[103278]: pgmap v2838: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:34.137 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:33.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:34.137 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:33.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:34.137 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:33.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:34 smithi146 ceph-mon[103278]: pgmap v2839: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:34 smithi078 ceph-mon[91136]: pgmap v2839: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:34 smithi078 ceph-mon[99959]: pgmap v2839: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:36 smithi146 ceph-mon[103278]: pgmap v2840: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:36 smithi078 ceph-mon[91136]: pgmap v2840: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:36 smithi078 ceph-mon[99959]: pgmap v2840: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:38 smithi146 ceph-mon[103278]: pgmap v2841: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:38 smithi078 ceph-mon[91136]: pgmap v2841: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:38 smithi078 ceph-mon[99959]: pgmap v2841: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:40 smithi078 ceph-mon[91136]: pgmap v2842: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:40 smithi078 ceph-mon[99959]: pgmap v2842: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:54:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:54:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:40.494Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:40.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:40.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:40 smithi146 ceph-mon[103278]: pgmap v2842: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:41 smithi146 ceph-mon[103278]: pgmap v2843: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:41 smithi078 ceph-mon[91136]: pgmap v2843: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:41 smithi078 ceph-mon[99959]: pgmap v2843: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:43.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:43.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:43.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:43.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:44.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:44 smithi146 ceph-mon[103278]: pgmap v2844: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:44.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:44 smithi078 ceph-mon[99959]: pgmap v2844: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:44 smithi078 ceph-mon[91136]: pgmap v2844: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:46 smithi146 ceph-mon[103278]: pgmap v2845: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:46 smithi078 ceph-mon[91136]: pgmap v2845: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:46 smithi078 ceph-mon[99959]: pgmap v2845: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:47 smithi146 ceph-mon[103278]: pgmap v2846: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[99959]: pgmap v2846: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[91136]: pgmap v2846: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:54:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:49 smithi146 ceph-mon[103278]: pgmap v2847: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:49 smithi078 ceph-mon[91136]: pgmap v2847: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:49 smithi078 ceph-mon[99959]: pgmap v2847: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:54:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:54:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:50.495Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:50.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:50.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:51 smithi146 ceph-mon[103278]: pgmap v2848: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:51 smithi078 ceph-mon[99959]: pgmap v2848: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:51 smithi078 ceph-mon[91136]: pgmap v2848: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:53.019 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:54:53.555 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:54:53.555 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 105m - - 2024-09-20T21:54:53.555 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 105m - - 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (99m) 11s ago 100m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (100m) 6s ago 100m 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (93m) 11s ago 98m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (91m) 6s ago 104m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (93m) 11s ago 106m 561M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (92m) 11s ago 106m 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (90m) 6s ago 105m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (91m) 11s ago 104m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (100m) 11s ago 100m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (100m) 6s ago 100m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (89m) 11s ago 104m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (88m) 11s ago 103m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (88m) 11s ago 103m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:54:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (87m) 11s ago 103m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (87m) 6s ago 102m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (86m) 6s ago 102m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (85m) 6s ago 102m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (84m) 6s ago 101m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (93m) 6s ago 101m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (84m) 11s ago 98m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (83m) 6s ago 98m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (83m) 11s ago 98m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:54:53.557 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (83m) 6s ago 98m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:54:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:54:53.881 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[91136]: pgmap v2849: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[91136]: from='client.50487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[91136]: from='client.50493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[91136]: from='client.70418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[99959]: pgmap v2849: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[99959]: from='client.50487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[99959]: from='client.50493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[99959]: from='client.70418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.085 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:54.085 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:53.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:54.085 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:53.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:54:54.085 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:54:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:54:53.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:54:54.160 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:54:54.160 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:54:54.160 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:54:54.160 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:54:54.160 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:54:54.161 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:54:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:53 smithi146 ceph-mon[103278]: pgmap v2849: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:53 smithi146 ceph-mon[103278]: from='client.50487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:53 smithi146 ceph-mon[103278]: from='client.50493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:53 smithi146 ceph-mon[103278]: from='client.70418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:54:54.482 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:54:54.482 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:54.483 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:54:54.483 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:54.483 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:54:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3064133134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:54 smithi078 ceph-mon[91136]: from='client.50511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1537078914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3064133134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:54 smithi078 ceph-mon[99959]: from='client.50511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1537078914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3064133134' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:54:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:54 smithi146 ceph-mon[103278]: from='client.50511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:54:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1537078914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:54:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:55 smithi146 ceph-mon[103278]: pgmap v2850: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:55 smithi078 ceph-mon[91136]: pgmap v2850: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:55 smithi078 ceph-mon[99959]: pgmap v2850: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:54:58.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:57 smithi146 ceph-mon[103278]: pgmap v2851: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:57 smithi078 ceph-mon[91136]: pgmap v2851: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:57 smithi078 ceph-mon[99959]: pgmap v2851: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:54:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:54:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:54:59 smithi146 ceph-mon[103278]: pgmap v2852: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:54:59 smithi078 ceph-mon[91136]: pgmap v2852: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:54:59 smithi078 ceph-mon[99959]: pgmap v2852: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:55:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:55:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:00.496Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:00.497Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:00.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:01 smithi146 ceph-mon[103278]: pgmap v2853: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:01 smithi078 ceph-mon[99959]: pgmap v2853: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:01 smithi078 ceph-mon[91136]: pgmap v2853: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:03.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:03.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:03.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:03.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:03.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:04.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:04 smithi146 ceph-mon[103278]: pgmap v2854: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:04.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:04 smithi078 ceph-mon[99959]: pgmap v2854: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:04 smithi078 ceph-mon[91136]: pgmap v2854: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:06.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:06 smithi146 ceph-mon[103278]: pgmap v2855: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:06 smithi078 ceph-mon[99959]: pgmap v2855: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:06 smithi078 ceph-mon[91136]: pgmap v2855: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:07 smithi146 ceph-mon[103278]: pgmap v2856: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[91136]: pgmap v2856: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[99959]: pgmap v2856: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:09 smithi146 ceph-mon[103278]: pgmap v2857: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:09 smithi078 ceph-mon[99959]: pgmap v2857: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:09 smithi078 ceph-mon[91136]: pgmap v2857: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:55:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:55:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:10.497Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:10.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:10.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:11 smithi146 ceph-mon[103278]: pgmap v2858: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:12.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:11 smithi078 ceph-mon[91136]: pgmap v2858: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:12.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:11 smithi078 ceph-mon[99959]: pgmap v2858: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:13 smithi146 ceph-mon[103278]: pgmap v2859: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-mon[91136]: pgmap v2859: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-mon[99959]: pgmap v2859: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:13.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:13.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:13.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:15 smithi146 ceph-mon[103278]: pgmap v2860: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:15 smithi078 ceph-mon[91136]: pgmap v2860: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:15 smithi078 ceph-mon[99959]: pgmap v2860: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:17 smithi146 ceph-mon[103278]: pgmap v2861: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:17 smithi078 ceph-mon[91136]: pgmap v2861: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:17 smithi078 ceph-mon[99959]: pgmap v2861: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:19 smithi146 ceph-mon[103278]: pgmap v2862: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:19 smithi078 ceph-mon[91136]: pgmap v2862: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:19 smithi078 ceph-mon[99959]: pgmap v2862: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:55:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:55:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:20.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:20.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:20.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:21 smithi078 ceph-mon[91136]: pgmap v2863: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:21 smithi078 ceph-mon[99959]: pgmap v2863: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:21 smithi146 ceph-mon[103278]: pgmap v2863: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:23.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:23.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:23.933Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:23.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:23.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:24.781 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:55:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:24 smithi146 ceph-mon[103278]: pgmap v2864: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:24 smithi078 ceph-mon[91136]: pgmap v2864: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:24 smithi078 ceph-mon[99959]: pgmap v2864: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:25.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:25.303 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:55:25.303 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 106m - - 2024-09-20T21:55:25.303 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 105m - - 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (100m) 2s ago 101m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (101m) 18s ago 101m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (93m) 2s ago 98m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (92m) 18s ago 105m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (94m) 2s ago 106m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (92m) 2s ago 107m 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (91m) 18s ago 105m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (91m) 2s ago 105m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (100m) 2s ago 100m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (100m) 18s ago 100m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (90m) 2s ago 104m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (89m) 2s ago 104m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (88m) 2s ago 104m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:55:25.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (88m) 2s ago 103m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (87m) 18s ago 103m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (86m) 18s ago 102m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (85m) 18s ago 102m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (84m) 18s ago 102m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (93m) 18s ago 101m 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (84m) 2s ago 99m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (84m) 18s ago 99m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (84m) 2s ago 99m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:55:25.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (84m) 18s ago 99m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:55:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:55:25.616 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:55:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3132563372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:25.886 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:55:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:55:25.888 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:55:25.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3132563372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:25.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3132563372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:26.194 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:55:26.194 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:26.194 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:26.194 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:26.194 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:26 smithi146 ceph-mon[103278]: pgmap v2865: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:26 smithi146 ceph-mon[103278]: from='client.50523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:26 smithi146 ceph-mon[103278]: from='client.70448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:26 smithi146 ceph-mon[103278]: from='client.50535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/585445042' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[91136]: pgmap v2865: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[91136]: from='client.50523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[91136]: from='client.70448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[91136]: from='client.50535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/585445042' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[99959]: pgmap v2865: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[99959]: from='client.50523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[99959]: from='client.70448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[99959]: from='client.50535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/585445042' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:27 smithi146 ceph-mon[103278]: from='client.70466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[91136]: from='client.70466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[99959]: from='client.70466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:28 smithi146 ceph-mon[103278]: pgmap v2866: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:28 smithi078 ceph-mon[91136]: pgmap v2866: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:28 smithi078 ceph-mon[99959]: pgmap v2866: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:30 smithi078 ceph-mon[91136]: pgmap v2867: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:30 smithi078 ceph-mon[99959]: pgmap v2867: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:55:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:55:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:30.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:30.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:30.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:30 smithi146 ceph-mon[103278]: pgmap v2867: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:32 smithi146 ceph-mon[103278]: pgmap v2868: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:32 smithi078 ceph-mon[91136]: pgmap v2868: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:32 smithi078 ceph-mon[99959]: pgmap v2868: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:33.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:33.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:33.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:34.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:34 smithi078 ceph-mon[91136]: pgmap v2869: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:34.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:34.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:34 smithi078 ceph-mon[99959]: pgmap v2869: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:34.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:34 smithi146 ceph-mon[103278]: pgmap v2869: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:36 smithi146 ceph-mon[103278]: pgmap v2870: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:36 smithi078 ceph-mon[91136]: pgmap v2870: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:36 smithi078 ceph-mon[99959]: pgmap v2870: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:38 smithi146 ceph-mon[103278]: pgmap v2871: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:38 smithi078 ceph-mon[91136]: pgmap v2871: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:38 smithi078 ceph-mon[99959]: pgmap v2871: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:40 smithi078 ceph-mon[91136]: pgmap v2872: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:40 smithi078 ceph-mon[99959]: pgmap v2872: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:55:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:55:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:40.500Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:40.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:40.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:40 smithi146 ceph-mon[103278]: pgmap v2872: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:42 smithi146 ceph-mon[103278]: pgmap v2873: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:42 smithi078 ceph-mon[91136]: pgmap v2873: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:42 smithi078 ceph-mon[99959]: pgmap v2873: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[99959]: pgmap v2874: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[91136]: pgmap v2874: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:43 smithi146 ceph-mon[103278]: pgmap v2874: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:43.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:43.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:43.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:45 smithi146 ceph-mon[103278]: pgmap v2875: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:45 smithi078 ceph-mon[91136]: pgmap v2875: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:45 smithi078 ceph-mon[99959]: pgmap v2875: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:48 smithi146 ceph-mon[103278]: pgmap v2876: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[91136]: pgmap v2876: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[99959]: pgmap v2876: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:55:50.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:50 smithi078 ceph-mon[91136]: pgmap v2877: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:50.502 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:50 smithi078 ceph-mon[99959]: pgmap v2877: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:50 smithi146 ceph-mon[103278]: pgmap v2877: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:55:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:55:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:50.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:50.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:50.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:52 smithi146 ceph-mon[103278]: pgmap v2878: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:52 smithi078 ceph-mon[99959]: pgmap v2878: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:52 smithi078 ceph-mon[91136]: pgmap v2878: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:54.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:53.936Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:54.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:53.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:55:54.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:55:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:55:53.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:55:54.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:54 smithi078 ceph-mon[91136]: pgmap v2879: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:54.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:54.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:54 smithi078 ceph-mon[99959]: pgmap v2879: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:54.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:54 smithi146 ceph-mon[103278]: pgmap v2879: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:55:56.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:56 smithi078 ceph-mon[99959]: pgmap v2880: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:56.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:56 smithi078 ceph-mon[91136]: pgmap v2880: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:55:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:56 smithi146 ceph-mon[103278]: pgmap v2880: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:55:57.051 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 106m - - 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 106m - - 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (100m) 14s ago 101m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (101m) 9s ago 101m 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (94m) 14s ago 99m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (92m) 9s ago 105m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (94m) 14s ago 107m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (93m) 14s ago 107m 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (91m) 9s ago 106m 131M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (92m) 14s ago 106m 137M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (101m) 14s ago 101m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (101m) 9s ago 101m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:55:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (90m) 14s ago 105m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (89m) 14s ago 105m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (89m) 14s ago 104m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (88m) 14s ago 104m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (88m) 9s ago 103m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (87m) 9s ago 103m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (86m) 9s ago 103m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (85m) 9s ago 102m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (94m) 9s ago 102m 69.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (85m) 14s ago 99m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (84m) 9s ago 99m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (85m) 14s ago 99m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:55:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (84m) 9s ago 99m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:55:57.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:57 smithi078 ceph-mon[99959]: from='client.50559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:57.312 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:57 smithi078 ceph-mon[91136]: from='client.50559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:55:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:55:57.387 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:55:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:57 smithi146 ceph-mon[103278]: from='client.50559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:57.665 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:55:57.666 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:55:57.975 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:55:57.975 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:57.975 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:55:57.975 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:57.975 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:55:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: pgmap v2881: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: from='client.70484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: from='client.50571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/496001271' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: from='client.50583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3476733148' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: pgmap v2881: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: from='client.70484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: from='client.50571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/496001271' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: from='client.50583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3476733148' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:58.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:55:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:55:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: pgmap v2881: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:55:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: from='client.70484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: from='client.50571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/496001271' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:55:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: from='client.50583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:55:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3476733148' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:55:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:55:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:00.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:00 smithi078 ceph-mon[91136]: pgmap v2882: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:00.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:00 smithi078 ceph-mon[99959]: pgmap v2882: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:00 smithi146 ceph-mon[103278]: pgmap v2882: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:56:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:56:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:00.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:00.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:00.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:02 smithi146 ceph-mon[103278]: pgmap v2883: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:02 smithi078 ceph-mon[91136]: pgmap v2883: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:02 smithi078 ceph-mon[99959]: pgmap v2883: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:03 smithi146 ceph-mon[103278]: pgmap v2884: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[91136]: pgmap v2884: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[99959]: pgmap v2884: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:04.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:03.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:04.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:03.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:04.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:03.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:05 smithi146 ceph-mon[103278]: pgmap v2885: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:05 smithi078 ceph-mon[91136]: pgmap v2885: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:06.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:05 smithi078 ceph-mon[99959]: pgmap v2885: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:08 smithi146 ceph-mon[103278]: pgmap v2886: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[99959]: pgmap v2886: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[91136]: pgmap v2886: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:10.588 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:10 smithi078 ceph-mon[99959]: pgmap v2887: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:10.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:10 smithi078 ceph-mon[91136]: pgmap v2887: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:10.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:10.503Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:10.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:10.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:10.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:10.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:10 smithi146 ceph-mon[103278]: pgmap v2887: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:56:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T21:56:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:12 smithi146 ceph-mon[103278]: pgmap v2888: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:12 smithi078 ceph-mon[91136]: pgmap v2888: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:12 smithi078 ceph-mon[99959]: pgmap v2888: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:13.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:13.656 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:13.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:13.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:13.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:14 smithi146 ceph-mon[103278]: pgmap v2889: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:14 smithi078 ceph-mon[91136]: pgmap v2889: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:14 smithi078 ceph-mon[99959]: pgmap v2889: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:16.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:16 smithi146 ceph-mon[103278]: pgmap v2890: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:16.585 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:16 smithi078 ceph-mon[99959]: pgmap v2890: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:16.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:16 smithi078 ceph-mon[91136]: pgmap v2890: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:18 smithi146 ceph-mon[103278]: pgmap v2891: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:18 smithi078 ceph-mon[91136]: pgmap v2891: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:18 smithi078 ceph-mon[99959]: pgmap v2891: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:20 smithi078 ceph-mon[99959]: pgmap v2892: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:20 smithi078 ceph-mon[91136]: pgmap v2892: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:20.504Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:20.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:20.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:20 smithi146 ceph-mon[103278]: pgmap v2892: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:56:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:56:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:22 smithi146 ceph-mon[103278]: pgmap v2893: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:22 smithi078 ceph-mon[91136]: pgmap v2893: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:22 smithi078 ceph-mon[99959]: pgmap v2893: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[99959]: pgmap v2894: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[91136]: pgmap v2894: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:24.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:23.939Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:24.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:23.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:24.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:23.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:24 smithi146 ceph-mon[103278]: pgmap v2894: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:26 smithi146 ceph-mon[103278]: pgmap v2895: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:26 smithi078 ceph-mon[99959]: pgmap v2895: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:26 smithi078 ceph-mon[91136]: pgmap v2895: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:28.278 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:56:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[99959]: pgmap v2896: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[91136]: pgmap v2896: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:28 smithi146 ceph-mon[103278]: pgmap v2896: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:28.801 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:56:28.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 107m - - 2024-09-20T21:56:28.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 106m - - 2024-09-20T21:56:28.801 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (101m) 5s ago 102m 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:56:28.801 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (102m) 1s ago 102m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:56:28.801 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (94m) 5s ago 100m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (93m) 1s ago 106m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (95m) 5s ago 107m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (93m) 5s ago 108m 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (92m) 1s ago 106m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (92m) 5s ago 106m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (101m) 5s ago 101m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (101m) 1s ago 101m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (91m) 5s ago 105m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (90m) 5s ago 105m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (89m) 5s ago 105m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (89m) 5s ago 104m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (88m) 1s ago 104m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (87m) 1s ago 104m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (86m) 1s ago 103m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:56:28.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (85m) 1s ago 103m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:56:28.803 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (94m) 1s ago 102m 69.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:56:28.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (85m) 5s ago 100m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:56:28.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (85m) 1s ago 100m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:56:28.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (85m) 5s ago 100m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:56:28.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (85m) 1s ago 100m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:56:29.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:56:29.120 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:56:29.389 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:56:29.389 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:56:29.389 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:56:29.389 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:56:29.389 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:56:29.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:56:29.700 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:56:29.700 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:56:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:56:29.700 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:56:29.700 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:56:29.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:29 smithi078 ceph-mon[99959]: from='client.70514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:29.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:29 smithi078 ceph-mon[99959]: from='client.70520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:29.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2174873205' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:29.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:29 smithi078 ceph-mon[91136]: from='client.70514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:29.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:29 smithi078 ceph-mon[91136]: from='client.70520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:29.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2174873205' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:29 smithi146 ceph-mon[103278]: from='client.70514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:29 smithi146 ceph-mon[103278]: from='client.70520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2174873205' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:30 smithi146 ceph-mon[103278]: pgmap v2897: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:30 smithi146 ceph-mon[103278]: from='client.50607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:30 smithi146 ceph-mon[103278]: from='client.70538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/272782572' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:56:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[91136]: pgmap v2897: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[91136]: from='client.50607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[91136]: from='client.70538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/272782572' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[99959]: pgmap v2897: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[99959]: from='client.50607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[99959]: from='client.70538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:56:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/272782572' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:56:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:30.505Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:30.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:30.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:32 smithi146 ceph-mon[103278]: pgmap v2898: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:32 smithi078 ceph-mon[91136]: pgmap v2898: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:32 smithi078 ceph-mon[99959]: pgmap v2898: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:33.940Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:33.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:33.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:34 smithi146 ceph-mon[103278]: pgmap v2899: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:34 smithi078 ceph-mon[99959]: pgmap v2899: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:34 smithi078 ceph-mon[91136]: pgmap v2899: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:36 smithi146 ceph-mon[103278]: pgmap v2900: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:36 smithi078 ceph-mon[91136]: pgmap v2900: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:36 smithi078 ceph-mon[99959]: pgmap v2900: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:38.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:38 smithi078 ceph-mon[91136]: pgmap v2901: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:38.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:38 smithi078 ceph-mon[99959]: pgmap v2901: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:38 smithi146 ceph-mon[103278]: pgmap v2901: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:40 smithi078 ceph-mon[91136]: pgmap v2902: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:40 smithi078 ceph-mon[99959]: pgmap v2902: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:40.506Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:40.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:40.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:40.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:40 smithi146 ceph-mon[103278]: pgmap v2902: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:56:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T21:56:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:42 smithi146 ceph-mon[103278]: pgmap v2903: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:42 smithi078 ceph-mon[91136]: pgmap v2903: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:42 smithi078 ceph-mon[99959]: pgmap v2903: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[91136]: pgmap v2904: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[99959]: pgmap v2904: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:43.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:43.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:43.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:44 smithi146 ceph-mon[103278]: pgmap v2904: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:46 smithi146 ceph-mon[103278]: pgmap v2905: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:46 smithi078 ceph-mon[91136]: pgmap v2905: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:46 smithi078 ceph-mon[99959]: pgmap v2905: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:48.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:48 smithi146 ceph-mon[103278]: pgmap v2906: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:48.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:48.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:48.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[99959]: pgmap v2906: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[91136]: pgmap v2906: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:56:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:50 smithi078 ceph-mon[91136]: pgmap v2907: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:50 smithi078 ceph-mon[99959]: pgmap v2907: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:56:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:56:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:50.507Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:50.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:50.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:50 smithi146 ceph-mon[103278]: pgmap v2907: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:52 smithi146 ceph-mon[103278]: pgmap v2908: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:52 smithi078 ceph-mon[91136]: pgmap v2908: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:52 smithi078 ceph-mon[99959]: pgmap v2908: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:53.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:53.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:56:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:56:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:56:53.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:56:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:54 smithi146 ceph-mon[103278]: pgmap v2909: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:54 smithi078 ceph-mon[99959]: pgmap v2909: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:54 smithi078 ceph-mon[91136]: pgmap v2909: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:56:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:56 smithi078 ceph-mon[91136]: pgmap v2910: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:56 smithi078 ceph-mon[99959]: pgmap v2910: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:56 smithi146 ceph-mon[103278]: pgmap v2910: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:56:58.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:58 smithi078 ceph-mon[99959]: pgmap v2911: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:58.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:56:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:58.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:58 smithi078 ceph-mon[91136]: pgmap v2911: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:58.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:56:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:58 smithi146 ceph-mon[103278]: pgmap v2911: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:56:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:56:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:56:59.982 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:57:00.507 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 107m - - 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 107m - - 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (101m) 17s ago 102m 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (102m) 12s ago 102m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (95m) 17s ago 100m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (93m) 12s ago 106m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (95m) 17s ago 108m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (94m) 17s ago 108m 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (92m) 12s ago 107m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (93m) 17s ago 107m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (102m) 17s ago 102m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (102m) 12s ago 102m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:57:00.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (91m) 17s ago 106m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (90m) 17s ago 106m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (90m) 17s ago 105m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (89m) 17s ago 105m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (89m) 12s ago 104m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (88m) 12s ago 104m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (87m) 12s ago 104m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (86m) 12s ago 103m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (95m) 12s ago 103m 69.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (86m) 17s ago 100m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (85m) 12s ago 101m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (86m) 17s ago 100m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:57:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (85m) 12s ago 100m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:57:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:00 smithi078 ceph-mon[99959]: pgmap v2912: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:00 smithi078 ceph-mon[91136]: pgmap v2912: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:57:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:57:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:00.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:00.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:00.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:00.818 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:57:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:57:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:57:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:57:00.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:57:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:00 smithi146 ceph-mon[103278]: pgmap v2912: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:57:01.089 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:57:01.090 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:57:01.090 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:57:01.090 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:57:01.090 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:57:01.090 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:57:01.401 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:57:01.401 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:01.401 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:01 smithi146 ceph-mon[103278]: from='client.70550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:01 smithi146 ceph-mon[103278]: from='client.50637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:01 smithi146 ceph-mon[103278]: from='client.70562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2991416599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1944423980' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[91136]: from='client.70550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[91136]: from='client.50637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[91136]: from='client.70562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2991416599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1944423980' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[99959]: from='client.70550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[99959]: from='client.50637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[99959]: from='client.70562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2991416599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1944423980' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:02 smithi146 ceph-mon[103278]: pgmap v2913: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:02 smithi146 ceph-mon[103278]: from='client.50655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:02 smithi078 ceph-mon[91136]: pgmap v2913: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:02 smithi078 ceph-mon[91136]: from='client.50655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:02 smithi078 ceph-mon[99959]: pgmap v2913: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:02 smithi078 ceph-mon[99959]: from='client.50655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:04.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:03.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:03.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:03.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:04 smithi146 ceph-mon[103278]: pgmap v2914: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[99959]: pgmap v2914: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[91136]: pgmap v2914: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:06 smithi146 ceph-mon[103278]: pgmap v2915: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:06 smithi078 ceph-mon[99959]: pgmap v2915: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:06 smithi078 ceph-mon[91136]: pgmap v2915: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:08.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:08 smithi146 ceph-mon[103278]: pgmap v2916: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:08.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[99959]: pgmap v2916: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[91136]: pgmap v2916: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:10.588 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:10 smithi078 ceph-mon[99959]: pgmap v2917: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:10.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:10 smithi078 ceph-mon[91136]: pgmap v2917: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:10.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:10.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:10.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:10.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:10.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:10.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:10 smithi146 ceph-mon[103278]: pgmap v2917: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:57:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T21:57:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:12 smithi146 ceph-mon[103278]: pgmap v2918: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:12 smithi078 ceph-mon[91136]: pgmap v2918: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:12 smithi078 ceph-mon[99959]: pgmap v2918: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:13.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:13.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:13.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:13.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:13.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:14 smithi146 ceph-mon[103278]: pgmap v2919: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:14 smithi078 ceph-mon[99959]: pgmap v2919: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:14 smithi078 ceph-mon[91136]: pgmap v2919: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:16 smithi146 ceph-mon[103278]: pgmap v2920: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:16 smithi078 ceph-mon[99959]: pgmap v2920: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:16 smithi078 ceph-mon[91136]: pgmap v2920: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:18 smithi078 ceph-mon[91136]: pgmap v2921: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:18 smithi078 ceph-mon[99959]: pgmap v2921: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:18 smithi146 ceph-mon[103278]: pgmap v2921: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:20 smithi146 ceph-mon[103278]: pgmap v2922: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:20 smithi078 ceph-mon[91136]: pgmap v2922: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:20.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:57:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:57:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:20 smithi078 ceph-mon[99959]: pgmap v2922: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:20.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:20.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:20.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:20.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:20.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:22 smithi146 ceph-mon[103278]: pgmap v2923: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:22 smithi078 ceph-mon[91136]: pgmap v2923: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:22 smithi078 ceph-mon[99959]: pgmap v2923: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:24.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:23.944Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:23.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:23.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:24 smithi146 ceph-mon[103278]: pgmap v2924: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[99959]: pgmap v2924: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[91136]: pgmap v2924: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:26 smithi146 ceph-mon[103278]: pgmap v2925: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:26 smithi078 ceph-mon[91136]: pgmap v2925: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:26 smithi078 ceph-mon[99959]: pgmap v2925: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:28 smithi146 ceph-mon[103278]: pgmap v2926: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[91136]: pgmap v2926: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[99959]: pgmap v2926: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:30 smithi146 ceph-mon[103278]: pgmap v2927: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:30 smithi078 ceph-mon[91136]: pgmap v2927: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:30 smithi078 ceph-mon[99959]: pgmap v2927: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:57:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:57:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:30.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:30.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:30.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:31.702 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 108m - - 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 107m - - 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (102m) 8s ago 103m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (103m) 4s ago 103m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (95m) 8s ago 101m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (94m) 4s ago 107m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (96m) 8s ago 108m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (94m) 8s ago 109m 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:57:32.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (93m) 4s ago 107m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (93m) 8s ago 107m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (102m) 8s ago 102m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (102m) 4s ago 102m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (92m) 8s ago 107m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (91m) 8s ago 106m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (90m) 8s ago 106m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (90m) 8s ago 105m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (89m) 4s ago 105m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (88m) 4s ago 105m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (87m) 4s ago 104m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (86m) 4s ago 104m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (95m) 4s ago 103m 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (86m) 8s ago 101m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:57:32.230 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (86m) 4s ago 101m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:57:32.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (86m) 8s ago 101m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:57:32.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (86m) 4s ago 101m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:57:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:32 smithi078 ceph-mon[91136]: pgmap v2928: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:32 smithi078 ceph-mon[91136]: from='client.50667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:32 smithi078 ceph-mon[99959]: pgmap v2928: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:32 smithi078 ceph-mon[99959]: from='client.50667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:57:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:57:32.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:57:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:32 smithi146 ceph-mon[103278]: pgmap v2928: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:32 smithi146 ceph-mon[103278]: from='client.50667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:57:32.808 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:57:32.809 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:57:32.809 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:57:32.809 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:57:32.809 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:57:32.809 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:57:33.121 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:57:33.122 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:57:33.122 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:57:33.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[91136]: from='client.70592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:33.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[91136]: from='client.50679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:33.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2891921277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:33.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1814461709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:33.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[99959]: from='client.70592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:33.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[99959]: from='client.50679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:33.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2891921277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:33.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1814461709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:33 smithi146 ceph-mon[103278]: from='client.70592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:33 smithi146 ceph-mon[103278]: from='client.50679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2891921277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:57:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1814461709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:57:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:33.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:33.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:33.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:34 smithi146 ceph-mon[103278]: pgmap v2929: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:34 smithi146 ceph-mon[103278]: from='client.50691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:34 smithi078 ceph-mon[91136]: pgmap v2929: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:34 smithi078 ceph-mon[91136]: from='client.50691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:34 smithi078 ceph-mon[99959]: pgmap v2929: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:34 smithi078 ceph-mon[99959]: from='client.50691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:57:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:36 smithi146 ceph-mon[103278]: pgmap v2930: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:36 smithi078 ceph-mon[91136]: pgmap v2930: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:36 smithi078 ceph-mon[99959]: pgmap v2930: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:38.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:38 smithi078 ceph-mon[99959]: pgmap v2931: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:38.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:38 smithi078 ceph-mon[91136]: pgmap v2931: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:38 smithi146 ceph-mon[103278]: pgmap v2931: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:40 smithi146 ceph-mon[103278]: pgmap v2932: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:40 smithi078 ceph-mon[91136]: pgmap v2932: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:40 smithi078 ceph-mon[99959]: pgmap v2932: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:57:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:57:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:40.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:40.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:40.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:42 smithi146 ceph-mon[103278]: pgmap v2933: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:42 smithi078 ceph-mon[91136]: pgmap v2933: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:42 smithi078 ceph-mon[99959]: pgmap v2933: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:43.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:43.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:43.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:43.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:43.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:44 smithi146 ceph-mon[103278]: pgmap v2934: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[91136]: pgmap v2934: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[99959]: pgmap v2934: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:46 smithi146 ceph-mon[103278]: pgmap v2935: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:46 smithi078 ceph-mon[91136]: pgmap v2935: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:46 smithi078 ceph-mon[99959]: pgmap v2935: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:48 smithi146 ceph-mon[103278]: pgmap v2936: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[91136]: pgmap v2936: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[99959]: pgmap v2936: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:57:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:50 smithi078 ceph-mon[91136]: pgmap v2937: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:50 smithi078 ceph-mon[99959]: pgmap v2937: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:57:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:57:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:50.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:50.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:50.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:50.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:50 smithi146 ceph-mon[103278]: pgmap v2937: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:52 smithi078 ceph-mon[91136]: pgmap v2938: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:52 smithi078 ceph-mon[99959]: pgmap v2938: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:52 smithi146 ceph-mon[103278]: pgmap v2938: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:53.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:53.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:57:54.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:57:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:57:53.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:57:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:54 smithi146 ceph-mon[103278]: pgmap v2939: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:54 smithi078 ceph-mon[91136]: pgmap v2939: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:54 smithi078 ceph-mon[99959]: pgmap v2939: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:57:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:56 smithi146 ceph-mon[103278]: pgmap v2940: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:56 smithi078 ceph-mon[91136]: pgmap v2940: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:56 smithi078 ceph-mon[99959]: pgmap v2940: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:57:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:58 smithi078 ceph-mon[91136]: pgmap v2941: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:58 smithi078 ceph-mon[99959]: pgmap v2941: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:57:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:58 smithi146 ceph-mon[103278]: pgmap v2941: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:57:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:00 smithi078 ceph-mon[91136]: pgmap v2942: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:58:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:58:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:00 smithi078 ceph-mon[99959]: pgmap v2942: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:00.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:00.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:00.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:00 smithi146 ceph-mon[103278]: pgmap v2942: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:02 smithi146 ceph-mon[103278]: pgmap v2943: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:02 smithi078 ceph-mon[91136]: pgmap v2943: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:02 smithi078 ceph-mon[99959]: pgmap v2943: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:03.416 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 108m - - 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 108m - - 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (102m) 0s ago 103m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (103m) 15s ago 103m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (96m) 0s ago 101m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:58:03.951 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (95m) 15s ago 107m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (96m) 0s ago 109m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (95m) 0s ago 109m 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (93m) 15s ago 108m 130M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (94m) 0s ago 108m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (103m) 0s ago 103m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (103m) 15s ago 103m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (92m) 0s ago 107m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (91m) 0s ago 107m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (91m) 0s ago 106m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (90m) 0s ago 106m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (90m) 15s ago 105m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (89m) 15s ago 105m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (88m) 15s ago 105m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:58:03.952 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (87m) 15s ago 104m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:58:03.953 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (96m) 15s ago 104m 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:58:03.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (87m) 0s ago 102m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:58:03.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (87m) 15s ago 102m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:58:03.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (87m) 0s ago 101m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:58:03.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (86m) 15s ago 101m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:58:04.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:03.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:04.119 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:03.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:04.119 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:03.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:04.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:58:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:58:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:58:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:58:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:58:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:58:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:58:04.264 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:58:04.529 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:58:04.529 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:58:04.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:58:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: pgmap v2944: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: from='client.50703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: from='client.50709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1856897281' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: pgmap v2944: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: from='client.50703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: from='client.50709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:04.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1856897281' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:04.842 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:58:04.842 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:58:04.842 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:58:04.842 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:58:04.842 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:58:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: pgmap v2944: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: from='client.50703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: from='client.50709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1856897281' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:05 smithi146 ceph-mon[103278]: from='client.70634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:05 smithi146 ceph-mon[103278]: from='client.70646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2848404328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[91136]: from='client.70634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[91136]: from='client.70646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2848404328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[99959]: from='client.70634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[99959]: from='client.70646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2848404328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:06 smithi146 ceph-mon[103278]: pgmap v2945: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:06 smithi078 ceph-mon[91136]: pgmap v2945: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:06 smithi078 ceph-mon[99959]: pgmap v2945: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:08.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:08 smithi146 ceph-mon[103278]: pgmap v2946: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:08.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:08.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:08.910 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[91136]: pgmap v2946: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[99959]: pgmap v2946: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:09.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:09.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:10 smithi078 ceph-mon[91136]: pgmap v2947: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:10 smithi078 ceph-mon[99959]: pgmap v2947: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:58:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T21:58:10.789 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:10.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:10.789 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:10.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:10.789 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:10.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:10.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:10 smithi146 ceph-mon[103278]: pgmap v2947: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:12 smithi146 ceph-mon[103278]: pgmap v2948: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:12 smithi078 ceph-mon[91136]: pgmap v2948: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:12 smithi078 ceph-mon[99959]: pgmap v2948: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:13.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:13.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:13.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:14 smithi146 ceph-mon[103278]: pgmap v2949: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:14 smithi078 ceph-mon[91136]: pgmap v2949: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:14 smithi078 ceph-mon[99959]: pgmap v2949: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:16 smithi146 ceph-mon[103278]: pgmap v2950: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:16 smithi078 ceph-mon[91136]: pgmap v2950: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:16 smithi078 ceph-mon[99959]: pgmap v2950: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:19.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:18 smithi078 ceph-mon[91136]: pgmap v2951: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:19.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:18 smithi078 ceph-mon[99959]: pgmap v2951: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:18 smithi146 ceph-mon[103278]: pgmap v2951: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:20.735 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:58:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:58:20.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:20.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:20.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:20.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:20.735 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:20.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:20 smithi146 ceph-mon[103278]: pgmap v2952: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:20 smithi078 ceph-mon[99959]: pgmap v2952: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:20 smithi078 ceph-mon[91136]: pgmap v2952: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:22 smithi146 ceph-mon[103278]: pgmap v2953: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:22 smithi078 ceph-mon[91136]: pgmap v2953: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:22 smithi078 ceph-mon[99959]: pgmap v2953: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:23.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:23.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:23.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:24 smithi146 ceph-mon[103278]: pgmap v2954: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[99959]: pgmap v2954: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[91136]: pgmap v2954: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:25.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:26 smithi146 ceph-mon[103278]: pgmap v2955: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:26 smithi078 ceph-mon[91136]: pgmap v2955: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:26 smithi078 ceph-mon[99959]: pgmap v2955: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:29.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:28 smithi146 ceph-mon[103278]: pgmap v2956: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:29.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:29.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[99959]: pgmap v2956: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[91136]: pgmap v2956: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:29.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:29 smithi146 ceph-mon[103278]: pgmap v2957: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:30.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:29 smithi078 ceph-mon[91136]: pgmap v2957: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:30.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:29 smithi078 ceph-mon[99959]: pgmap v2957: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:58:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:58:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:30.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:30.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:30.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:32.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:31 smithi146 ceph-mon[103278]: pgmap v2958: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:31 smithi078 ceph-mon[99959]: pgmap v2958: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:31 smithi078 ceph-mon[91136]: pgmap v2958: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:33 smithi078 ceph-mon[91136]: pgmap v2959: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:33 smithi078 ceph-mon[99959]: pgmap v2959: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:33.948Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:33.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:34.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:33.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:33 smithi146 ceph-mon[103278]: pgmap v2959: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:35.144 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:35.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:35.144 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:58:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 109m - - 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 108m - - 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (103m) 11s ago 104m 30.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (104m) 7s ago 104m 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (96m) 11s ago 102m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (95m) 7s ago 108m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (97m) 11s ago 109m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:58:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (95m) 11s ago 110m 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (94m) 7s ago 108m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (94m) 11s ago 108m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (103m) 11s ago 103m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (103m) 7s ago 103m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (93m) 11s ago 108m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (92m) 11s ago 107m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (91m) 11s ago 107m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (91m) 11s ago 106m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (90m) 7s ago 106m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (89m) 7s ago 106m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (88m) 7s ago 105m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (88m) 7s ago 105m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (96m) 7s ago 104m 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:58:35.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (87m) 11s ago 102m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:58:35.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (87m) 7s ago 102m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:58:35.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (87m) 11s ago 102m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:58:35.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (87m) 7s ago 102m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:58:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:58:35.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[91136]: pgmap v2960: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:35.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[91136]: from='client.50739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:35.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[91136]: from='client.50745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:35.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[91136]: from='client.70670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:35.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[99959]: pgmap v2960: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:35.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[99959]: from='client.50739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:35.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[99959]: from='client.50745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:35.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:35 smithi078 ceph-mon[99959]: from='client.70670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:36.253 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:58:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:58:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:58:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:58:36.253 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:58:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:58:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:58:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:58:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:58:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:58:36.254 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:58:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:58:36.255 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:58:36.255 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:35 smithi146 ceph-mon[103278]: pgmap v2960: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:35 smithi146 ceph-mon[103278]: from='client.50739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:35 smithi146 ceph-mon[103278]: from='client.50745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:35 smithi146 ceph-mon[103278]: from='client.70670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:36.559 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:58:36.559 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:58:36.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:58:36.559 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:58:36.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2888293543' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:36 smithi146 ceph-mon[103278]: from='client.70682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2252640452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2888293543' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:36 smithi078 ceph-mon[91136]: from='client.70682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2252640452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2888293543' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:58:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:36 smithi078 ceph-mon[99959]: from='client.70682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:58:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2252640452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:58:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:37 smithi146 ceph-mon[103278]: pgmap v2961: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:37 smithi078 ceph-mon[91136]: pgmap v2961: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:37 smithi078 ceph-mon[99959]: pgmap v2961: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:39 smithi078 ceph-mon[91136]: pgmap v2962: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:39 smithi078 ceph-mon[99959]: pgmap v2962: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:39 smithi146 ceph-mon[103278]: pgmap v2962: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:40.751 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:58:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:58:40.751 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:40.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:40.751 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:40.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:40.752 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:40.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:41 smithi146 ceph-mon[103278]: pgmap v2963: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:42.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:41 smithi078 ceph-mon[91136]: pgmap v2963: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:42.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:41 smithi078 ceph-mon[99959]: pgmap v2963: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:43 smithi146 ceph-mon[103278]: pgmap v2964: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-mon[91136]: pgmap v2964: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-mon[99959]: pgmap v2964: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:43.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:43.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:43.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:45 smithi146 ceph-mon[103278]: pgmap v2965: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:45 smithi078 ceph-mon[91136]: pgmap v2965: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:45 smithi078 ceph-mon[99959]: pgmap v2965: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:47 smithi146 ceph-mon[103278]: pgmap v2966: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:47 smithi078 ceph-mon[91136]: pgmap v2966: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:47 smithi078 ceph-mon[99959]: pgmap v2966: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:58:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:50 smithi146 ceph-mon[103278]: pgmap v2967: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:50 smithi078 ceph-mon[91136]: pgmap v2967: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:50 smithi078 ceph-mon[99959]: pgmap v2967: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:58:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:58:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:50.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:50.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:50.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:52 smithi146 ceph-mon[103278]: pgmap v2968: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:52 smithi078 ceph-mon[91136]: pgmap v2968: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:52 smithi078 ceph-mon[99959]: pgmap v2968: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:53.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:53.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:58:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:58:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:58:53.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:58:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:54 smithi146 ceph-mon[103278]: pgmap v2969: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:54 smithi078 ceph-mon[91136]: pgmap v2969: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:54 smithi078 ceph-mon[99959]: pgmap v2969: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:58:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:58:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:56 smithi146 ceph-mon[103278]: pgmap v2970: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:56 smithi078 ceph-mon[91136]: pgmap v2970: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:56 smithi078 ceph-mon[99959]: pgmap v2970: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:58:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:58 smithi146 ceph-mon[103278]: pgmap v2971: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:58:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:58:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:58 smithi078 ceph-mon[91136]: pgmap v2971: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:58:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:58:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:58:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:58 smithi078 ceph-mon[99959]: pgmap v2971: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:58:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:58:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:00.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:00 smithi078 ceph-mon[91136]: pgmap v2972: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:00.655 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:59:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:59:00.655 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:00 smithi078 ceph-mon[99959]: pgmap v2972: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:00.655 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:00.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:00.655 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:00.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:00.656 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:00.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:00 smithi146 ceph-mon[103278]: pgmap v2972: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:02 smithi146 ceph-mon[103278]: pgmap v2973: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:02.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:02 smithi078 ceph-mon[99959]: pgmap v2973: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:02.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:02 smithi078 ceph-mon[91136]: pgmap v2973: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:04.112 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:03.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:04.112 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:03.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:04.112 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:03.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:04 smithi146 ceph-mon[103278]: pgmap v2974: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:59:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[91136]: pgmap v2974: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[99959]: pgmap v2974: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:06 smithi146 ceph-mon[103278]: pgmap v2975: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:06 smithi078 ceph-mon[99959]: pgmap v2975: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:06 smithi078 ceph-mon[91136]: pgmap v2975: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:06.857 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 109m - - 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 109m - - 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (103m) 3s ago 104m 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (104m) 18s ago 104m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (97m) 3s ago 102m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (96m) 18s ago 108m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (97m) 3s ago 110m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (96m) 3s ago 110m 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (94m) 18s ago 109m 133M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (95m) 3s ago 109m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:59:07.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (104m) 3s ago 104m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (104m) 18s ago 104m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (93m) 3s ago 108m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (92m) 3s ago 108m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (92m) 3s ago 107m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (91m) 3s ago 107m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (91m) 18s ago 107m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (90m) 18s ago 106m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (89m) 18s ago 106m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (88m) 18s ago 105m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (97m) 18s ago 105m 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (88m) 3s ago 103m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (88m) 18s ago 103m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:59:07.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (88m) 3s ago 103m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:59:07.385 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (87m) 18s ago 102m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:59:07.699 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:59:07.970 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:59:07.970 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:59:07.970 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:59:07.970 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:59:07.970 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:59:07.971 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:59:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:59:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[99959]: pgmap v2976: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[99959]: from='client.50775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[99959]: from='client.50781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[99959]: from='client.70706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1736658413' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3078444679' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[91136]: pgmap v2976: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[91136]: from='client.50775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[91136]: from='client.50781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[91136]: from='client.70706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1736658413' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:08.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3078444679' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:08 smithi146 ceph-mon[103278]: pgmap v2976: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:08 smithi146 ceph-mon[103278]: from='client.50775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:08 smithi146 ceph-mon[103278]: from='client.50781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:08 smithi146 ceph-mon[103278]: from='client.70706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1736658413' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3078444679' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:09 smithi146 ceph-mon[103278]: from='client.70718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[91136]: from='client.70718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[99959]: from='client.70718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:10 smithi078 ceph-mon[91136]: pgmap v2977: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:10 smithi078 ceph-mon[99959]: pgmap v2977: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:59:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:59:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:10.517Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:10.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:10.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:10.841 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:10 smithi146 ceph-mon[103278]: pgmap v2977: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:12 smithi146 ceph-mon[103278]: pgmap v2978: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:12 smithi078 ceph-mon[91136]: pgmap v2978: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:12 smithi078 ceph-mon[99959]: pgmap v2978: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:13.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:13.817 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:13.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:13.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:13.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:14 smithi146 ceph-mon[103278]: pgmap v2979: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:14 smithi078 ceph-mon[91136]: pgmap v2979: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:14 smithi078 ceph-mon[99959]: pgmap v2979: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:16.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:16 smithi146 ceph-mon[103278]: pgmap v2980: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:16 smithi078 ceph-mon[91136]: pgmap v2980: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:16 smithi078 ceph-mon[99959]: pgmap v2980: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:18 smithi146 ceph-mon[103278]: pgmap v2981: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:18 smithi078 ceph-mon[91136]: pgmap v2981: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:18 smithi078 ceph-mon[99959]: pgmap v2981: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:20 smithi078 ceph-mon[91136]: pgmap v2982: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:20 smithi078 ceph-mon[99959]: pgmap v2982: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:59:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T21:59:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:20.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:20.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:20.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:21.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:20 smithi146 ceph-mon[103278]: pgmap v2982: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:22 smithi146 ceph-mon[103278]: pgmap v2983: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:23.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:22 smithi078 ceph-mon[99959]: pgmap v2983: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:23.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:22 smithi078 ceph-mon[91136]: pgmap v2983: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:23.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:23.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:24.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:23.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:24.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[91136]: pgmap v2984: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:24.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:24.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:24.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:24.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[99959]: pgmap v2984: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:24.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:24.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:24.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:24 smithi146 ceph-mon[103278]: pgmap v2984: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:26 smithi146 ceph-mon[103278]: pgmap v2985: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:26 smithi078 ceph-mon[91136]: pgmap v2985: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:26 smithi078 ceph-mon[99959]: pgmap v2985: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:28 smithi146 ceph-mon[103278]: pgmap v2986: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:28 smithi078 ceph-mon[91136]: pgmap v2986: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:28 smithi078 ceph-mon[99959]: pgmap v2986: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:30 smithi078 ceph-mon[99959]: pgmap v2987: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:30 smithi078 ceph-mon[91136]: pgmap v2987: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:59:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:59:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:30.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:30.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:30.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:31.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:30 smithi146 ceph-mon[103278]: pgmap v2987: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:32.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:32 smithi146 ceph-mon[103278]: pgmap v2988: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:32 smithi078 ceph-mon[99959]: pgmap v2988: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:32 smithi078 ceph-mon[91136]: pgmap v2988: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:33.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:33.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:33.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:34.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:34 smithi078 ceph-mon[91136]: pgmap v2989: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:34.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:34 smithi078 ceph-mon[99959]: pgmap v2989: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:34 smithi146 ceph-mon[103278]: pgmap v2989: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:36 smithi146 ceph-mon[103278]: pgmap v2990: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:36 smithi078 ceph-mon[91136]: pgmap v2990: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:36 smithi078 ceph-mon[99959]: pgmap v2990: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:38.579 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T21:59:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:38 smithi146 ceph-mon[103278]: pgmap v2991: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:38 smithi078 ceph-mon[99959]: pgmap v2991: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:38 smithi078 ceph-mon[91136]: pgmap v2991: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:39.107 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T21:59:39.107 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 110m - - 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 110m - - 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (104m) 14s ago 105m 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (105m) 10s ago 105m 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (97m) 14s ago 103m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (96m) 10s ago 109m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (98m) 14s ago 111m 562M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (96m) 14s ago 111m 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (95m) 10s ago 109m 134M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (95m) 14s ago 109m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (104m) 14s ago 104m 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (104m) 10s ago 104m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T21:59:39.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (94m) 14s ago 109m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (93m) 14s ago 108m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (92m) 14s ago 108m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (92m) 14s ago 107m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (91m) 10s ago 107m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (90m) 10s ago 107m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (90m) 10s ago 106m 68.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (89m) 10s ago 106m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (97m) 10s ago 105m 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (88m) 14s ago 103m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (88m) 10s ago 103m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (88m) 14s ago 103m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T21:59:39.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (88m) 10s ago 103m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T21:59:39.416 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T21:59:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T21:59:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T21:59:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T21:59:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T21:59:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:59:39.690 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T21:59:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T21:59:39.691 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T21:59:40.003 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T21:59:40.003 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:40.004 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T21:59:40.004 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:40.004 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T21:59:40.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:39 smithi078 ceph-mon[99959]: from='client.70730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4278831847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:40.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:39 smithi078 ceph-mon[91136]: from='client.70730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4278831847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:39 smithi146 ceph-mon[103278]: from='client.70730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4278831847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T21:59:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:59:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T21:59:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[91136]: pgmap v2992: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[91136]: from='client.70736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[91136]: from='client.70742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[91136]: from='client.70754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2280917222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[99959]: pgmap v2992: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[99959]: from='client.70736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[99959]: from='client.70742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[99959]: from='client.70754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2280917222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:40.520Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:40.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:40.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:40 smithi146 ceph-mon[103278]: pgmap v2992: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:40 smithi146 ceph-mon[103278]: from='client.70736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:40 smithi146 ceph-mon[103278]: from='client.70742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:40 smithi146 ceph-mon[103278]: from='client.70754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T21:59:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2280917222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T21:59:42.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:42 smithi078 ceph-mon[91136]: pgmap v2993: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:42.766 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:42 smithi078 ceph-mon[99959]: pgmap v2993: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:42 smithi146 ceph-mon[103278]: pgmap v2993: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:43.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:43.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:43.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:43.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:44 smithi146 ceph-mon[103278]: pgmap v2994: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[91136]: pgmap v2994: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[99959]: pgmap v2994: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:46 smithi146 ceph-mon[103278]: pgmap v2995: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:46 smithi078 ceph-mon[99959]: pgmap v2995: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:46 smithi078 ceph-mon[91136]: pgmap v2995: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:48 smithi146 ceph-mon[103278]: pgmap v2996: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:48 smithi078 ceph-mon[91136]: pgmap v2996: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:48 smithi078 ceph-mon[99959]: pgmap v2996: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:49 smithi146 ceph-mon[103278]: pgmap v2997: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[99959]: pgmap v2997: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T21:59:50.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:49 smithi078 ceph-mon[91136]: pgmap v2997: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 21:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:21:59:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T21:59:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:50.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:50.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:50.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:51.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:51 smithi146 ceph-mon[103278]: pgmap v2998: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:51 smithi078 ceph-mon[99959]: pgmap v2998: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:51 smithi078 ceph-mon[91136]: pgmap v2998: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:53 smithi146 ceph-mon[103278]: pgmap v2999: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:53 smithi078 ceph-mon[91136]: pgmap v2999: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:53 smithi078 ceph-mon[99959]: pgmap v2999: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:53.954Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:53.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T21:59:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 21:59:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T21:59:53.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T21:59:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T21:59:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:55 smithi146 ceph-mon[103278]: pgmap v3000: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:55 smithi078 ceph-mon[91136]: pgmap v3000: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:55 smithi078 ceph-mon[99959]: pgmap v3000: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T21:59:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:57 smithi146 ceph-mon[103278]: pgmap v3001: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:57 smithi078 ceph-mon[91136]: pgmap v3001: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:57 smithi078 ceph-mon[99959]: pgmap v3001: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T21:59:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T21:59:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 21:59:59 smithi146 ceph-mon[103278]: pgmap v3002: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 21:59:59 smithi078 ceph-mon[99959]: pgmap v3002: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 21:59:59 smithi078 ceph-mon[91136]: pgmap v3002: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T21:59:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:00:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:00:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:00.522Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:00.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:00.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:02.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:01 smithi078 ceph-mon[99959]: pgmap v3003: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:02.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:01 smithi078 ceph-mon[91136]: pgmap v3003: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:01 smithi146 ceph-mon[103278]: pgmap v3003: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:03 smithi146 ceph-mon[103278]: pgmap v3004: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:04.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:03 smithi078 ceph-mon[99959]: pgmap v3004: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:04.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:03 smithi078 ceph-mon[91136]: pgmap v3004: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:04.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:03.954Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:04.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:03.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:04.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:03.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:06 smithi146 ceph-mon[103278]: pgmap v3005: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:06 smithi078 ceph-mon[91136]: pgmap v3005: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:06 smithi078 ceph-mon[99959]: pgmap v3005: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:08 smithi146 ceph-mon[103278]: pgmap v3006: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:08 smithi078 ceph-mon[99959]: pgmap v3006: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:08 smithi078 ceph-mon[91136]: pgmap v3006: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:09 smithi146 ceph-mon[103278]: pgmap v3007: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[91136]: pgmap v3007: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[99959]: pgmap v3007: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:10.300 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:00:10.609 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:10.523Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:10.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:10.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:10.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 110m - - 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 110m - - 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (104m) 6s ago 106m 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (105m) 1s ago 105m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (98m) 6s ago 103m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (97m) 1s ago 109m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (99m) 6s ago 111m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (97m) 6s ago 111m 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (96m) 1s ago 110m 135M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (96m) 6s ago 110m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (105m) 6s ago 105m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (105m) 1s ago 105m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:00:10.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (95m) 6s ago 109m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (94m) 6s ago 109m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (93m) 6s ago 108m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (92m) 6s ago 108m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (92m) 1s ago 108m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (91m) 1s ago 107m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (90m) 1s ago 107m 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (89m) 1s ago 106m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (98m) 1s ago 106m 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (89m) 6s ago 104m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (89m) 1s ago 104m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (89m) 6s ago 104m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:00:10.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (89m) 1s ago 104m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:00:10.943 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:00:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:00:11.139 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:00:11.139 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:00:11.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:00:11.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:11.139 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:00:11.140 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:00:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:10 smithi146 ceph-mon[103278]: from='client.70766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:10 smithi146 ceph-mon[103278]: from='client.70772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-mon[91136]: from='client.70766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-mon[91136]: from='client.70772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-mon[99959]: from='client.70766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:10 smithi078 ceph-mon[99959]: from='client.70772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:11.409 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:00:11.409 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:00:11.409 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:00:11.410 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:00:11.724 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:11.724 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:11.724 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:11.724 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:11.724 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[91136]: from='client.50859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[91136]: pgmap v3008: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2429455054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[91136]: from='client.70790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2858351390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[99959]: from='client.50859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[99959]: pgmap v3008: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2429455054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[99959]: from='client.70790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2858351390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:11 smithi146 ceph-mon[103278]: from='client.50859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:11 smithi146 ceph-mon[103278]: pgmap v3008: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2429455054' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:11 smithi146 ceph-mon[103278]: from='client.70790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2858351390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:13 smithi146 ceph-mon[103278]: pgmap v3009: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-mon[91136]: pgmap v3009: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-mon[99959]: pgmap v3009: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:13.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:13.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:13.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:15 smithi146 ceph-mon[103278]: pgmap v3010: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:15 smithi078 ceph-mon[91136]: pgmap v3010: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:15 smithi078 ceph-mon[99959]: pgmap v3010: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:17 smithi146 ceph-mon[103278]: pgmap v3011: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:17 smithi078 ceph-mon[91136]: pgmap v3011: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:17 smithi078 ceph-mon[99959]: pgmap v3011: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:19 smithi146 ceph-mon[103278]: pgmap v3012: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:19 smithi078 ceph-mon[91136]: pgmap v3012: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:19 smithi078 ceph-mon[99959]: pgmap v3012: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:20.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:20.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:20.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:00:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:00:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:21 smithi146 ceph-mon[103278]: pgmap v3013: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:21 smithi078 ceph-mon[91136]: pgmap v3013: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:21 smithi078 ceph-mon[99959]: pgmap v3013: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:24.181 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:23 smithi078 ceph-mon[99959]: pgmap v3014: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:24.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:23 smithi078 ceph-mon[91136]: pgmap v3014: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:24.182 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:23.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:24.182 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:23.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:24.182 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:23.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:23 smithi146 ceph-mon[103278]: pgmap v3014: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:25.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:26 smithi146 ceph-mon[103278]: pgmap v3015: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:26 smithi078 ceph-mon[91136]: pgmap v3015: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:26 smithi078 ceph-mon[99959]: pgmap v3015: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:28 smithi146 ceph-mon[103278]: pgmap v3016: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:28 smithi078 ceph-mon[91136]: pgmap v3016: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:28 smithi078 ceph-mon[99959]: pgmap v3016: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:30 smithi146 ceph-mon[103278]: pgmap v3017: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[91136]: pgmap v3017: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[99959]: pgmap v3017: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:00:30] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:00:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:30.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:30.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:30.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:32.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:32 smithi146 ceph-mon[103278]: pgmap v3018: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:32 smithi078 ceph-mon[91136]: pgmap v3018: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:32 smithi078 ceph-mon[99959]: pgmap v3018: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:34 smithi146 ceph-mon[103278]: pgmap v3019: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:34 smithi078 ceph-mon[91136]: pgmap v3019: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:34 smithi078 ceph-mon[99959]: pgmap v3019: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:33.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:33.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:33.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:36 smithi146 ceph-mon[103278]: pgmap v3020: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:36 smithi078 ceph-mon[91136]: pgmap v3020: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:36 smithi078 ceph-mon[99959]: pgmap v3020: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:38 smithi146 ceph-mon[103278]: pgmap v3021: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:38 smithi078 ceph-mon[99959]: pgmap v3021: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:38 smithi078 ceph-mon[91136]: pgmap v3021: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:40.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:40 smithi078 ceph-mon[99959]: pgmap v3022: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:40.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:40 smithi078 ceph-mon[91136]: pgmap v3022: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:40 smithi146 ceph-mon[103278]: pgmap v3022: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:40.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:40.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:40.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:41.038 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:00:40] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:00:42.016 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:00:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:42 smithi078 ceph-mon[91136]: pgmap v3023: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:42 smithi078 ceph-mon[99959]: pgmap v3023: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:42 smithi146 ceph-mon[103278]: pgmap v3023: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 111m - - 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 111m - - 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (105m) 17s ago 106m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (106m) 13s ago 106m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (98m) 17s ago 104m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (97m) 13s ago 110m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (99m) 17s ago 112m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:00:42.544 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (97m) 17s ago 112m 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (96m) 13s ago 110m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (96m) 17s ago 110m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (106m) 17s ago 106m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (105m) 13s ago 105m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (95m) 17s ago 110m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (94m) 17s ago 109m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (93m) 17s ago 109m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (93m) 17s ago 109m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (92m) 13s ago 108m 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (91m) 13s ago 108m 64.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (91m) 13s ago 107m 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (90m) 13s ago 107m 64.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:00:42.545 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (99m) 13s ago 106m 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:00:42.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (89m) 17s ago 104m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:00:42.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (89m) 13s ago 104m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:00:42.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (89m) 17s ago 104m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:00:42.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (89m) 13s ago 104m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:00:42.854 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:00:42.854 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:00:42.854 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:00:42.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:00:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:00:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:00:43.126 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[99959]: from='client.50883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[99959]: from='client.50889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[99959]: from='client.50895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1222656213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[91136]: from='client.50883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[91136]: from='client.50889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[91136]: from='client.50895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1222656213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:00:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:00:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:00:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:43 smithi146 ceph-mon[103278]: from='client.50883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:43 smithi146 ceph-mon[103278]: from='client.50889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:43 smithi146 ceph-mon[103278]: from='client.50895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1222656213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:00:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[99959]: pgmap v3024: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[99959]: from='client.50907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1945120089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[91136]: pgmap v3024: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[91136]: from='client.50907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1945120089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:43.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:43.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:44.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:43.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:44 smithi146 ceph-mon[103278]: pgmap v3024: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:44 smithi146 ceph-mon[103278]: from='client.50907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:00:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1945120089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:00:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:45.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:45.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:46 smithi146 ceph-mon[103278]: pgmap v3025: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:46 smithi078 ceph-mon[91136]: pgmap v3025: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:46 smithi078 ceph-mon[99959]: pgmap v3025: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:48 smithi146 ceph-mon[103278]: pgmap v3026: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:48 smithi078 ceph-mon[91136]: pgmap v3026: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:48 smithi078 ceph-mon[99959]: pgmap v3026: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:50 smithi146 ceph-mon[103278]: pgmap v3027: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[91136]: pgmap v3027: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:50.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[99959]: pgmap v3027: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:00:50.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:00:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:00:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:00:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:50.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:50.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:50.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:52 smithi146 ceph-mon[103278]: pgmap v3028: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:52 smithi078 ceph-mon[91136]: pgmap v3028: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:52 smithi078 ceph-mon[99959]: pgmap v3028: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:54.207 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:53.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:54.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:53.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:00:54.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:00:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:00:53.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:00:54.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:54 smithi146 ceph-mon[103278]: pgmap v3029: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:54 smithi078 ceph-mon[91136]: pgmap v3029: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:54 smithi078 ceph-mon[99959]: pgmap v3029: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:00:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:00:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:56 smithi146 ceph-mon[103278]: pgmap v3030: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:00:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:56 smithi078 ceph-mon[91136]: pgmap v3030: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:00:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:56 smithi078 ceph-mon[99959]: pgmap v3030: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:00:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:58 smithi146 ceph-mon[103278]: pgmap v3031: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:00:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:58 smithi078 ceph-mon[91136]: pgmap v3031: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:00:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:58 smithi078 ceph-mon[99959]: pgmap v3031: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:00:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:00:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:00 smithi146 ceph-mon[103278]: pgmap v3032: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:01:00.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:00 smithi078 ceph-mon[91136]: pgmap v3032: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:01:00.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:00 smithi078 ceph-mon[99959]: pgmap v3032: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:01:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:01:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:01:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:00.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:00.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:00.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:02 smithi146 ceph-mon[103278]: pgmap v3033: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:02 smithi078 ceph-mon[99959]: pgmap v3033: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:02 smithi078 ceph-mon[91136]: pgmap v3033: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:04.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:03.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:04.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:03.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:04.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:03.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:04 smithi146 ceph-mon[103278]: pgmap v3034: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:04 smithi078 ceph-mon[91136]: pgmap v3034: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:04 smithi078 ceph-mon[99959]: pgmap v3034: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:05.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:05.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:05.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:06.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:06 smithi078 ceph-mon[91136]: pgmap v3035: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:01:06.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:06 smithi078 ceph-mon[99959]: pgmap v3035: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:01:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:06 smithi146 ceph-mon[103278]: pgmap v3035: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:01:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:08 smithi146 ceph-mon[103278]: pgmap v3036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T22:01:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:08 smithi078 ceph-mon[91136]: pgmap v3036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T22:01:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:08 smithi078 ceph-mon[99959]: pgmap v3036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-09-20T22:01:10.580 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[99959]: pgmap v3037: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:01:10.580 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.580 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.580 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[91136]: pgmap v3037: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:01:10.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.581 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:10.529Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:10.581 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:10.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:10.581 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:10.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:10 smithi146 ceph-mon[103278]: pgmap v3037: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:01:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:01:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:01:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:12 smithi146 ceph-mon[103278]: pgmap v3038: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:01:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:12 smithi078 ceph-mon[91136]: pgmap v3038: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:01:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:12 smithi078 ceph-mon[99959]: pgmap v3038: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:01:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:13.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:13.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:13.744 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:01:14.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:13.960Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:14.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:13.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:14.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:13.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:14.273 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:01:14.273 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 111m - - 2024-09-20T22:01:14.273 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 111m - - 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (105m) 9s ago 107m 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (106m) 4s ago 106m 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (99m) 9s ago 104m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (98m) 4s ago 111m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (100m) 9s ago 112m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (98m) 9s ago 112m 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (97m) 4s ago 111m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (97m) 9s ago 111m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (106m) 9s ago 106m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (106m) 4s ago 106m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (96m) 9s ago 110m 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (95m) 9s ago 110m 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (94m) 9s ago 109m 64.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (93m) 9s ago 109m 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:01:14.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (93m) 4s ago 109m 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (92m) 4s ago 108m 64.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (91m) 4s ago 108m 69.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (90m) 4s ago 107m 65.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (99m) 4s ago 107m 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (90m) 9s ago 105m 87.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (90m) 4s ago 105m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (90m) 9s ago 105m 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:01:14.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (90m) 4s ago 105m 88.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:01:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:14 smithi078 ceph-mon[91136]: pgmap v3039: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:01:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:14 smithi078 ceph-mon[91136]: from='client.70838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:14 smithi078 ceph-mon[99959]: pgmap v3039: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:01:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:14 smithi078 ceph-mon[99959]: from='client.70838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:14.583 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:01:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:01:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:01:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:01:14.584 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:01:14.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:14 smithi146 ceph-mon[103278]: pgmap v3039: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:01:14.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:14 smithi146 ceph-mon[103278]: from='client.70838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:01:14.849 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:01:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:01:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:01:14.850 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:01:14.850 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:01:15.159 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:01:15.159 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:15.159 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:15.159 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:15.159 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[91136]: from='client.50925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[91136]: from='client.70850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3140635901' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3548455865' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[99959]: from='client.50925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[99959]: from='client.70850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3140635901' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3548455865' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:15 smithi146 ceph-mon[103278]: from='client.50925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:15 smithi146 ceph-mon[103278]: from='client.70850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3140635901' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3548455865' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:16 smithi146 ceph-mon[103278]: from='client.70862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:16 smithi146 ceph-mon[103278]: pgmap v3040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:16 smithi078 ceph-mon[91136]: from='client.70862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:16 smithi078 ceph-mon[91136]: pgmap v3040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:16 smithi078 ceph-mon[99959]: from='client.70862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:16 smithi078 ceph-mon[99959]: pgmap v3040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:01:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:18 smithi146 ceph-mon[103278]: pgmap v3041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:01:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:18 smithi078 ceph-mon[91136]: pgmap v3041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:01:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:18 smithi078 ceph-mon[99959]: pgmap v3041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:01:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:20 smithi078 ceph-mon[99959]: pgmap v3042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T22:01:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:20 smithi078 ceph-mon[91136]: pgmap v3042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T22:01:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:20.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:20.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:20.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:20.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:20 smithi146 ceph-mon[103278]: pgmap v3042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-20T22:01:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:01:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:01:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:22 smithi146 ceph-mon[103278]: pgmap v3043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:01:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:22 smithi078 ceph-mon[91136]: pgmap v3043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:01:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:22 smithi078 ceph-mon[99959]: pgmap v3043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:01:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:23.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:23.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:23.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:24.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:24 smithi078 ceph-mon[91136]: pgmap v3044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T22:01:24.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:24 smithi078 ceph-mon[99959]: pgmap v3044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T22:01:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:24 smithi146 ceph-mon[103278]: pgmap v3044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-09-20T22:01:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:26 smithi078 ceph-mon[91136]: pgmap v3045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:01:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:26 smithi078 ceph-mon[99959]: pgmap v3045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:01:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:26 smithi146 ceph-mon[103278]: pgmap v3045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:01:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:28 smithi146 ceph-mon[103278]: pgmap v3046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:01:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:28 smithi078 ceph-mon[99959]: pgmap v3046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:01:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:28 smithi078 ceph-mon[91136]: pgmap v3046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:01:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:30 smithi146 ceph-mon[103278]: pgmap v3047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:01:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:01:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[91136]: pgmap v3047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[99959]: pgmap v3047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:30.531Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:30.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:30.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:30.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:30.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:32 smithi146 ceph-mon[103278]: pgmap v3048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:32 smithi078 ceph-mon[91136]: pgmap v3048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:32 smithi078 ceph-mon[99959]: pgmap v3048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:33.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:33.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:33.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:34.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:34 smithi146 ceph-mon[103278]: pgmap v3049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:34 smithi078 ceph-mon[91136]: pgmap v3049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:34 smithi078 ceph-mon[99959]: pgmap v3049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:36 smithi146 ceph-mon[103278]: pgmap v3050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:36 smithi078 ceph-mon[91136]: pgmap v3050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:36 smithi078 ceph-mon[99959]: pgmap v3050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:38 smithi146 ceph-mon[103278]: pgmap v3051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:38 smithi078 ceph-mon[99959]: pgmap v3051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:38 smithi078 ceph-mon[91136]: pgmap v3051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:40 smithi146 ceph-mon[103278]: pgmap v3052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:40 smithi078 ceph-mon[99959]: pgmap v3052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:40 smithi078 ceph-mon[91136]: pgmap v3052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:01:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:01:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:40.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:40.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:40.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:42 smithi146 ceph-mon[103278]: pgmap v3053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:42 smithi078 ceph-mon[91136]: pgmap v3053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:42 smithi078 ceph-mon[99959]: pgmap v3053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:43.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:43.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:43.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:44 smithi146 ceph-mon[103278]: pgmap v3054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:44 smithi078 ceph-mon[91136]: pgmap v3054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:44 smithi078 ceph-mon[99959]: pgmap v3054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:45.450 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:01:45.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:45.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:45.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 112m - - 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 112m - - 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (106m) 0s ago 107m 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (107m) 16s ago 107m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (99m) 0s ago 105m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:01:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (98m) 16s ago 111m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (100m) 0s ago 113m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (98m) 0s ago 113m 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (97m) 16s ago 111m 136M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (97m) 0s ago 111m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (107m) 0s ago 107m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (107m) 16s ago 107m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (96m) 0s ago 111m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (95m) 0s ago 110m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (95m) 0s ago 110m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (94m) 0s ago 110m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (93m) 16s ago 109m 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (92m) 16s ago 109m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:01:45.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (92m) 16s ago 108m 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:01:45.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (91m) 16s ago 108m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:01:45.995 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (100m) 16s ago 107m 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:01:45.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (90m) 0s ago 105m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:01:45.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (90m) 16s ago 105m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:01:45.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (90m) 0s ago 105m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:01:45.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (90m) 16s ago 105m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:01:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:01:46.307 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[91136]: pgmap v3055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[91136]: from='client.70874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[91136]: from='client.70880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3541938270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[99959]: pgmap v3055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[99959]: from='client.70874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[99959]: from='client.70880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:46.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3541938270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:46 smithi146 ceph-mon[103278]: pgmap v3055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:46 smithi146 ceph-mon[103278]: from='client.70874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:46 smithi146 ceph-mon[103278]: from='client.70880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3541938270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:01:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:01:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:46.905 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:01:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:46.905 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:01:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:47 smithi146 ceph-mon[103278]: from='client.50967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:47 smithi146 ceph-mon[103278]: from='client.70898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3615265001' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:47 smithi078 ceph-mon[99959]: from='client.50967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:47 smithi078 ceph-mon[99959]: from='client.70898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3615265001' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:47 smithi078 ceph-mon[91136]: from='client.50967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:47 smithi078 ceph-mon[91136]: from='client.70898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:01:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3615265001' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:01:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:48 smithi146 ceph-mon[103278]: pgmap v3056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:48 smithi078 ceph-mon[91136]: pgmap v3056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:48 smithi078 ceph-mon[99959]: pgmap v3056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:01:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:01:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[91136]: pgmap v3057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[99959]: pgmap v3057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:50.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:50.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:50.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:50.533Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:50.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:50.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:50 smithi146 ceph-mon[103278]: pgmap v3057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:01:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:52 smithi146 ceph-mon[103278]: pgmap v3058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:52 smithi078 ceph-mon[91136]: pgmap v3058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:52 smithi078 ceph-mon[99959]: pgmap v3058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:53.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:53.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:01:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:01:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:01:53.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:01:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:54 smithi146 ceph-mon[103278]: pgmap v3059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:54 smithi078 ceph-mon[91136]: pgmap v3059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:54 smithi078 ceph-mon[99959]: pgmap v3059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:01:56.683 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:56 smithi146 ceph-mon[103278]: pgmap v3060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:56 smithi078 ceph-mon[91136]: pgmap v3060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:56 smithi078 ceph-mon[99959]: pgmap v3060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:01:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:58 smithi146 ceph-mon[103278]: pgmap v3061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:58 smithi078 ceph-mon[91136]: pgmap v3061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:01:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:58 smithi078 ceph-mon[99959]: pgmap v3061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:01:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:00 smithi078 ceph-mon[91136]: pgmap v3062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:00 smithi078 ceph-mon[99959]: pgmap v3062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:02:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:02:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:00.533Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:00.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:00.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:00 smithi146 ceph-mon[103278]: pgmap v3062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:02 smithi146 ceph-mon[103278]: pgmap v3063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:02 smithi078 ceph-mon[99959]: pgmap v3063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:02 smithi078 ceph-mon[91136]: pgmap v3063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:03.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:03.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:03.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:04 smithi146 ceph-mon[103278]: pgmap v3064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:04 smithi078 ceph-mon[99959]: pgmap v3064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:04 smithi078 ceph-mon[91136]: pgmap v3064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:06.924 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:06 smithi078 ceph-mon[99959]: pgmap v3065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:06.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:06 smithi078 ceph-mon[91136]: pgmap v3065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:06 smithi146 ceph-mon[103278]: pgmap v3065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:08 smithi078 ceph-mon[91136]: pgmap v3066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:08 smithi078 ceph-mon[99959]: pgmap v3066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:08 smithi146 ceph-mon[103278]: pgmap v3066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[99959]: pgmap v3067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[91136]: pgmap v3067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:02:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:02:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:10.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:10.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:10.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:10 smithi146 ceph-mon[103278]: pgmap v3067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:12 smithi146 ceph-mon[103278]: pgmap v3068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:12 smithi078 ceph-mon[91136]: pgmap v3068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:12 smithi078 ceph-mon[99959]: pgmap v3068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:13.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:13.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:13.991 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:13.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:13.991 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:13.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:13.991 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:13.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:14 smithi146 ceph-mon[103278]: pgmap v3069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:14 smithi078 ceph-mon[91136]: pgmap v3069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:14 smithi078 ceph-mon[99959]: pgmap v3069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:16.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:16 smithi146 ceph-mon[103278]: pgmap v3070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:16 smithi078 ceph-mon[99959]: pgmap v3070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:16 smithi078 ceph-mon[91136]: pgmap v3070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:17.200 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 112m - - 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 112m - - 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (107m) 12s ago 108m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (107m) 8s ago 107m 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (100m) 12s ago 105m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:02:17.730 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (99m) 8s ago 112m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (101m) 12s ago 113m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (99m) 12s ago 113m 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (98m) 8s ago 112m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (98m) 12s ago 112m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (107m) 12s ago 107m 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (107m) 8s ago 107m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (97m) 12s ago 111m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (96m) 12s ago 111m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (95m) 12s ago 111m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (95m) 12s ago 110m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (94m) 8s ago 110m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (93m) 8s ago 109m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (92m) 8s ago 109m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:02:17.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (91m) 8s ago 109m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:02:17.732 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (100m) 8s ago 108m 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:02:17.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (91m) 12s ago 106m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:02:17.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (91m) 8s ago 106m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:02:17.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (91m) 12s ago 106m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:02:17.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (91m) 8s ago 106m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:02:18.039 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:02:18.040 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:02:18.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:02:18.041 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:02:18.305 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:02:18.305 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:02:18.305 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:02:18.305 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:02:18.305 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:02:18.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:02:18.306 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:02:18.616 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:02:18.616 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:18.616 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:18 smithi146 ceph-mon[103278]: pgmap v3071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:18 smithi146 ceph-mon[103278]: from='client.70910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:18 smithi146 ceph-mon[103278]: from='client.70916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:18 smithi146 ceph-mon[103278]: from='client.70922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3502461255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4196780354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[99959]: pgmap v3071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[99959]: from='client.70910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[99959]: from='client.70916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[99959]: from='client.70922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3502461255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4196780354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[91136]: pgmap v3071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[91136]: from='client.70910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[91136]: from='client.70916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[91136]: from='client.70922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3502461255' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4196780354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:19 smithi146 ceph-mon[103278]: from='client.51015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:19 smithi078 ceph-mon[91136]: from='client.51015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:19 smithi078 ceph-mon[99959]: from='client.51015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:20 smithi146 ceph-mon[103278]: pgmap v3072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:20 smithi078 ceph-mon[91136]: pgmap v3072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:20 smithi078 ceph-mon[99959]: pgmap v3072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:02:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:02:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:20.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:20.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:20.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:22 smithi146 ceph-mon[103278]: pgmap v3073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:22 smithi078 ceph-mon[99959]: pgmap v3073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:22 smithi078 ceph-mon[91136]: pgmap v3073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:23.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:23.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:23.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:24 smithi146 ceph-mon[103278]: pgmap v3074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:24 smithi078 ceph-mon[91136]: pgmap v3074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:24 smithi078 ceph-mon[99959]: pgmap v3074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:26 smithi078 ceph-mon[91136]: pgmap v3075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:26 smithi078 ceph-mon[99959]: pgmap v3075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:26 smithi146 ceph-mon[103278]: pgmap v3075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:28 smithi146 ceph-mon[103278]: pgmap v3076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:28 smithi078 ceph-mon[91136]: pgmap v3076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:28 smithi078 ceph-mon[99959]: pgmap v3076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[91136]: pgmap v3077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:02:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:02:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[99959]: pgmap v3077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:30.535Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:30.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:30.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:30 smithi146 ceph-mon[103278]: pgmap v3077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:32 smithi078 ceph-mon[91136]: pgmap v3078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:32 smithi078 ceph-mon[99959]: pgmap v3078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:32 smithi146 ceph-mon[103278]: pgmap v3078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:33.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:33.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:33.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:34 smithi078 ceph-mon[91136]: pgmap v3079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:34 smithi078 ceph-mon[99959]: pgmap v3079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:34 smithi146 ceph-mon[103278]: pgmap v3079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:02:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:36 smithi146 ceph-mon[103278]: pgmap v3080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:36 smithi078 ceph-mon[91136]: pgmap v3080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:36 smithi078 ceph-mon[99959]: pgmap v3080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:37.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:37 smithi146 ceph-mon[103278]: pgmap v3081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:37 smithi078 ceph-mon[99959]: pgmap v3081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:37 smithi078 ceph-mon[91136]: pgmap v3081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:39 smithi146 ceph-mon[103278]: pgmap v3082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:39 smithi078 ceph-mon[99959]: pgmap v3082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:39 smithi078 ceph-mon[91136]: pgmap v3082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:40.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:40.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:40.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:02:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:02:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:41 smithi146 ceph-mon[103278]: pgmap v3083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:41 smithi078 ceph-mon[99959]: pgmap v3083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:41 smithi078 ceph-mon[91136]: pgmap v3083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:43 smithi146 ceph-mon[103278]: pgmap v3084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-mon[99959]: pgmap v3084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-mon[91136]: pgmap v3084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:43.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:43.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:43.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:46 smithi146 ceph-mon[103278]: pgmap v3085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[91136]: pgmap v3085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[99959]: pgmap v3085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:48 smithi146 ceph-mon[103278]: pgmap v3086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:48 smithi078 ceph-mon[91136]: pgmap v3086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:48 smithi078 ceph-mon[99959]: pgmap v3086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:48.920 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 113m - - 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 113m - - 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (107m) 4s ago 108m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (108m) 19s ago 108m 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (100m) 4s ago 106m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (99m) 19s ago 112m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (101m) 4s ago 114m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (99m) 4s ago 114m 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (98m) 19s ago 113m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:02:49.486 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (99m) 4s ago 112m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (108m) 4s ago 108m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (108m) 19s ago 108m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (97m) 4s ago 112m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (96m) 4s ago 111m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (96m) 4s ago 111m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (95m) 4s ago 111m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (95m) 19s ago 110m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (93m) 19s ago 110m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (93m) 19s ago 109m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (92m) 19s ago 109m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (101m) 19s ago 108m 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (91m) 4s ago 106m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:02:49.487 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (91m) 19s ago 106m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:02:49.488 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (91m) 4s ago 106m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:02:49.488 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (91m) 19s ago 106m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:02:49.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:02:49.813 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:02:50.091 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:02:50.092 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:02:50.092 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:02:50.404 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:02:50.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:50.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:02:50.405 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:50.405 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: pgmap v3087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='client.51027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='client.51033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='client.51039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3557757610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/325616155' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: pgmap v3087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='client.51027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='client.51033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='client.51039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3557757610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/325616155' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: pgmap v3087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:50.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='client.51027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='client.51033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='client.51039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:50.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3557757610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:02:50.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:02:50.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/325616155' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:02:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:02:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:02:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:50.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:50.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:50.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:51 smithi146 ceph-mon[103278]: from='client.51051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:51 smithi078 ceph-mon[91136]: from='client.51051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:51 smithi078 ceph-mon[99959]: from='client.51051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:02:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:52 smithi146 ceph-mon[103278]: pgmap v3088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:52 smithi078 ceph-mon[91136]: pgmap v3088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:52 smithi078 ceph-mon[99959]: pgmap v3088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:53.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:53.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:02:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:02:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:02:53.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:02:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:54 smithi146 ceph-mon[103278]: pgmap v3089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:54 smithi078 ceph-mon[91136]: pgmap v3089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:54 smithi078 ceph-mon[99959]: pgmap v3089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:02:56.698 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:56 smithi146 ceph-mon[103278]: pgmap v3090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:56 smithi078 ceph-mon[91136]: pgmap v3090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:56 smithi078 ceph-mon[99959]: pgmap v3090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:02:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:58 smithi146 ceph-mon[103278]: pgmap v3091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:02:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:58 smithi078 ceph-mon[91136]: pgmap v3091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:02:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:02:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:58 smithi078 ceph-mon[99959]: pgmap v3091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:02:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:02:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:00 smithi146 ceph-mon[103278]: pgmap v3092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:00 smithi078 ceph-mon[91136]: pgmap v3092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:00 smithi078 ceph-mon[99959]: pgmap v3092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:03:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:03:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:00.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:00.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:00.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:02 smithi146 ceph-mon[103278]: pgmap v3093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:02 smithi078 ceph-mon[91136]: pgmap v3093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:02 smithi078 ceph-mon[99959]: pgmap v3093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:03.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:03.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:03.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:04 smithi078 ceph-mon[91136]: pgmap v3094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:03:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:04 smithi078 ceph-mon[99959]: pgmap v3094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:03:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:04 smithi146 ceph-mon[103278]: pgmap v3094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:03:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:06 smithi146 ceph-mon[103278]: pgmap v3095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[91136]: pgmap v3095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[99959]: pgmap v3095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:08.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:08 smithi078 ceph-mon[99959]: pgmap v3096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:08.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:08 smithi078 ceph-mon[91136]: pgmap v3096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:08 smithi146 ceph-mon[103278]: pgmap v3096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:11.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[91136]: pgmap v3097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:11.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.006 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:03:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:03:11.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[99959]: pgmap v3097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:10.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:11.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:10.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:11.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:10.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:10 smithi146 ceph-mon[103278]: pgmap v3097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:12 smithi146 ceph-mon[103278]: pgmap v3098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:12 smithi078 ceph-mon[91136]: pgmap v3098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:12 smithi078 ceph-mon[99959]: pgmap v3098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:13.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:13.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:13.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:13.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:13.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:14 smithi146 ceph-mon[103278]: pgmap v3099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:14 smithi078 ceph-mon[91136]: pgmap v3099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:14 smithi078 ceph-mon[99959]: pgmap v3099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:16.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:16 smithi146 ceph-mon[103278]: pgmap v3100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:16 smithi078 ceph-mon[91136]: pgmap v3100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:16 smithi078 ceph-mon[99959]: pgmap v3100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:18 smithi146 ceph-mon[103278]: pgmap v3101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:18 smithi078 ceph-mon[91136]: pgmap v3101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:18 smithi078 ceph-mon[99959]: pgmap v3101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:20.699 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:03:20.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:20 smithi146 ceph-mon[103278]: pgmap v3102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:20.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:20 smithi078 ceph-mon[99959]: pgmap v3102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:20.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:20 smithi078 ceph-mon[91136]: pgmap v3102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:20.943 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:03:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:03:20.944 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:20.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:20.944 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:20.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:20.944 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:20.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:21.247 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 113m - - 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 113m - - 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (108m) 15s ago 109m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (108m) 11s ago 108m 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (101m) 15s ago 106m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (100m) 11s ago 113m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (102m) 15s ago 114m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (100m) 15s ago 115m 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (99m) 11s ago 113m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (99m) 15s ago 113m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (108m) 15s ago 108m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (108m) 11s ago 108m 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:03:21.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (98m) 15s ago 112m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (97m) 15s ago 112m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (96m) 15s ago 112m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (96m) 15s ago 111m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (95m) 11s ago 111m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (94m) 11s ago 110m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (93m) 11s ago 110m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (92m) 11s ago 110m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (101m) 11s ago 109m 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (92m) 15s ago 107m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (92m) 11s ago 107m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (92m) 15s ago 107m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:03:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (92m) 11s ago 107m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:21.560 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:03:21.561 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:03:21.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:03:21.561 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:03:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:21 smithi078 ceph-mon[99959]: from='client.70982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1974636145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:21 smithi078 ceph-mon[91136]: from='client.70982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1974636145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:21.842 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:03:21.842 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:03:21.843 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:03:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:21 smithi146 ceph-mon[103278]: from='client.70982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1974636145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:22.153 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:03:22.154 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:22.154 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:22.154 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:22.154 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:22 smithi146 ceph-mon[103278]: pgmap v3103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:22 smithi146 ceph-mon[103278]: from='client.70988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:22 smithi146 ceph-mon[103278]: from='client.70994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3690661868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:23.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[99959]: pgmap v3103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:23.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[99959]: from='client.70988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[99959]: from='client.70994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3690661868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:23.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[91136]: pgmap v3103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:23.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[91136]: from='client.70988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[91136]: from='client.70994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3690661868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:23.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:23 smithi078 ceph-mon[99959]: from='client.71006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:23.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:23 smithi078 ceph-mon[91136]: from='client.71006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:23 smithi146 ceph-mon[103278]: from='client.71006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:23.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:23.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:23.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:24 smithi146 ceph-mon[103278]: pgmap v3104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:24 smithi078 ceph-mon[99959]: pgmap v3104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:24 smithi078 ceph-mon[91136]: pgmap v3104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:26 smithi146 ceph-mon[103278]: pgmap v3105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[99959]: pgmap v3105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[91136]: pgmap v3105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:28 smithi146 ceph-mon[103278]: pgmap v3106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:28 smithi078 ceph-mon[91136]: pgmap v3106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:28 smithi078 ceph-mon[99959]: pgmap v3106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:30.691 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:03:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:03:30.691 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:30.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:30.691 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:30.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:30.691 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:30.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:31.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:30 smithi146 ceph-mon[103278]: pgmap v3107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:31.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[91136]: pgmap v3107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[99959]: pgmap v3107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:32 smithi146 ceph-mon[103278]: pgmap v3108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:32 smithi078 ceph-mon[91136]: pgmap v3108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:32 smithi078 ceph-mon[99959]: pgmap v3108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:33.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:33.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:33.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:34 smithi078 ceph-mon[91136]: pgmap v3109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:34 smithi078 ceph-mon[99959]: pgmap v3109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:34 smithi146 ceph-mon[103278]: pgmap v3109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:36 smithi078 ceph-mon[91136]: pgmap v3110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:36 smithi078 ceph-mon[99959]: pgmap v3110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:36 smithi146 ceph-mon[103278]: pgmap v3110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:38 smithi146 ceph-mon[103278]: pgmap v3111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:38 smithi078 ceph-mon[91136]: pgmap v3111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:38 smithi078 ceph-mon[99959]: pgmap v3111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:40.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:40 smithi078 ceph-mon[99959]: pgmap v3112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:40.838 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:03:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:03:40.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:40 smithi078 ceph-mon[91136]: pgmap v3112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:40.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:40.539Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:40.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:40.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:40.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:40.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:40 smithi146 ceph-mon[103278]: pgmap v3112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:42 smithi078 ceph-mon[91136]: pgmap v3113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:42 smithi078 ceph-mon[99959]: pgmap v3113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:42 smithi146 ceph-mon[103278]: pgmap v3113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:44.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:44.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:44.067 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:43.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:44.067 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:43.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:44.067 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:43.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:44 smithi146 ceph-mon[103278]: pgmap v3114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:44 smithi078 ceph-mon[91136]: pgmap v3114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:44 smithi078 ceph-mon[99959]: pgmap v3114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:46.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:46 smithi146 ceph-mon[103278]: pgmap v3115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[91136]: pgmap v3115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[99959]: pgmap v3115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:48 smithi146 ceph-mon[103278]: pgmap v3116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:48 smithi078 ceph-mon[91136]: pgmap v3116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:48 smithi078 ceph-mon[99959]: pgmap v3116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:50.801 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:03:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:03:50.801 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:50.539Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:50.801 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:50.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:50.801 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:50.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:51.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[91136]: pgmap v3117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[99959]: pgmap v3117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:50 smithi146 ceph-mon[103278]: pgmap v3117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:03:52.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:51 smithi078 ceph-mon[99959]: pgmap v3118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:52.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:51 smithi078 ceph-mon[91136]: pgmap v3118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:51 smithi146 ceph-mon[103278]: pgmap v3118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 114m - - 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 114m - - 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (108m) 7s ago 109m 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (109m) 2s ago 109m 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (102m) 7s ago 107m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:03:52.976 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (100m) 2s ago 113m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (102m) 7s ago 115m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (101m) 7s ago 115m 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (99m) 2s ago 114m 140M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (100m) 7s ago 113m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (109m) 7s ago 109m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (109m) 2s ago 109m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (98m) 7s ago 113m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (97m) 7s ago 112m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (97m) 7s ago 112m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (96m) 7s ago 112m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (96m) 2s ago 111m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (95m) 2s ago 111m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (94m) 2s ago 111m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (93m) 2s ago 110m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:03:52.977 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (102m) 2s ago 110m 76.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:03:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (93m) 7s ago 107m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:03:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (92m) 2s ago 107m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:03:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (92m) 7s ago 107m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:03:52.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (92m) 2s ago 107m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:03:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:52 smithi146 ceph-mon[103278]: from='client.71018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:52 smithi146 ceph-mon[103278]: from='client.51105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:52 smithi078 ceph-mon[99959]: from='client.71018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:52 smithi078 ceph-mon[99959]: from='client.51105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:52 smithi078 ceph-mon[91136]: from='client.71018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:52 smithi078 ceph-mon[91136]: from='client.51105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:03:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:03:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:03:53.553 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:03:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:03:53.555 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:03:53.867 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:03:53.868 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:53.868 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:03:53.868 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:53.868 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:03:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:53 smithi146 ceph-mon[103278]: pgmap v3119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:53 smithi146 ceph-mon[103278]: from='client.71030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2381309658' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:53 smithi146 ceph-mon[103278]: from='client.71042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[91136]: pgmap v3119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[91136]: from='client.71030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2381309658' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[91136]: from='client.71042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[99959]: pgmap v3119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[99959]: from='client.71030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2381309658' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:03:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-mon[99959]: from='client.71042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:03:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:53.973Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:53.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:03:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:03:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:03:53.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:03:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3494499778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3494499778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3494499778' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:03:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:03:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:55 smithi146 ceph-mon[103278]: pgmap v3120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:55 smithi078 ceph-mon[91136]: pgmap v3120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:55 smithi078 ceph-mon[99959]: pgmap v3120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:03:58.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:57 smithi146 ceph-mon[103278]: pgmap v3121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:57 smithi078 ceph-mon[91136]: pgmap v3121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:57 smithi078 ceph-mon[99959]: pgmap v3121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:03:59.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:03:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:03:59 smithi146 ceph-mon[103278]: pgmap v3122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:03:59 smithi078 ceph-mon[99959]: pgmap v3122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:03:59 smithi078 ceph-mon[91136]: pgmap v3122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:04:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:04:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:00.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:00.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:00.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:01 smithi146 ceph-mon[103278]: pgmap v3123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:01 smithi078 ceph-mon[91136]: pgmap v3123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:01 smithi078 ceph-mon[99959]: pgmap v3123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:03 smithi146 ceph-mon[103278]: pgmap v3124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:03 smithi078 ceph-mon[91136]: pgmap v3124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:03 smithi078 ceph-mon[99959]: pgmap v3124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:03.973Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:03.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:03.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:05 smithi146 ceph-mon[103278]: pgmap v3125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:05 smithi078 ceph-mon[91136]: pgmap v3125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:05 smithi078 ceph-mon[99959]: pgmap v3125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:08 smithi078 ceph-mon[99959]: pgmap v3126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:08 smithi078 ceph-mon[91136]: pgmap v3126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:08 smithi146 ceph-mon[103278]: pgmap v3126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:10 smithi078 ceph-mon[91136]: pgmap v3127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:10 smithi078 ceph-mon[99959]: pgmap v3127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:10 smithi146 ceph-mon[103278]: pgmap v3127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:10.854 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:04:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:04:10.854 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:10.541Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:10.855 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:10.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:10.855 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:10.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:12 smithi146 ceph-mon[103278]: pgmap v3128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:12 smithi078 ceph-mon[99959]: pgmap v3128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:12 smithi078 ceph-mon[91136]: pgmap v3128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:13.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:13.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:13.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:14 smithi078 ceph-mon[91136]: pgmap v3129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:14 smithi078 ceph-mon[99959]: pgmap v3129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:14 smithi146 ceph-mon[103278]: pgmap v3129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:16 smithi078 ceph-mon[99959]: pgmap v3130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:16 smithi078 ceph-mon[91136]: pgmap v3130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:16.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:16 smithi146 ceph-mon[103278]: pgmap v3130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:18 smithi146 ceph-mon[103278]: pgmap v3131: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:18 smithi078 ceph-mon[91136]: pgmap v3131: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:18 smithi078 ceph-mon[99959]: pgmap v3131: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:20.642 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:20 smithi146 ceph-mon[103278]: pgmap v3132: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:20 smithi078 ceph-mon[91136]: pgmap v3132: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:20 smithi078 ceph-mon[99959]: pgmap v3132: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:04:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:04:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:20.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:20.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:20.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:22 smithi146 ceph-mon[103278]: pgmap v3133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:22 smithi078 ceph-mon[91136]: pgmap v3133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:22 smithi078 ceph-mon[99959]: pgmap v3133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:04:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:23.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:23.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:23.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:04:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 115m - - 2024-09-20T22:04:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 114m - - 2024-09-20T22:04:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (109m) 18s ago 110m 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:04:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (110m) 14s ago 110m 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:04:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (102m) 18s ago 107m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (101m) 14s ago 114m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (103m) 18s ago 115m 563M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (101m) 18s ago 116m 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (100m) 14s ago 114m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (100m) 18s ago 114m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (109m) 18s ago 109m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (109m) 14s ago 109m 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (99m) 18s ago 113m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (98m) 18s ago 113m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (97m) 18s ago 113m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (97m) 18s ago 112m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (96m) 14s ago 112m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (95m) 14s ago 111m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (94m) 14s ago 111m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:04:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (93m) 14s ago 111m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:04:24.692 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (102m) 14s ago 110m 76.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:04:24.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (93m) 18s ago 108m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:04:24.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (93m) 14s ago 108m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:04:24.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (93m) 18s ago 108m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:04:24.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (93m) 14s ago 108m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:04:25.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:24 smithi078 ceph-mon[91136]: pgmap v3134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:25.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:24 smithi078 ceph-mon[99959]: pgmap v3134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:04:25.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:04:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:04:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:04:25.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:04:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:24 smithi146 ceph-mon[103278]: pgmap v3134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:25.269 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:04:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:04:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:04:25.270 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:04:25.581 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:04:25.581 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:25.581 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:25.581 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:25.582 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:25.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:25.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:25 smithi146 ceph-mon[103278]: from='client.51135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:25.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:25 smithi146 ceph-mon[103278]: from='client.51141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:25.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:25 smithi146 ceph-mon[103278]: from='client.51147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:25.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1263980404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[91136]: from='client.51135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[91136]: from='client.51141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[91136]: from='client.51147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1263980404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[99959]: from='client.51135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[99959]: from='client.51141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[99959]: from='client.51147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1263980404' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 22:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T22:04:25.847131261Z level=info msg="Completed cleanup jobs" duration=11.906029ms 2024-09-20T22:04:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:26 smithi146 ceph-mon[103278]: pgmap v3135: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:26 smithi146 ceph-mon[103278]: from='client.51159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/291174201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[91136]: pgmap v3135: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[91136]: from='client.51159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/291174201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[99959]: pgmap v3135: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[99959]: from='client.51159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/291174201' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:28 smithi146 ceph-mon[103278]: pgmap v3136: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:28 smithi078 ceph-mon[91136]: pgmap v3136: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:28 smithi078 ceph-mon[99959]: pgmap v3136: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:30 smithi146 ceph-mon[103278]: pgmap v3137: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:30 smithi078 ceph-mon[91136]: pgmap v3137: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:30 smithi078 ceph-mon[99959]: pgmap v3137: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:04:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T22:04:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:30.543Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:30.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:30.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:31.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:31.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:31.685 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:31.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:31.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:31.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:32.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:32 smithi078 ceph-mon[99959]: pgmap v3138: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:32.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:32 smithi078 ceph-mon[91136]: pgmap v3138: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:32 smithi146 ceph-mon[103278]: pgmap v3138: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:34.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:33.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:34.104 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:33.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:34.104 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:33.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:34 smithi146 ceph-mon[103278]: pgmap v3139: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:34 smithi078 ceph-mon[91136]: pgmap v3139: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:34 smithi078 ceph-mon[99959]: pgmap v3139: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:36 smithi146 ceph-mon[103278]: pgmap v3140: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:36 smithi078 ceph-mon[91136]: pgmap v3140: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:36 smithi078 ceph-mon[99959]: pgmap v3140: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:38 smithi146 ceph-mon[103278]: pgmap v3141: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:38 smithi078 ceph-mon[91136]: pgmap v3141: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:38 smithi078 ceph-mon[99959]: pgmap v3141: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:40 smithi146 ceph-mon[103278]: pgmap v3142: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:40.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:40 smithi078 ceph-mon[99959]: pgmap v3142: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:40.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:40 smithi078 ceph-mon[91136]: pgmap v3142: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:40.839 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:04:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T22:04:40.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:40.544Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:40.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:40.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:40.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:40.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:42 smithi146 ceph-mon[103278]: pgmap v3143: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:43.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:42 smithi078 ceph-mon[99959]: pgmap v3143: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:43.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:42 smithi078 ceph-mon[91136]: pgmap v3143: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:43.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:43.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:43.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:43.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:43.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:44 smithi146 ceph-mon[103278]: pgmap v3144: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:44 smithi078 ceph-mon[91136]: pgmap v3144: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:44 smithi078 ceph-mon[99959]: pgmap v3144: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:46 smithi146 ceph-mon[103278]: pgmap v3145: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[91136]: pgmap v3145: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[99959]: pgmap v3145: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:48 smithi146 ceph-mon[103278]: pgmap v3146: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:48 smithi078 ceph-mon[91136]: pgmap v3146: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:48 smithi078 ceph-mon[99959]: pgmap v3146: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:50 smithi146 ceph-mon[103278]: pgmap v3147: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:50 smithi078 ceph-mon[91136]: pgmap v3147: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:50 smithi078 ceph-mon[99959]: pgmap v3147: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:04:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:04:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:50.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:50.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:50.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:04:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:52 smithi078 ceph-mon[91136]: pgmap v3148: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:52 smithi078 ceph-mon[99959]: pgmap v3148: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:52 smithi146 ceph-mon[103278]: pgmap v3148: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:53.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:53.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:04:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:04:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:04:53.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:04:54.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:54 smithi078 ceph-mon[99959]: pgmap v3149: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:54.898 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:54 smithi078 ceph-mon[91136]: pgmap v3149: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:54 smithi146 ceph-mon[103278]: pgmap v3149: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:55.874 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:04:56.078 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:56.078 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:04:56.407 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:04:56.407 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 115m - - 2024-09-20T22:04:56.407 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 115m - - 2024-09-20T22:04:56.407 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (109m) 10s ago 110m 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:04:56.407 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (110m) 5s ago 110m 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (103m) 10s ago 108m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (101m) 5s ago 114m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (103m) 10s ago 116m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (102m) 10s ago 116m 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (100m) 5s ago 115m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (101m) 10s ago 115m 146M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (110m) 10s ago 110m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (110m) 5s ago 110m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (99m) 10s ago 114m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (98m) 10s ago 114m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (98m) 10s ago 113m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (97m) 10s ago 113m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (97m) 5s ago 112m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (96m) 5s ago 112m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:04:56.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (95m) 5s ago 112m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:04:56.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (94m) 5s ago 111m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:04:56.409 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (103m) 5s ago 111m 76.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:04:56.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (94m) 10s ago 108m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:04:56.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (93m) 5s ago 108m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:04:56.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (94m) 10s ago 108m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:04:56.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (93m) 5s ago 108m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:04:56.720 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:04:56.720 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:04:56.720 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:04:56.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:04:56.721 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:04:56.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:04:56.722 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:04:56.722 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:04:56.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:04:56.722 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:04:56.987 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:04:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:04:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:04:56.988 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:04:56.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:56 smithi078 ceph-mon[99959]: pgmap v3150: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:56.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3151280676' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:56.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:56 smithi078 ceph-mon[91136]: pgmap v3150: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:56.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3151280676' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:56 smithi146 ceph-mon[103278]: pgmap v3150: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:04:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3151280676' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:04:57.297 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:04:57.297 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:57.297 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:04:57.297 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:57.297 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:04:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:57 smithi146 ceph-mon[103278]: from='client.51171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:57 smithi146 ceph-mon[103278]: from='client.71096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:57 smithi146 ceph-mon[103278]: from='client.51183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2339932471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[91136]: from='client.51171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[91136]: from='client.71096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[91136]: from='client.51183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2339932471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[99959]: from='client.51171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[99959]: from='client.71096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[99959]: from='client.51183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2339932471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:04:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:58 smithi146 ceph-mon[103278]: pgmap v3151: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:58 smithi146 ceph-mon[103278]: from='client.71114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:58 smithi078 ceph-mon[91136]: pgmap v3151: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:58 smithi078 ceph-mon[91136]: from='client.71114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:04:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:58 smithi078 ceph-mon[99959]: pgmap v3151: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:04:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:58 smithi078 ceph-mon[99959]: from='client.71114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:04:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:00.798 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:05:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:05:00.798 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:00.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:00.798 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:00.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:00.798 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:00.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:01.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:00 smithi146 ceph-mon[103278]: pgmap v3152: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:00 smithi078 ceph-mon[91136]: pgmap v3152: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:00 smithi078 ceph-mon[99959]: pgmap v3152: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:02 smithi146 ceph-mon[103278]: pgmap v3153: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:02 smithi078 ceph-mon[91136]: pgmap v3153: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:02 smithi078 ceph-mon[99959]: pgmap v3153: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:03.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:03.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:03.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:04 smithi146 ceph-mon[103278]: pgmap v3154: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:04 smithi078 ceph-mon[91136]: pgmap v3154: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:04 smithi078 ceph-mon[99959]: pgmap v3154: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:06 smithi146 ceph-mon[103278]: pgmap v3155: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[91136]: pgmap v3155: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[99959]: pgmap v3155: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:08 smithi146 ceph-mon[103278]: pgmap v3156: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:08 smithi078 ceph-mon[91136]: pgmap v3156: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:08 smithi078 ceph-mon[99959]: pgmap v3156: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:10.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:09 smithi078 ceph-mon[99959]: pgmap v3157: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:10.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:09 smithi078 ceph-mon[91136]: pgmap v3157: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:09 smithi146 ceph-mon[103278]: pgmap v3157: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:05:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:05:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:10.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:10.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:10.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:11 smithi146 ceph-mon[103278]: pgmap v3158: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[91136]: pgmap v3158: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:12.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:11 smithi078 ceph-mon[99959]: pgmap v3158: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:14.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-mon[91136]: pgmap v3159: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:14.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:14.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-mon[99959]: pgmap v3159: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:14.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:14.137 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:13.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:14.137 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:13.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:14.137 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:13.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:13 smithi146 ceph-mon[103278]: pgmap v3159: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:15.199 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:15.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:15 smithi146 ceph-mon[103278]: pgmap v3160: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:15 smithi078 ceph-mon[91136]: pgmap v3160: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:15 smithi078 ceph-mon[99959]: pgmap v3160: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:18 smithi146 ceph-mon[103278]: pgmap v3161: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:18 smithi078 ceph-mon[91136]: pgmap v3161: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:18 smithi078 ceph-mon[99959]: pgmap v3161: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:20 smithi146 ceph-mon[103278]: pgmap v3162: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:20 smithi078 ceph-mon[91136]: pgmap v3162: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:20 smithi078 ceph-mon[99959]: pgmap v3162: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:20.868 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:05:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:05:20.868 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:20.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:20.868 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:20.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:20.868 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:20.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:22 smithi146 ceph-mon[103278]: pgmap v3163: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:22 smithi078 ceph-mon[91136]: pgmap v3163: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:22 smithi078 ceph-mon[99959]: pgmap v3163: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:24 smithi146 ceph-mon[103278]: pgmap v3164: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:24 smithi078 ceph-mon[91136]: pgmap v3164: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:24 smithi078 ceph-mon[99959]: pgmap v3164: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:23.980Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:23.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:23.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:26 smithi078 ceph-mon[91136]: pgmap v3165: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:26 smithi078 ceph-mon[99959]: pgmap v3165: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:26 smithi146 ceph-mon[103278]: pgmap v3165: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:27.594 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:05:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:28.119 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:05:28.119 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 116m - - 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 115m - - 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (110m) 1s ago 111m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (111m) 17s ago 111m 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (103m) 1s ago 109m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (102m) 17s ago 115m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (104m) 1s ago 116m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (102m) 1s ago 117m 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (101m) 17s ago 115m 138M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (101m) 1s ago 115m 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (110m) 1s ago 110m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (110m) 17s ago 110m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (100m) 1s ago 114m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (99m) 1s ago 114m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:05:28.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (98m) 1s ago 114m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (98m) 1s ago 113m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (97m) 17s ago 113m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (96m) 17s ago 112m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (95m) 17s ago 112m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (94m) 17s ago 112m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (103m) 17s ago 111m 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (94m) 1s ago 109m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (94m) 17s ago 109m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (94m) 1s ago 109m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:05:28.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (94m) 17s ago 109m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:05:28.446 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:05:28.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:05:28.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:05:28.715 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:05:28.715 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:05:28.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:05:28.716 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[99959]: pgmap v3166: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[99959]: from='client.71126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2003578622' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[91136]: pgmap v3166: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[91136]: from='client.71126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:28.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2003578622' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:28 smithi146 ceph-mon[103278]: pgmap v3166: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:28 smithi146 ceph-mon[103278]: from='client.71126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2003578622' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:05:29.022 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:05:29.022 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:05:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:05:29.022 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:05:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:05:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[91136]: from='client.51213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[91136]: from='client.71138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[91136]: from='client.51231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4075547269' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[99959]: from='client.51213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[99959]: from='client.71138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[99959]: from='client.51231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4075547269' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:29 smithi146 ceph-mon[103278]: from='client.51213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:29 smithi146 ceph-mon[103278]: from='client.71138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:29 smithi146 ceph-mon[103278]: from='client.51231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:05:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4075547269' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:05:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:30 smithi078 ceph-mon[91136]: pgmap v3167: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:30 smithi078 ceph-mon[99959]: pgmap v3167: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:05:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T22:05:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:30.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:30.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:30.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:30 smithi146 ceph-mon[103278]: pgmap v3167: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[91136]: pgmap v3168: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[99959]: pgmap v3168: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:32 smithi146 ceph-mon[103278]: pgmap v3168: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:32.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:32.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:34 smithi078 ceph-mon[99959]: pgmap v3169: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:34 smithi078 ceph-mon[91136]: pgmap v3169: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:33.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:33.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:33.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:34 smithi146 ceph-mon[103278]: pgmap v3169: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:36 smithi078 ceph-mon[99959]: pgmap v3170: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:36 smithi078 ceph-mon[91136]: pgmap v3170: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:36 smithi146 ceph-mon[103278]: pgmap v3170: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:38 smithi146 ceph-mon[103278]: pgmap v3171: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:38 smithi078 ceph-mon[91136]: pgmap v3171: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:38 smithi078 ceph-mon[99959]: pgmap v3171: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:40.338 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:40 smithi078 ceph-mon[99959]: pgmap v3172: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:40.338 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:40 smithi078 ceph-mon[91136]: pgmap v3172: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:40 smithi146 ceph-mon[103278]: pgmap v3172: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:40.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:40.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:40.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:40.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:05:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T22:05:42.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:42 smithi146 ceph-mon[103278]: pgmap v3173: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:42 smithi078 ceph-mon[91136]: pgmap v3173: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:42 smithi078 ceph-mon[99959]: pgmap v3173: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:44 smithi078 ceph-mon[91136]: pgmap v3174: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:44 smithi078 ceph-mon[99959]: pgmap v3174: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:43.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:43.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:43.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:44 smithi146 ceph-mon[103278]: pgmap v3174: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:46.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:46 smithi146 ceph-mon[103278]: pgmap v3175: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:46 smithi078 ceph-mon[91136]: pgmap v3175: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:46 smithi078 ceph-mon[99959]: pgmap v3175: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:48 smithi146 ceph-mon[103278]: pgmap v3176: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:48 smithi078 ceph-mon[91136]: pgmap v3176: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:48 smithi078 ceph-mon[99959]: pgmap v3176: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:50 smithi146 ceph-mon[103278]: pgmap v3177: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:50 smithi078 ceph-mon[91136]: pgmap v3177: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:50 smithi078 ceph-mon[99959]: pgmap v3177: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:05:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:05:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:50.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:50.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:50.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:52 smithi146 ceph-mon[103278]: pgmap v3178: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[99959]: pgmap v3178: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[91136]: pgmap v3178: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:05:54.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:54 smithi078 ceph-mon[91136]: pgmap v3179: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:54.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:54 smithi078 ceph-mon[99959]: pgmap v3179: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:54.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:53.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:54.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:53.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:05:54.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:05:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:05:53.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:05:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:54 smithi146 ceph-mon[103278]: pgmap v3179: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:55.268 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:55.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:05:56.436 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:56 smithi078 ceph-mon[91136]: pgmap v3180: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:56.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:56 smithi078 ceph-mon[99959]: pgmap v3180: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:56 smithi146 ceph-mon[103278]: pgmap v3180: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:05:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:58 smithi146 ceph-mon[103278]: pgmap v3181: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:58 smithi078 ceph-mon[99959]: pgmap v3181: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:58 smithi078 ceph-mon[91136]: pgmap v3181: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:05:59.321 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:05:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:05:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:05:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:05:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:05:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:05:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 116m - - 2024-09-20T22:05:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 116m - - 2024-09-20T22:05:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (110m) 13s ago 111m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (111m) 8s ago 111m 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (104m) 13s ago 109m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (102m) 8s ago 115m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (104m) 13s ago 117m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (103m) 13s ago 117m 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (101m) 8s ago 116m 139M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (102m) 13s ago 116m 146M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (111m) 13s ago 111m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (111m) 8s ago 111m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (100m) 13s ago 115m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (99m) 13s ago 115m 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (99m) 13s ago 114m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (98m) 13s ago 114m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:05:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (98m) 8s ago 113m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (97m) 8s ago 113m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (96m) 8s ago 113m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (95m) 8s ago 112m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (104m) 8s ago 112m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (95m) 13s ago 109m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (94m) 8s ago 110m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (95m) 13s ago 109m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:05:59.850 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (94m) 8s ago 109m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:06:00.158 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:06:00.427 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:06:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:06:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:06:00.427 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:06:00.428 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:06:00.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[99959]: pgmap v3182: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[99959]: from='client.51243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[99959]: from='client.51249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3925662153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[91136]: pgmap v3182: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[91136]: from='client.51243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[91136]: from='client.51249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:00.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3925662153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:00 smithi146 ceph-mon[103278]: pgmap v3182: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:00 smithi146 ceph-mon[103278]: from='client.51243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:00 smithi146 ceph-mon[103278]: from='client.51249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3925662153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:00.741 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:06:00.741 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:00.741 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:00.741 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:06:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:06:00.742 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:00.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:00.742 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:00.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:00.742 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:00.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:01 smithi146 ceph-mon[103278]: from='client.51255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:01 smithi146 ceph-mon[103278]: from='client.51267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3695492841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:01 smithi078 ceph-mon[99959]: from='client.51255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:01 smithi078 ceph-mon[99959]: from='client.51267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3695492841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:01 smithi078 ceph-mon[91136]: from='client.51255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:01 smithi078 ceph-mon[91136]: from='client.51267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3695492841' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:02 smithi146 ceph-mon[103278]: pgmap v3183: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:02 smithi078 ceph-mon[91136]: pgmap v3183: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:02 smithi078 ceph-mon[99959]: pgmap v3183: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:04 smithi078 ceph-mon[91136]: pgmap v3184: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:04 smithi078 ceph-mon[99959]: pgmap v3184: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:03.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:03.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:03.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:04 smithi146 ceph-mon[103278]: pgmap v3184: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:06 smithi146 ceph-mon[103278]: pgmap v3185: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:06 smithi078 ceph-mon[91136]: pgmap v3185: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:06 smithi078 ceph-mon[99959]: pgmap v3185: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:08 smithi078 ceph-mon[91136]: pgmap v3186: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:08 smithi078 ceph-mon[99959]: pgmap v3186: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:08 smithi146 ceph-mon[103278]: pgmap v3186: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:10.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:10 smithi078 ceph-mon[99959]: pgmap v3187: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:10.838 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:06:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:06:10.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:10 smithi078 ceph-mon[91136]: pgmap v3187: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:10.551Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:10.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:10.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:10 smithi146 ceph-mon[103278]: pgmap v3187: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:12 smithi146 ceph-mon[103278]: pgmap v3188: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[99959]: pgmap v3188: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[91136]: pgmap v3188: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:13.985Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:13.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:13.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:14 smithi146 ceph-mon[103278]: pgmap v3189: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:14.569 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:14 smithi078 ceph-mon[91136]: pgmap v3189: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:14.569 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:14 smithi078 ceph-mon[99959]: pgmap v3189: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:16 smithi078 ceph-mon[91136]: pgmap v3190: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:16 smithi078 ceph-mon[99959]: pgmap v3190: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:16.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:16 smithi146 ceph-mon[103278]: pgmap v3190: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:18 smithi146 ceph-mon[103278]: pgmap v3191: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:18 smithi078 ceph-mon[91136]: pgmap v3191: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:18 smithi078 ceph-mon[99959]: pgmap v3191: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:20 smithi078 ceph-mon[99959]: pgmap v3192: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:20 smithi078 ceph-mon[91136]: pgmap v3192: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:20.551Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:20.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:20.611 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:20.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:20 smithi146 ceph-mon[103278]: pgmap v3192: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:06:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:06:22.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:22 smithi146 ceph-mon[103278]: pgmap v3193: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:22 smithi078 ceph-mon[99959]: pgmap v3193: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:22 smithi078 ceph-mon[91136]: pgmap v3193: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:23.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:23.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:23.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:24.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:24 smithi146 ceph-mon[103278]: pgmap v3194: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:24 smithi078 ceph-mon[91136]: pgmap v3194: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:24 smithi078 ceph-mon[99959]: pgmap v3194: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:25.791 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:25.867 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:26 smithi078 ceph-mon[91136]: pgmap v3195: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:26 smithi078 ceph-mon[99959]: pgmap v3195: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:26 smithi146 ceph-mon[103278]: pgmap v3195: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:27 smithi146 ceph-mon[103278]: pgmap v3196: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[91136]: pgmap v3196: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:27 smithi078 ceph-mon[99959]: pgmap v3196: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:29 smithi146 ceph-mon[103278]: pgmap v3197: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:29 smithi078 ceph-mon[91136]: pgmap v3197: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:29 smithi078 ceph-mon[99959]: pgmap v3197: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:31.032 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:06:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:06:31.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:30.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:31.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:30.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:31.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:30.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:31.033 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:06:31.565 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:06:31.565 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 117m - - 2024-09-20T22:06:31.565 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 116m - - 2024-09-20T22:06:31.565 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (111m) 4s ago 112m 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (112m) 0s ago 112m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (104m) 4s ago 110m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (103m) 0s ago 116m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (105m) 4s ago 117m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (103m) 4s ago 118m 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (102m) 0s ago 116m 141M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (102m) 4s ago 116m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (111m) 4s ago 111m 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (111m) 0s ago 111m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (101m) 4s ago 116m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (100m) 4s ago 115m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (99m) 4s ago 115m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:06:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (99m) 4s ago 114m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (98m) 0s ago 114m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (97m) 0s ago 114m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (96m) 0s ago 113m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (95m) 0s ago 113m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (104m) 0s ago 112m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (95m) 4s ago 110m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (95m) 0s ago 110m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (95m) 4s ago 110m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:06:31.567 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (95m) 0s ago 110m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:06:31.873 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:06:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:06:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:06:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:06:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:06:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:31.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:06:31.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:06:31.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:31.875 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:06:31.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:06:31.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:06:31.876 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:06:31.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:06:31.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:06:31.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:06:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:06:32.143 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:06:32.143 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:06:32.143 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:06:32.454 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:06:32.454 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:32.454 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:06:32.455 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:32.455 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:06:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: pgmap v3198: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='client.71198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='client.71204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='client.51291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1573665158' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: pgmap v3198: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='client.71198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='client.71204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='client.51291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1573665158' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: pgmap v3198: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='client.71198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='client.71204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='client.51291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1573665158' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:06:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:33 smithi146 ceph-mon[103278]: from='client.51303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2668218194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-mon[91136]: from='client.51303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2668218194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-mon[99959]: from='client.51303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:06:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2668218194' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:06:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:33.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:33.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:33.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:34 smithi146 ceph-mon[103278]: pgmap v3199: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:34 smithi078 ceph-mon[91136]: pgmap v3199: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:34 smithi078 ceph-mon[99959]: pgmap v3199: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:36 smithi078 ceph-mon[91136]: pgmap v3200: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:36 smithi078 ceph-mon[99959]: pgmap v3200: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:36 smithi146 ceph-mon[103278]: pgmap v3200: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:38 smithi146 ceph-mon[103278]: pgmap v3201: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:38 smithi078 ceph-mon[91136]: pgmap v3201: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:38 smithi078 ceph-mon[99959]: pgmap v3201: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:40 smithi146 ceph-mon[103278]: pgmap v3202: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:40 smithi078 ceph-mon[91136]: pgmap v3202: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:40 smithi078 ceph-mon[99959]: pgmap v3202: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:06:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:06:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:40.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:40.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:40.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:42.757 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:42 smithi146 ceph-mon[103278]: pgmap v3203: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:42 smithi078 ceph-mon[91136]: pgmap v3203: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:42 smithi078 ceph-mon[99959]: pgmap v3203: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:43.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:43.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:43.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:43.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:43.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:44 smithi146 ceph-mon[103278]: pgmap v3204: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:44 smithi078 ceph-mon[91136]: pgmap v3204: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:44 smithi078 ceph-mon[99959]: pgmap v3204: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:45.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:46.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:46 smithi146 ceph-mon[103278]: pgmap v3205: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:46 smithi078 ceph-mon[99959]: pgmap v3205: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:46 smithi078 ceph-mon[91136]: pgmap v3205: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[99959]: pgmap v3206: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[91136]: pgmap v3206: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:48 smithi146 ceph-mon[103278]: pgmap v3206: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:50 smithi078 ceph-mon[99959]: pgmap v3207: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:50 smithi078 ceph-mon[91136]: pgmap v3207: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:50 smithi146 ceph-mon[103278]: pgmap v3207: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:06:50] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-20T22:06:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:50.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:50.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:50.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:52 smithi146 ceph-mon[103278]: pgmap v3208: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[99959]: pgmap v3208: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[91136]: pgmap v3208: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:06:54.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:53.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:54.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:53.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:06:54.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:06:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:06:53.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:06:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:54 smithi146 ceph-mon[103278]: pgmap v3209: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:54 smithi078 ceph-mon[99959]: pgmap v3209: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:54 smithi078 ceph-mon[91136]: pgmap v3209: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:55.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:55.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:06:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:56 smithi146 ceph-mon[103278]: pgmap v3210: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:56.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:56 smithi078 ceph-mon[91136]: pgmap v3210: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:56.920 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:56 smithi078 ceph-mon[99959]: pgmap v3210: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:06:58.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:58 smithi078 ceph-mon[91136]: pgmap v3211: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:58.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:58.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:58 smithi078 ceph-mon[99959]: pgmap v3211: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:58.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:06:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:58 smithi146 ceph-mon[103278]: pgmap v3211: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:06:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:00 smithi146 ceph-mon[103278]: pgmap v3212: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:00 smithi078 ceph-mon[99959]: pgmap v3212: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:00 smithi078 ceph-mon[91136]: pgmap v3212: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:07:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:07:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:00.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:00.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:00.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:02.755 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:07:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:02 smithi146 ceph-mon[103278]: pgmap v3213: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:02 smithi078 ceph-mon[99959]: pgmap v3213: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:02 smithi078 ceph-mon[91136]: pgmap v3213: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 117m - - 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 117m - - 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (111m) 16s ago 112m 30.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (112m) 11s ago 112m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (105m) 16s ago 110m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (103m) 11s ago 116m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (105m) 16s ago 118m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (104m) 16s ago 118m 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (102m) 11s ago 117m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (103m) 16s ago 117m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:07:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (112m) 16s ago 112m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (112m) 11s ago 112m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (101m) 16s ago 116m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (100m) 16s ago 116m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (100m) 16s ago 115m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (99m) 16s ago 115m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (99m) 11s ago 114m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (98m) 11s ago 114m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (97m) 11s ago 114m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (96m) 11s ago 113m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (105m) 11s ago 113m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (96m) 16s ago 110m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (96m) 11s ago 111m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (96m) 16s ago 110m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:07:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (95m) 11s ago 110m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:07:03.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:07:03.594 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:07:03.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:03 smithi146 ceph-mon[103278]: from='client.71234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:03.860 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:03 smithi078 ceph-mon[99959]: from='client.71234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:03.861 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:03 smithi078 ceph-mon[91136]: from='client.71234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:03.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:07:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:07:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:07:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:07:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:07:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:07:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:07:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:07:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:07:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:07:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:07:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:07:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:07:03.863 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:07:04.172 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:07:04.173 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:07:04.173 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:07:04.173 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:07:04.173 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:07:04.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:03.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:04.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:03.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:04.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:03.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:04.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:04 smithi146 ceph-mon[103278]: pgmap v3214: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:04.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:04 smithi146 ceph-mon[103278]: from='client.71240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:04.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:04 smithi146 ceph-mon[103278]: from='client.71246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:04.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/605058084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:04.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3136735697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[91136]: pgmap v3214: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[91136]: from='client.71240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[91136]: from='client.71246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/605058084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3136735697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[99959]: pgmap v3214: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[99959]: from='client.71240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[99959]: from='client.71246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/605058084' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3136735697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:05 smithi146 ceph-mon[103278]: from='client.71258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:05 smithi078 ceph-mon[91136]: from='client.71258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:05 smithi078 ceph-mon[99959]: from='client.71258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:06 smithi146 ceph-mon[103278]: pgmap v3215: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:06 smithi078 ceph-mon[91136]: pgmap v3215: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:06 smithi078 ceph-mon[99959]: pgmap v3215: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:08 smithi146 ceph-mon[103278]: pgmap v3216: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[91136]: pgmap v3216: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[99959]: pgmap v3216: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:10 smithi146 ceph-mon[103278]: pgmap v3217: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:10 smithi078 ceph-mon[91136]: pgmap v3217: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:10 smithi078 ceph-mon[99959]: pgmap v3217: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:07:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:07:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:10.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:10.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:10.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:12 smithi146 ceph-mon[103278]: pgmap v3218: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[91136]: pgmap v3218: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[99959]: pgmap v3218: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:13.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:13.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:14.261 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:13.989Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:14.262 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:13.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:14.262 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:13.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:14 smithi146 ceph-mon[103278]: pgmap v3219: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:14 smithi078 ceph-mon[91136]: pgmap v3219: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:14 smithi078 ceph-mon[99959]: pgmap v3219: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:16 smithi078 ceph-mon[91136]: pgmap v3220: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:16 smithi078 ceph-mon[99959]: pgmap v3220: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:16 smithi146 ceph-mon[103278]: pgmap v3220: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:18.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:18 smithi078 ceph-mon[91136]: pgmap v3221: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:18.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:18 smithi078 ceph-mon[99959]: pgmap v3221: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:18 smithi146 ceph-mon[103278]: pgmap v3221: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:20 smithi146 ceph-mon[103278]: pgmap v3222: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:20 smithi078 ceph-mon[99959]: pgmap v3222: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:20 smithi078 ceph-mon[91136]: pgmap v3222: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:07:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:07:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:20.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:20.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:20.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:22 smithi146 ceph-mon[103278]: pgmap v3223: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:22 smithi078 ceph-mon[91136]: pgmap v3223: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:22 smithi078 ceph-mon[99959]: pgmap v3223: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:24.272 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:23.989Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:24.273 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:23.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:24.273 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:23.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:24 smithi146 ceph-mon[103278]: pgmap v3224: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:24 smithi078 ceph-mon[91136]: pgmap v3224: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:24 smithi078 ceph-mon[99959]: pgmap v3224: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:25.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:26 smithi146 ceph-mon[103278]: pgmap v3225: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:26 smithi078 ceph-mon[91136]: pgmap v3225: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:26 smithi078 ceph-mon[99959]: pgmap v3225: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:28 smithi146 ceph-mon[103278]: pgmap v3226: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[91136]: pgmap v3226: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[99959]: pgmap v3226: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:30.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:30 smithi078 ceph-mon[91136]: pgmap v3227: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:30.559 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:30 smithi078 ceph-mon[99959]: pgmap v3227: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:30 smithi146 ceph-mon[103278]: pgmap v3227: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:07:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:07:31.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:30.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:30.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:30.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:32 smithi146 ceph-mon[103278]: pgmap v3228: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[91136]: pgmap v3228: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[99959]: pgmap v3228: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:34.283 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:33.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:34.283 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:33.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:34.283 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:33.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:34.463 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:07:34.993 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:07:34.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 118m - - 2024-09-20T22:07:34.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 117m - - 2024-09-20T22:07:34.993 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (112m) 7s ago 113m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:07:34.993 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (113m) 3s ago 113m 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (105m) 7s ago 111m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (104m) 3s ago 117m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (106m) 7s ago 118m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (104m) 7s ago 119m 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (103m) 3s ago 117m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (103m) 7s ago 117m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (112m) 7s ago 112m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (112m) 3s ago 112m 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (102m) 7s ago 117m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (101m) 7s ago 116m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (100m) 7s ago 116m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (100m) 7s ago 115m 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (99m) 3s ago 115m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (98m) 3s ago 115m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:07:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (97m) 3s ago 114m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:07:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (97m) 3s ago 114m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:07:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (105m) 3s ago 113m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:07:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (96m) 7s ago 111m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:07:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (96m) 3s ago 111m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:07:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (96m) 7s ago 111m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:07:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (96m) 3s ago 111m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:07:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:34 smithi146 ceph-mon[103278]: pgmap v3229: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:34 smithi078 ceph-mon[99959]: pgmap v3229: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:34 smithi078 ceph-mon[91136]: pgmap v3229: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:07:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:07:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:07:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:07:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:07:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:07:35.304 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:07:35.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:07:35.574 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:07:35.883 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:07:35.883 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:07:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:07:35.883 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:07:35.883 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[91136]: from='client.71270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[91136]: from='client.71276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1341174777' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[99959]: from='client.71270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[99959]: from='client.71276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:35.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1341174777' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:35 smithi146 ceph-mon[103278]: from='client.71270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:35 smithi146 ceph-mon[103278]: from='client.71276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1341174777' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:07:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:36 smithi146 ceph-mon[103278]: from='client.51363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:36 smithi146 ceph-mon[103278]: pgmap v3230: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:36 smithi146 ceph-mon[103278]: from='client.51375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3844993768' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:37.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[99959]: from='client.51363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:37.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[99959]: pgmap v3230: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:37.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[99959]: from='client.51375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:37.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3844993768' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:37.065 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[91136]: from='client.51363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:37.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[91136]: pgmap v3230: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:37.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[91136]: from='client.51375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:07:37.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3844993768' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:07:39.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:38 smithi078 ceph-mon[91136]: pgmap v3231: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:39.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:38 smithi078 ceph-mon[99959]: pgmap v3231: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:38 smithi146 ceph-mon[103278]: pgmap v3231: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:40.819 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:07:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:07:40.819 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:40.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:40.819 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:40.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:40.820 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:40.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:40 smithi146 ceph-mon[103278]: pgmap v3232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:40 smithi078 ceph-mon[91136]: pgmap v3232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:40 smithi078 ceph-mon[99959]: pgmap v3232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:42 smithi146 ceph-mon[103278]: pgmap v3233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:42 smithi078 ceph-mon[91136]: pgmap v3233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:42 smithi078 ceph-mon[99959]: pgmap v3233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:43.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:43.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:43.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:44.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:44 smithi146 ceph-mon[103278]: pgmap v3234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:44 smithi078 ceph-mon[99959]: pgmap v3234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:44 smithi078 ceph-mon[91136]: pgmap v3234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:46 smithi146 ceph-mon[103278]: pgmap v3235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:46 smithi078 ceph-mon[99959]: pgmap v3235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:46 smithi078 ceph-mon[91136]: pgmap v3235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:48.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:48 smithi146 ceph-mon[103278]: pgmap v3236: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:48.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[99959]: pgmap v3236: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[91136]: pgmap v3236: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:48.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:48.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:50 smithi078 ceph-mon[99959]: pgmap v3237: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:50 smithi078 ceph-mon[91136]: pgmap v3237: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:50.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:50.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:50.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:50.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:50 smithi146 ceph-mon[103278]: pgmap v3237: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:50.910 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:07:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:07:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:52 smithi146 ceph-mon[103278]: pgmap v3238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[91136]: pgmap v3238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[99959]: pgmap v3238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:07:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:53.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:53.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:07:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:07:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:07:53.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:07:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:54 smithi146 ceph-mon[103278]: pgmap v3239: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:54 smithi078 ceph-mon[99959]: pgmap v3239: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:54 smithi078 ceph-mon[91136]: pgmap v3239: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:07:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:56 smithi146 ceph-mon[103278]: pgmap v3240: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:56 smithi078 ceph-mon[91136]: pgmap v3240: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:56 smithi078 ceph-mon[99959]: pgmap v3240: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:07:58.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:58 smithi078 ceph-mon[99959]: pgmap v3241: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:58.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:58.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:58 smithi078 ceph-mon[91136]: pgmap v3241: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:58.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:07:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:58 smithi146 ceph-mon[103278]: pgmap v3241: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:07:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:00 smithi146 ceph-mon[103278]: pgmap v3242: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:00 smithi078 ceph-mon[91136]: pgmap v3242: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:08:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:08:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:00 smithi078 ceph-mon[99959]: pgmap v3242: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:00.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:00.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:00.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:02 smithi146 ceph-mon[103278]: pgmap v3243: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:02 smithi078 ceph-mon[99959]: pgmap v3243: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:02 smithi078 ceph-mon[91136]: pgmap v3243: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:03.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:03.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:03.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:04.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:04 smithi146 ceph-mon[103278]: pgmap v3244: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:04 smithi078 ceph-mon[91136]: pgmap v3244: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:04 smithi078 ceph-mon[99959]: pgmap v3244: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:05.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:06.176 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:08:06.608 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:06 smithi146 ceph-mon[103278]: pgmap v3245: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 118m - - 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 118m - - 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (112m) 19s ago 113m 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (113m) 14s ago 113m 87.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (106m) 19s ago 111m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (105m) 14s ago 117m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:08:06.705 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (106m) 19s ago 119m 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (105m) 19s ago 119m 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (103m) 14s ago 118m 142M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (104m) 19s ago 118m 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (113m) 19s ago 113m 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (113m) 14s ago 113m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (102m) 19s ago 117m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (101m) 19s ago 117m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (101m) 19s ago 116m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (100m) 19s ago 116m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (100m) 14s ago 116m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (99m) 14s ago 115m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (98m) 14s ago 115m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (97m) 14s ago 114m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:08:06.706 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (106m) 14s ago 114m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:08:06.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (97m) 19s ago 112m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:08:06.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (97m) 14s ago 112m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:08:06.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (97m) 19s ago 111m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:08:06.707 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (96m) 14s ago 111m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:08:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:06 smithi078 ceph-mon[99959]: pgmap v3245: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:06 smithi078 ceph-mon[91136]: pgmap v3245: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:07.016 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:08:07.016 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:08:07.016 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:08:07.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:07.016 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:08:07.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:08:07.285 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:08:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:08:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:08:07.285 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:08:07.286 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[91136]: from='client.51387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[91136]: from='client.71312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[91136]: from='client.71318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1394352781' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[99959]: from='client.51387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[99959]: from='client.71312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.562 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[99959]: from='client.71318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.563 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1394352781' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:07.608 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:08:07.608 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:07.608 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:07 smithi146 ceph-mon[103278]: from='client.51387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:07 smithi146 ceph-mon[103278]: from='client.71312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:07 smithi146 ceph-mon[103278]: from='client.71318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1394352781' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:08.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:08 smithi146 ceph-mon[103278]: pgmap v3246: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:08.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:08 smithi146 ceph-mon[103278]: from='client.51411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:08.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:08.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:08.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:08.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3203495685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[91136]: pgmap v3246: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[91136]: from='client.51411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3203495685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[99959]: pgmap v3246: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[99959]: from='client.51411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3203495685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:10.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:10 smithi078 ceph-mon[99959]: pgmap v3247: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:10.838 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:08:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:08:10.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:10 smithi078 ceph-mon[91136]: pgmap v3247: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:10.838 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:10.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:10.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:10.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:10 smithi146 ceph-mon[103278]: pgmap v3247: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:12 smithi146 ceph-mon[103278]: pgmap v3248: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[91136]: pgmap v3248: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[99959]: pgmap v3248: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:13.995 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:13.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:13.993Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:13.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:13 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:13.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:14 smithi078 ceph-mon[99959]: pgmap v3249: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:14 smithi078 ceph-mon[91136]: pgmap v3249: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:14 smithi146 ceph-mon[103278]: pgmap v3249: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:16.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:16 smithi146 ceph-mon[103278]: pgmap v3250: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:16 smithi078 ceph-mon[91136]: pgmap v3250: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:16 smithi078 ceph-mon[99959]: pgmap v3250: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:18.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:18 smithi078 ceph-mon[99959]: pgmap v3251: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:18.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:18 smithi078 ceph-mon[91136]: pgmap v3251: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:18 smithi146 ceph-mon[103278]: pgmap v3251: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:20 smithi078 ceph-mon[91136]: pgmap v3252: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:08:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:08:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:20 smithi078 ceph-mon[99959]: pgmap v3252: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:20.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:20.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:20.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:20 smithi146 ceph-mon[103278]: pgmap v3252: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:22 smithi146 ceph-mon[103278]: pgmap v3253: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:22 smithi078 ceph-mon[91136]: pgmap v3253: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:22 smithi078 ceph-mon[99959]: pgmap v3253: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:23.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:23.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:23 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:23.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:24 smithi146 ceph-mon[103278]: pgmap v3254: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:24 smithi078 ceph-mon[91136]: pgmap v3254: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:24 smithi078 ceph-mon[99959]: pgmap v3254: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:25.865 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:26 smithi146 ceph-mon[103278]: pgmap v3255: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:26 smithi078 ceph-mon[91136]: pgmap v3255: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:26 smithi078 ceph-mon[99959]: pgmap v3255: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:28 smithi146 ceph-mon[103278]: pgmap v3256: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[99959]: pgmap v3256: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[91136]: pgmap v3256: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:30.987 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:30 smithi078 ceph-mon[99959]: pgmap v3257: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:30.987 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:08:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:08:30.987 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:30 smithi078 ceph-mon[91136]: pgmap v3257: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:30.988 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:30.563Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:30.988 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:30.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:30.988 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:30.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:30 smithi146 ceph-mon[103278]: pgmap v3257: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:32 smithi146 ceph-mon[103278]: pgmap v3258: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[99959]: pgmap v3258: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[91136]: pgmap v3258: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:33.995Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:33.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:33 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:33.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:34 smithi146 ceph-mon[103278]: pgmap v3259: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:34 smithi078 ceph-mon[91136]: pgmap v3259: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:34 smithi078 ceph-mon[99959]: pgmap v3259: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:36 smithi146 ceph-mon[103278]: pgmap v3260: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:36 smithi078 ceph-mon[91136]: pgmap v3260: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:36 smithi078 ceph-mon[99959]: pgmap v3260: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 119m - - 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 119m - - 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (113m) 10s ago 114m 30.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (114m) 6s ago 114m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (106m) 10s ago 112m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (105m) 6s ago 118m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (107m) 10s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:08:38.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (105m) 10s ago 2h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (104m) 6s ago 118m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (104m) 10s ago 118m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (113m) 10s ago 113m 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (113m) 6s ago 113m 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (103m) 10s ago 118m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (102m) 10s ago 117m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (101m) 10s ago 117m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (101m) 10s ago 116m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (100m) 6s ago 116m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (99m) 6s ago 116m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (98m) 6s ago 115m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (98m) 6s ago 115m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (106m) 6s ago 114m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (97m) 10s ago 112m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:08:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (97m) 6s ago 112m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:08:38.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (97m) 10s ago 112m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:08:38.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (97m) 6s ago 112m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:08:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:08:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:08:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:08:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:08:38.756 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:08:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:38 smithi146 ceph-mon[103278]: pgmap v3261: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:39.031 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:08:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:08:39.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:08:39.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:38 smithi078 ceph-mon[99959]: pgmap v3261: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:39.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:38 smithi078 ceph-mon[91136]: pgmap v3261: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:39.342 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:08:39.342 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:08:39.342 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:08:39.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[99959]: from='client.51423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[99959]: from='client.51429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[99959]: from='client.51435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/675108058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3714273483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[91136]: from='client.51423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[91136]: from='client.51429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[91136]: from='client.51435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:39.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/675108058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:39.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3714273483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:39 smithi146 ceph-mon[103278]: from='client.51423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:39 smithi146 ceph-mon[103278]: from='client.51429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:39 smithi146 ceph-mon[103278]: from='client.51435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/675108058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:08:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3714273483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:08:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-mon[91136]: pgmap v3262: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-mon[91136]: from='client.71366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:08:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:08:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-mon[99959]: pgmap v3262: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-mon[99959]: from='client.71366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:40.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:40.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:40.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:40 smithi146 ceph-mon[103278]: pgmap v3262: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:40 smithi146 ceph-mon[103278]: from='client.71366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:08:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:42 smithi146 ceph-mon[103278]: pgmap v3263: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:42 smithi078 ceph-mon[91136]: pgmap v3263: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:42 smithi078 ceph-mon[99959]: pgmap v3263: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:43.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:43.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:43.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:43.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:43 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:43.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:44 smithi146 ceph-mon[103278]: pgmap v3264: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:44 smithi078 ceph-mon[99959]: pgmap v3264: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:44 smithi078 ceph-mon[91136]: pgmap v3264: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:45.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:46 smithi146 ceph-mon[103278]: pgmap v3265: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:46 smithi078 ceph-mon[91136]: pgmap v3265: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:46 smithi078 ceph-mon[99959]: pgmap v3265: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:48 smithi146 ceph-mon[103278]: pgmap v3266: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[91136]: pgmap v3266: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[99959]: pgmap v3266: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:50 smithi078 ceph-mon[91136]: pgmap v3267: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:08:50] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-20T22:08:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:50 smithi078 ceph-mon[99959]: pgmap v3267: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:50.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:50.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:50.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:50 smithi146 ceph-mon[103278]: pgmap v3267: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:52 smithi146 ceph-mon[103278]: pgmap v3268: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[99959]: pgmap v3268: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[91136]: pgmap v3268: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:08:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:53.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:54.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:53.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:08:54.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:08:53 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:08:53.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:08:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:54 smithi146 ceph-mon[103278]: pgmap v3269: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:54 smithi078 ceph-mon[91136]: pgmap v3269: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:54 smithi078 ceph-mon[99959]: pgmap v3269: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:08:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:56 smithi146 ceph-mon[103278]: pgmap v3270: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:56 smithi078 ceph-mon[91136]: pgmap v3270: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:56 smithi078 ceph-mon[99959]: pgmap v3270: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:08:59.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:58 smithi078 ceph-mon[91136]: pgmap v3271: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:59.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:59.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:58 smithi078 ceph-mon[99959]: pgmap v3271: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:59.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:08:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:58 smithi146 ceph-mon[103278]: pgmap v3271: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:08:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:00.823 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:09:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:09:00.823 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:00.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:00.823 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:00.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:00.823 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:00.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:01.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:00 smithi078 ceph-mon[91136]: pgmap v3272: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:01.119 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:00 smithi078 ceph-mon[99959]: pgmap v3272: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:00 smithi146 ceph-mon[103278]: pgmap v3272: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:02 smithi146 ceph-mon[103278]: pgmap v3273: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:02 smithi078 ceph-mon[91136]: pgmap v3273: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:02 smithi078 ceph-mon[99959]: pgmap v3273: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:03.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:03.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:03 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:03.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:04 smithi146 ceph-mon[103278]: pgmap v3274: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:04 smithi078 ceph-mon[91136]: pgmap v3274: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:04 smithi078 ceph-mon[99959]: pgmap v3274: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:06 smithi146 ceph-mon[103278]: pgmap v3275: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:06 smithi078 ceph-mon[91136]: pgmap v3275: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:06 smithi078 ceph-mon[99959]: pgmap v3275: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:08 smithi146 ceph-mon[103278]: pgmap v3276: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[99959]: pgmap v3276: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[91136]: pgmap v3276: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:09.636 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 119m - - 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 119m - - 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (113m) 2s ago 114m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (114m) 17s ago 114m 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (107m) 2s ago 112m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:09:10.166 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (106m) 17s ago 118m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (108m) 2s ago 2h 564M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (106m) 2s ago 2h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (104m) 17s ago 119m 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (105m) 2s ago 119m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (114m) 2s ago 114m 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (114m) 17s ago 114m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (104m) 2s ago 118m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (103m) 2s ago 118m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (102m) 2s ago 117m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (101m) 2s ago 117m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (101m) 17s ago 117m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:09:10.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (100m) 17s ago 116m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (99m) 17s ago 116m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (98m) 17s ago 115m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (107m) 17s ago 115m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (98m) 2s ago 113m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (98m) 17s ago 113m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (98m) 2s ago 113m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:09:10.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (98m) 17s ago 113m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:09:10.473 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:09:10.473 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:09:10.473 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:09:10.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:10.473 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:09:10.473 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:09:10.474 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:09:10.743 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:09:10.743 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:09:10.743 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:09:10.743 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:09:10.743 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:09:10.744 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:09:10.744 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:09:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:09:10.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:10.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:10.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:10.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:10.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:10.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-mon[99959]: pgmap v3277: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-mon[99959]: from='client.71378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1635604435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-mon[91136]: pgmap v3277: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-mon[91136]: from='client.71378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1635604435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:11.056 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:09:11.056 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:11.056 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:10 smithi146 ceph-mon[103278]: pgmap v3277: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:10 smithi146 ceph-mon[103278]: from='client.71378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1635604435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:11 smithi146 ceph-mon[103278]: from='client.71384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:11 smithi146 ceph-mon[103278]: from='client.71390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:11 smithi146 ceph-mon[103278]: from='client.51483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/291904943' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[99959]: from='client.71384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[99959]: from='client.71390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[99959]: from='client.51483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/291904943' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[91136]: from='client.71384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[91136]: from='client.71390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[91136]: from='client.51483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/291904943' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:12 smithi146 ceph-mon[103278]: pgmap v3278: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[91136]: pgmap v3278: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[99959]: pgmap v3278: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:13 smithi146 ceph-mon[103278]: pgmap v3279: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:13 smithi078 ceph-mon[91136]: pgmap v3279: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:13 smithi078 ceph-mon[99959]: pgmap v3279: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:13.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:14.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:14.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:16.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:15 smithi146 ceph-mon[103278]: pgmap v3280: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:15 smithi078 ceph-mon[91136]: pgmap v3280: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:15 smithi078 ceph-mon[99959]: pgmap v3280: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:18 smithi146 ceph-mon[103278]: pgmap v3281: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:18 smithi078 ceph-mon[91136]: pgmap v3281: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:18 smithi078 ceph-mon[99959]: pgmap v3281: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:20.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:20 smithi078 ceph-mon[91136]: pgmap v3282: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:20.105 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:20 smithi078 ceph-mon[99959]: pgmap v3282: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:20 smithi146 ceph-mon[103278]: pgmap v3282: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:20.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:09:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:09:20.872 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:20.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:20.872 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:20.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:20.873 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:20.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:22 smithi078 ceph-mon[91136]: pgmap v3283: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:22 smithi078 ceph-mon[99959]: pgmap v3283: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:22 smithi146 ceph-mon[103278]: pgmap v3283: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:24.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:24.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:24.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:24 smithi078 ceph-mon[91136]: pgmap v3284: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:24 smithi078 ceph-mon[99959]: pgmap v3284: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:24 smithi146 ceph-mon[103278]: pgmap v3284: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:26.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:26 smithi146 ceph-mon[103278]: pgmap v3285: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:26 smithi078 ceph-mon[91136]: pgmap v3285: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:26 smithi078 ceph-mon[99959]: pgmap v3285: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:28 smithi146 ceph-mon[103278]: pgmap v3286: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[91136]: pgmap v3286: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[99959]: pgmap v3286: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:31.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:30 smithi146 ceph-mon[103278]: pgmap v3287: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:30 smithi078 ceph-mon[91136]: pgmap v3287: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:30 smithi078 ceph-mon[99959]: pgmap v3287: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:09:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:09:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:30.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:30.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:30.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:32 smithi146 ceph-mon[103278]: pgmap v3288: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[99959]: pgmap v3288: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[91136]: pgmap v3288: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:34.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:34.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:34.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:34 smithi078 ceph-mon[91136]: pgmap v3289: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:34 smithi078 ceph-mon[99959]: pgmap v3289: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:34 smithi146 ceph-mon[103278]: pgmap v3289: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:36 smithi146 ceph-mon[103278]: pgmap v3290: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:36 smithi078 ceph-mon[99959]: pgmap v3290: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:36 smithi078 ceph-mon[91136]: pgmap v3290: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:38 smithi146 ceph-mon[103278]: pgmap v3291: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:38 smithi078 ceph-mon[99959]: pgmap v3291: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:38 smithi078 ceph-mon[91136]: pgmap v3291: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:40.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:40 smithi078 ceph-mon[99959]: pgmap v3292: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:40.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:40 smithi078 ceph-mon[91136]: pgmap v3292: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:40.720 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:09:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:09:40.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:40.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:40.721 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:40.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:40.721 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:40.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:41.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:40 smithi146 ceph-mon[103278]: pgmap v3292: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:41.356 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:09:41.912 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:09:41.912 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 2h - - 2024-09-20T22:09:41.912 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 2h - - 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (114m) 13s ago 115m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (115m) 9s ago 115m 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (107m) 13s ago 113m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (106m) 9s ago 119m 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (108m) 13s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (106m) 13s ago 2h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (105m) 9s ago 119m 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (105m) 13s ago 119m 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (115m) 13s ago 115m 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (114m) 9s ago 114m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (104m) 13s ago 119m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (103m) 13s ago 118m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (102m) 13s ago 118m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:09:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (102m) 13s ago 118m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (101m) 9s ago 117m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (100m) 9s ago 117m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (100m) 9s ago 116m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (99m) 9s ago 116m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (108m) 9s ago 115m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (98m) 13s ago 113m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (98m) 9s ago 113m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (98m) 13s ago 113m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:09:41.914 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (98m) 9s ago 113m 88.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:09:42.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:09:42.257 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:09:42.551 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:09:42.551 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:09:42.552 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[91136]: pgmap v3293: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[91136]: from='client.71414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[91136]: from='client.71420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3986217269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[99959]: pgmap v3293: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[99959]: from='client.71414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:42.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[99959]: from='client.71420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:42.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3986217269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:42.868 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:09:42.868 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:42.868 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:09:42.868 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:42.868 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:09:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:42 smithi146 ceph-mon[103278]: pgmap v3293: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:42 smithi146 ceph-mon[103278]: from='client.71414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:42 smithi146 ceph-mon[103278]: from='client.71420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3986217269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:09:44.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[91136]: from='client.51507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:44.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[91136]: from='client.51519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:44.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2193670268' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:44.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:44.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[99959]: from='client.51507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:44.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[99959]: from='client.51519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:44.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2193670268' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:44.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:43 smithi146 ceph-mon[103278]: from='client.51507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:43 smithi146 ceph-mon[103278]: from='client.51519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:09:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2193670268' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:09:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:44.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:44.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:44.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:44 smithi078 ceph-mon[91136]: pgmap v3294: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:44 smithi078 ceph-mon[99959]: pgmap v3294: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:44 smithi146 ceph-mon[103278]: pgmap v3294: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:46.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:46 smithi146 ceph-mon[103278]: pgmap v3295: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:46 smithi078 ceph-mon[91136]: pgmap v3295: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:46 smithi078 ceph-mon[99959]: pgmap v3295: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:48.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:48 smithi146 ceph-mon[103278]: pgmap v3296: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:48.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:48.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:48.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[99959]: pgmap v3296: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[91136]: pgmap v3296: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:50 smithi146 ceph-mon[103278]: pgmap v3297: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:50 smithi078 ceph-mon[91136]: pgmap v3297: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:50 smithi078 ceph-mon[99959]: pgmap v3297: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:09:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:09:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:50.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:50.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:50.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:52 smithi078 ceph-mon[91136]: pgmap v3298: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:52 smithi078 ceph-mon[99959]: pgmap v3298: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:52 smithi146 ceph-mon[103278]: pgmap v3298: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:54.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:09:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:54.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:54.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:09:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:09:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:09:54.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:09:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:54 smithi146 ceph-mon[103278]: pgmap v3299: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:54 smithi078 ceph-mon[91136]: pgmap v3299: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:54 smithi078 ceph-mon[99959]: pgmap v3299: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:09:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:56 smithi146 ceph-mon[103278]: pgmap v3300: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:56 smithi078 ceph-mon[91136]: pgmap v3300: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:56 smithi078 ceph-mon[99959]: pgmap v3300: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:09:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:58 smithi146 ceph-mon[103278]: pgmap v3301: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:58 smithi078 ceph-mon[99959]: pgmap v3301: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:09:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:58 smithi078 ceph-mon[91136]: pgmap v3301: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:09:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T22:09:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:00 smithi146 ceph-mon[103278]: pgmap v3302: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:10:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:10:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-mon[99959]: pgmap v3302: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-mon[91136]: pgmap v3302: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:00.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:00.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:00.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:02.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:02 smithi078 ceph-mon[99959]: pgmap v3303: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:02.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:02 smithi078 ceph-mon[91136]: pgmap v3303: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:02 smithi146 ceph-mon[103278]: pgmap v3303: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:04.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:04.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:04.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:04 smithi146 ceph-mon[103278]: pgmap v3304: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:04 smithi078 ceph-mon[91136]: pgmap v3304: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:04 smithi078 ceph-mon[99959]: pgmap v3304: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:06 smithi146 ceph-mon[103278]: pgmap v3305: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:06 smithi078 ceph-mon[91136]: pgmap v3305: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:06 smithi078 ceph-mon[99959]: pgmap v3305: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:08 smithi146 ceph-mon[103278]: pgmap v3306: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[91136]: pgmap v3306: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[99959]: pgmap v3306: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:10.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:10 smithi078 ceph-mon[99959]: pgmap v3307: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:10.838 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:10:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:10:10.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:10 smithi078 ceph-mon[91136]: pgmap v3307: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:10.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:10.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:10.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:10.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:10 smithi146 ceph-mon[103278]: pgmap v3307: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:12 smithi146 ceph-mon[103278]: pgmap v3308: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:12 smithi078 ceph-mon[99959]: pgmap v3308: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:12 smithi078 ceph-mon[91136]: pgmap v3308: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:13.155 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 2h - - 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (114m) 5s ago 116m 30.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (115m) 0s ago 115m 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (108m) 5s ago 113m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (107m) 0s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:10:13.685 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (109m) 5s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (107m) 5s ago 2h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (106m) 0s ago 2h 144M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (106m) 5s ago 2h 151M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (115m) 5s ago 115m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (115m) 0s ago 115m 14.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (105m) 5s ago 119m 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (104m) 5s ago 119m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (103m) 5s ago 118m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (102m) 5s ago 118m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (102m) 0s ago 118m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (101m) 0s ago 117m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (100m) 0s ago 117m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (99m) 0s ago 116m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:10:13.686 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (108m) 0s ago 116m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:10:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (99m) 5s ago 114m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:10:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (99m) 0s ago 114m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:10:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (99m) 5s ago 114m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:10:13.687 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (99m) 0s ago 114m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:10:14.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:10:14.001 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:10:14.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:14.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:10:14.267 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:10:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:10:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:10:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:10:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:10:14.268 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:10:14.268 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:10:14.268 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:14.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:14.268 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:14.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:14.268 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:14.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:14.577 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:14.577 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:14.577 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:14 smithi146 ceph-mon[103278]: pgmap v3309: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:14 smithi146 ceph-mon[103278]: from='client.51531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:14 smithi146 ceph-mon[103278]: from='client.51537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:14 smithi146 ceph-mon[103278]: from='client.51543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1502750361' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/382738084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[91136]: pgmap v3309: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[91136]: from='client.51531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[91136]: from='client.51537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[91136]: from='client.51543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1502750361' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/382738084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[99959]: pgmap v3309: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[99959]: from='client.51531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[99959]: from='client.51537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[99959]: from='client.51543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1502750361' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/382738084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:16.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:15 smithi146 ceph-mon[103278]: from='client.71474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:16.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:15 smithi078 ceph-mon[91136]: from='client.71474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:15 smithi078 ceph-mon[99959]: from='client.71474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:16 smithi146 ceph-mon[103278]: pgmap v3310: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:16 smithi078 ceph-mon[91136]: pgmap v3310: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:16 smithi078 ceph-mon[99959]: pgmap v3310: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:17 smithi146 ceph-mon[103278]: pgmap v3311: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:17 smithi078 ceph-mon[99959]: pgmap v3311: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:17 smithi078 ceph-mon[91136]: pgmap v3311: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:20 smithi146 ceph-mon[103278]: pgmap v3312: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:20 smithi078 ceph-mon[99959]: pgmap v3312: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:20 smithi078 ceph-mon[91136]: pgmap v3312: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:10:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:10:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:20.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:20.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:20.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:22 smithi078 ceph-mon[91136]: pgmap v3313: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:22 smithi078 ceph-mon[99959]: pgmap v3313: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:22 smithi146 ceph-mon[103278]: pgmap v3313: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:24 smithi078 ceph-mon[91136]: pgmap v3314: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:24 smithi078 ceph-mon[99959]: pgmap v3314: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:24.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:24.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:24.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:24 smithi146 ceph-mon[103278]: pgmap v3314: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:26 smithi146 ceph-mon[103278]: pgmap v3315: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:26 smithi078 ceph-mon[91136]: pgmap v3315: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:26 smithi078 ceph-mon[99959]: pgmap v3315: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:28.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:28 smithi146 ceph-mon[103278]: pgmap v3316: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:28.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:28 smithi078 ceph-mon[91136]: pgmap v3316: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:28.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:28 smithi078 ceph-mon[99959]: pgmap v3316: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:29.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:29.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:30 smithi146 ceph-mon[103278]: pgmap v3317: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:30 smithi078 ceph-mon[91136]: pgmap v3317: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:30 smithi078 ceph-mon[99959]: pgmap v3317: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:10:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:10:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:30.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:30.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:30.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:32 smithi146 ceph-mon[103278]: pgmap v3318: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:32 smithi078 ceph-mon[99959]: pgmap v3318: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:32 smithi078 ceph-mon[91136]: pgmap v3318: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:33 smithi146 ceph-mon[103278]: pgmap v3319: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[99959]: pgmap v3319: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:33 smithi078 ceph-mon[91136]: pgmap v3319: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:34.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:34.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:34.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:35 smithi146 ceph-mon[103278]: pgmap v3320: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:35 smithi078 ceph-mon[99959]: pgmap v3320: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:35 smithi078 ceph-mon[91136]: pgmap v3320: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:38 smithi146 ceph-mon[103278]: pgmap v3321: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:38 smithi078 ceph-mon[91136]: pgmap v3321: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:38 smithi078 ceph-mon[99959]: pgmap v3321: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:40 smithi146 ceph-mon[103278]: pgmap v3322: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:40 smithi078 ceph-mon[99959]: pgmap v3322: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:40 smithi078 ceph-mon[91136]: pgmap v3322: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:10:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:10:41.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:40.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:41.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:40.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:41.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:40.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:42.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:42 smithi078 ceph-mon[91136]: pgmap v3323: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:42.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:42 smithi078 ceph-mon[99959]: pgmap v3323: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:42 smithi146 ceph-mon[103278]: pgmap v3323: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:44.100 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:44.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:44.100 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:44.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:44.100 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:44.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:44.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-mon[99959]: pgmap v3324: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:44.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:44.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-mon[91136]: pgmap v3324: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:44.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:44 smithi146 ceph-mon[103278]: pgmap v3324: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:44.864 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:10:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:45 smithi078 ceph-mon[99959]: from='client.71486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:45 smithi078 ceph-mon[91136]: from='client.71486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:45.400 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:10:45.400 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 2h - - 2024-09-20T22:10:45.400 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 2h - - 2024-09-20T22:10:45.400 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (115m) 16s ago 116m 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (116m) 12s ago 116m 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (108m) 16s ago 114m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (107m) 12s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (109m) 16s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (107m) 16s ago 2h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (106m) 12s ago 2h 145M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (106m) 16s ago 2h 152M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (116m) 16s ago 116m 15.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (116m) 12s ago 116m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (105m) 16s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (104m) 16s ago 119m 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (103m) 16s ago 119m 64.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (103m) 16s ago 119m 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:10:45.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (102m) 12s ago 118m 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (101m) 12s ago 118m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (101m) 12s ago 117m 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (100m) 12s ago 117m 65.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (109m) 12s ago 116m 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (99m) 16s ago 114m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (99m) 12s ago 114m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (99m) 16s ago 114m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:10:45.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (99m) 12s ago 114m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:10:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:45 smithi146 ceph-mon[103278]: from='client.71486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:10:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:10:45.711 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:10:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:10:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:10:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:10:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:10:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:10:45.980 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:10:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[91136]: pgmap v3325: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[91136]: from='client.51573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[91136]: from='client.51579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1597260315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[99959]: pgmap v3325: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[99959]: from='client.51573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[99959]: from='client.51579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1597260315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:46.286 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:10:46.286 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:46.286 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:10:46.286 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:46.286 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:10:46.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:46 smithi146 ceph-mon[103278]: pgmap v3325: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:46.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:46 smithi146 ceph-mon[103278]: from='client.51573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:46.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:46 smithi146 ceph-mon[103278]: from='client.51579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:46.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1597260315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:10:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:47 smithi146 ceph-mon[103278]: from='client.51591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1143810781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:47 smithi078 ceph-mon[99959]: from='client.51591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1143810781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:47 smithi078 ceph-mon[91136]: from='client.51591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:10:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1143810781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:10:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:48 smithi146 ceph-mon[103278]: pgmap v3326: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:48 smithi078 ceph-mon[91136]: pgmap v3326: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:48 smithi078 ceph-mon[99959]: pgmap v3326: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:50.881 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:50 smithi146 ceph-mon[103278]: pgmap v3327: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:51.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:50 smithi078 ceph-mon[91136]: pgmap v3327: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:51.025 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:50 smithi078 ceph-mon[99959]: pgmap v3327: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:10:51.026 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:10:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:10:51.026 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:50.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:51.026 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:50.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:51.026 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:50.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:52 smithi146 ceph-mon[103278]: pgmap v3328: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:52 smithi078 ceph-mon[91136]: pgmap v3328: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:52 smithi078 ceph-mon[99959]: pgmap v3328: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:10:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[99959]: pgmap v3329: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:10:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[91136]: pgmap v3329: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:10:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:54.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:54.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:10:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:10:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:10:54.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:10:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:54 smithi146 ceph-mon[103278]: pgmap v3329: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:10:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:10:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:10:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:56 smithi146 ceph-mon[103278]: pgmap v3330: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:10:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:56 smithi078 ceph-mon[91136]: pgmap v3330: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:10:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:56 smithi078 ceph-mon[99959]: pgmap v3330: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-20T22:10:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:58 smithi146 ceph-mon[103278]: pgmap v3331: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:10:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:58 smithi078 ceph-mon[91136]: pgmap v3331: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:10:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:58 smithi078 ceph-mon[99959]: pgmap v3331: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:10:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:00 smithi146 ceph-mon[103278]: pgmap v3332: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:11:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:00 smithi078 ceph-mon[91136]: pgmap v3332: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:11:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:00 smithi078 ceph-mon[99959]: pgmap v3332: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:11:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:11:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:11:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:00.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:00.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:00.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:02 smithi146 ceph-mon[103278]: pgmap v3333: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:11:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:02 smithi078 ceph-mon[91136]: pgmap v3333: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:11:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:02 smithi078 ceph-mon[99959]: pgmap v3333: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:11:04.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:04.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:04.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:04.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:04.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:04.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:04 smithi078 ceph-mon[91136]: pgmap v3334: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:11:04.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:04 smithi078 ceph-mon[99959]: pgmap v3334: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:11:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:04 smithi146 ceph-mon[103278]: pgmap v3334: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:11:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:06 smithi078 ceph-mon[91136]: pgmap v3335: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:11:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:06 smithi078 ceph-mon[99959]: pgmap v3335: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:11:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:06 smithi146 ceph-mon[103278]: pgmap v3335: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:11:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:08 smithi146 ceph-mon[103278]: pgmap v3336: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-20T22:11:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:08 smithi078 ceph-mon[91136]: pgmap v3336: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-20T22:11:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:08 smithi078 ceph-mon[99959]: pgmap v3336: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-20T22:11:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:10.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:10 smithi078 ceph-mon[99959]: pgmap v3337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:11:10.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:10 smithi078 ceph-mon[91136]: pgmap v3337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:11:10.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:11:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:11:10.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:10.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:10.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:10.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:10.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:10.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:10 smithi146 ceph-mon[103278]: pgmap v3337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:11:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:12 smithi146 ceph-mon[103278]: pgmap v3338: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:11:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:12 smithi078 ceph-mon[91136]: pgmap v3338: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:11:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:12 smithi078 ceph-mon[99959]: pgmap v3338: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:11:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[91136]: pgmap v3339: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[99959]: pgmap v3339: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:14.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:14.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:14.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:14 smithi146 ceph-mon[103278]: pgmap v3339: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-20T22:11:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:16 smithi146 ceph-mon[103278]: pgmap v3340: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:11:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:16 smithi078 ceph-mon[99959]: pgmap v3340: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:11:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:16 smithi078 ceph-mon[91136]: pgmap v3340: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-20T22:11:16.578 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:11:17.118 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:11:17.118 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 2h - - 2024-09-20T22:11:17.118 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 2h - - 2024-09-20T22:11:17.118 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (116m) 8s ago 117m 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:11:17.118 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (116m) 3s ago 116m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (109m) 8s ago 114m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (108m) 3s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (110m) 8s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (108m) 8s ago 2h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (107m) 3s ago 2h 146M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (107m) 8s ago 2h 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (116m) 8s ago 116m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (116m) 3s ago 116m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (106m) 8s ago 2h 71.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (105m) 8s ago 2h 73.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (104m) 8s ago 119m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (104m) 8s ago 119m 73.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (103m) 3s ago 119m 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (102m) 3s ago 118m 66.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:11:17.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (101m) 3s ago 118m 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:11:17.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (100m) 3s ago 118m 66.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:11:17.120 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (109m) 3s ago 117m 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:11:17.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (100m) 8s ago 115m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:11:17.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (100m) 3s ago 115m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:11:17.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (100m) 8s ago 115m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:11:17.120 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (100m) 3s ago 115m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:17.440 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:11:17.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:11:17.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:17.441 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:11:17.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:11:17.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:11:17.441 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:11:17.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:17 smithi078 ceph-mon[91136]: from='client.71522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:17.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:17 smithi078 ceph-mon[91136]: from='client.71528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:17.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:17 smithi078 ceph-mon[99959]: from='client.71522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:17.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:17 smithi078 ceph-mon[99959]: from='client.71528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:17 smithi146 ceph-mon[103278]: from='client.71522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:17 smithi146 ceph-mon[103278]: from='client.71528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:11:17.712 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:11:17.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:11:18.022 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:11:18.022 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:18.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:18.022 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:18.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[91136]: pgmap v3341: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:11:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[91136]: from='client.51615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2971319638' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[91136]: from='client.51627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4198399470' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[99959]: pgmap v3341: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[99959]: from='client.51615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2971319638' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[99959]: from='client.51627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4198399470' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:18 smithi146 ceph-mon[103278]: pgmap v3341: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:11:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:18 smithi146 ceph-mon[103278]: from='client.51615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2971319638' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:18 smithi146 ceph-mon[103278]: from='client.51627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4198399470' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:20 smithi146 ceph-mon[103278]: pgmap v3342: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T22:11:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:20 smithi078 ceph-mon[91136]: pgmap v3342: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T22:11:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:20 smithi078 ceph-mon[99959]: pgmap v3342: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-20T22:11:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:11:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:11:21.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:20.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:20.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:20.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:22 smithi146 ceph-mon[103278]: pgmap v3343: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T22:11:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:22 smithi078 ceph-mon[91136]: pgmap v3343: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T22:11:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:22 smithi078 ceph-mon[99959]: pgmap v3343: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T22:11:24.254 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:24.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:24.255 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:24.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:24.255 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:24.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:24 smithi146 ceph-mon[103278]: pgmap v3344: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:11:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:24 smithi078 ceph-mon[99959]: pgmap v3344: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:11:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:24 smithi078 ceph-mon[91136]: pgmap v3344: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:11:25.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:25.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:26 smithi146 ceph-mon[103278]: pgmap v3345: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:11:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:26 smithi078 ceph-mon[91136]: pgmap v3345: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:11:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:26 smithi078 ceph-mon[99959]: pgmap v3345: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:11:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:28 smithi146 ceph-mon[103278]: pgmap v3346: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:11:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:28 smithi078 ceph-mon[91136]: pgmap v3346: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:11:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:28 smithi078 ceph-mon[99959]: pgmap v3346: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:11:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:30.871 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:11:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:11:30.871 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:30.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:30.871 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:30.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:30.871 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:30.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:30 smithi146 ceph-mon[103278]: pgmap v3347: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:30 smithi078 ceph-mon[91136]: pgmap v3347: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:30 smithi078 ceph-mon[99959]: pgmap v3347: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:32 smithi146 ceph-mon[103278]: pgmap v3348: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:32 smithi078 ceph-mon[91136]: pgmap v3348: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:33.304 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:32 smithi078 ceph-mon[99959]: pgmap v3348: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:34.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:34.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:34.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:34 smithi146 ceph-mon[103278]: pgmap v3349: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[99959]: pgmap v3349: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[91136]: pgmap v3349: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:36 smithi146 ceph-mon[103278]: pgmap v3350: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:36 smithi078 ceph-mon[91136]: pgmap v3350: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:36 smithi078 ceph-mon[99959]: pgmap v3350: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:38 smithi146 ceph-mon[103278]: pgmap v3351: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:38 smithi078 ceph-mon[99959]: pgmap v3351: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:38 smithi078 ceph-mon[91136]: pgmap v3351: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:40 smithi146 ceph-mon[103278]: pgmap v3352: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:40 smithi078 ceph-mon[99959]: pgmap v3352: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:40 smithi078 ceph-mon[91136]: pgmap v3352: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:11:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:11:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:40.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:40.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:40.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:42 smithi146 ceph-mon[103278]: pgmap v3353: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:42 smithi078 ceph-mon[91136]: pgmap v3353: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:42 smithi078 ceph-mon[99959]: pgmap v3353: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:44.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:44.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:44.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:44.652 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:44 smithi078 ceph-mon[99959]: pgmap v3354: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:44.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:44 smithi078 ceph-mon[91136]: pgmap v3354: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:44 smithi146 ceph-mon[103278]: pgmap v3354: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:46.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:46 smithi146 ceph-mon[103278]: pgmap v3355: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:46.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:46 smithi078 ceph-mon[91136]: pgmap v3355: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:46.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:46 smithi078 ceph-mon[99959]: pgmap v3355: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:48.318 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:11:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:48 smithi146 ceph-mon[103278]: pgmap v3356: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:48 smithi078 ceph-mon[99959]: pgmap v3356: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:48 smithi078 ceph-mon[91136]: pgmap v3356: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:48.843 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:11:48.843 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 2h - - 2024-09-20T22:11:48.843 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 2h - - 2024-09-20T22:11:48.843 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (116m) 19s ago 117m 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:11:48.843 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (117m) 15s ago 117m 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (109m) 19s ago 115m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (108m) 15s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (110m) 19s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (108m) 19s ago 2h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (107m) 15s ago 2h 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (108m) 19s ago 2h 154M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (117m) 19s ago 117m 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (117m) 15s ago 117m 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (106m) 19s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (105m) 19s ago 2h 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (105m) 19s ago 2h 66.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (104m) 19s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (104m) 15s ago 119m 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (102m) 15s ago 119m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:11:48.844 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (102m) 15s ago 118m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:11:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (101m) 15s ago 118m 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:11:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (110m) 15s ago 117m 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:11:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (100m) 19s ago 115m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:11:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (100m) 15s ago 115m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:11:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (100m) 19s ago 115m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:11:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (100m) 15s ago 115m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:11:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:11:49.163 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:11:49.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:11:49.435 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:11:49.436 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:11:49.436 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='client.51639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='client.71564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='client.51651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1699595596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='client.51639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='client.71564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='client.51651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1699595596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:49.751 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:11:49.751 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:11:49.751 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='client.51639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='client.71564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='client.51651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1699595596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:50 smithi146 ceph-mon[103278]: pgmap v3357: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:50 smithi146 ceph-mon[103278]: from='client.51663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/924589766' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-mon[99959]: pgmap v3357: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-mon[99959]: from='client.51663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/924589766' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-mon[91136]: pgmap v3357: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-mon[91136]: from='client.51663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:11:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/924589766' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:11:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:11:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:11:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:50.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:50.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:50.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:52 smithi146 ceph-mon[103278]: pgmap v3358: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:52 smithi078 ceph-mon[99959]: pgmap v3358: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:52 smithi078 ceph-mon[91136]: pgmap v3358: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:54.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:54.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:11:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:11:54.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:11:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:54 smithi146 ceph-mon[103278]: pgmap v3359: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[91136]: pgmap v3359: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[99959]: pgmap v3359: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:11:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:11:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:11:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:56 smithi146 ceph-mon[103278]: pgmap v3360: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:56 smithi078 ceph-mon[91136]: pgmap v3360: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:56 smithi078 ceph-mon[99959]: pgmap v3360: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:58 smithi146 ceph-mon[103278]: pgmap v3361: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:11:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:58 smithi078 ceph-mon[91136]: pgmap v3361: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:11:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:11:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:58 smithi078 ceph-mon[99959]: pgmap v3361: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:11:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:11:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:00 smithi146 ceph-mon[103278]: pgmap v3362: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:00 smithi078 ceph-mon[91136]: pgmap v3362: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:00 smithi078 ceph-mon[99959]: pgmap v3362: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:00.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:12:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:12:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:00.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:00.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:00.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:00.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:02 smithi146 ceph-mon[103278]: pgmap v3363: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:02 smithi078 ceph-mon[91136]: pgmap v3363: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:02 smithi078 ceph-mon[99959]: pgmap v3363: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:04.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:04.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:04.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:04 smithi146 ceph-mon[103278]: pgmap v3364: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:04 smithi078 ceph-mon[91136]: pgmap v3364: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:04 smithi078 ceph-mon[99959]: pgmap v3364: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:06.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:06 smithi078 ceph-mon[91136]: pgmap v3365: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:06.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:06 smithi078 ceph-mon[99959]: pgmap v3365: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:06 smithi146 ceph-mon[103278]: pgmap v3365: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:08 smithi078 ceph-mon[91136]: pgmap v3366: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:08 smithi078 ceph-mon[99959]: pgmap v3366: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:08 smithi146 ceph-mon[103278]: pgmap v3366: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[99959]: pgmap v3367: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[91136]: pgmap v3367: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:10 smithi146 ceph-mon[103278]: pgmap v3367: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:10.690 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:12:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:12:10.690 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:10.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:10.690 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:10.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:10.690 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:10.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:12.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:12 smithi146 ceph-mon[103278]: pgmap v3368: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:12 smithi078 ceph-mon[99959]: pgmap v3368: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:12 smithi078 ceph-mon[91136]: pgmap v3368: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[99959]: pgmap v3369: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[91136]: pgmap v3369: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:14.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:14.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:14.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:14.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:14 smithi146 ceph-mon[103278]: pgmap v3369: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:16.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:16 smithi146 ceph-mon[103278]: pgmap v3370: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:16 smithi078 ceph-mon[99959]: pgmap v3370: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:16 smithi078 ceph-mon[91136]: pgmap v3370: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:18 smithi146 ceph-mon[103278]: pgmap v3371: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:18 smithi078 ceph-mon[99959]: pgmap v3371: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:18 smithi078 ceph-mon[91136]: pgmap v3371: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:20.046 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:12:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:20 smithi146 ceph-mon[103278]: pgmap v3372: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:20 smithi078 ceph-mon[99959]: pgmap v3372: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:20 smithi078 ceph-mon[91136]: pgmap v3372: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:20.574 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 2h - - 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 2h - - 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (117m) 11s ago 118m 30.2M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (117m) 6s ago 117m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (110m) 11s ago 115m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (109m) 6s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (111m) 11s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (109m) 11s ago 2h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (108m) 6s ago 2h 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (108m) 11s ago 2h 154M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (117m) 11s ago 117m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:12:20.575 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (117m) 6s ago 117m 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (107m) 11s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (106m) 11s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (105m) 11s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (105m) 11s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (104m) 6s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (103m) 6s ago 119m 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (102m) 6s ago 119m 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (101m) 6s ago 119m 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (110m) 6s ago 118m 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (101m) 11s ago 116m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (101m) 6s ago 116m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (101m) 11s ago 116m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:12:20.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (101m) 6s ago 116m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:12:20.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:12:20.882 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:12:20.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:12:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:12:20.883 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:20.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:20.883 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:20.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:20.883 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:20.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:21.173 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:12:21.173 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:12:21.173 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:12:21.173 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:12:21.173 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:12:21.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:12:21.174 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:12:21.483 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:12:21.483 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:21.483 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:21.483 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:21.483 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[99959]: from='client.71594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[99959]: from='client.51681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[99959]: from='client.51687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/767284652' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[91136]: from='client.71594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[91136]: from='client.51681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[91136]: from='client.51687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/767284652' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:21 smithi146 ceph-mon[103278]: from='client.71594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:21 smithi146 ceph-mon[103278]: from='client.51681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:21 smithi146 ceph-mon[103278]: from='client.51687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/767284652' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:22 smithi146 ceph-mon[103278]: pgmap v3373: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:22 smithi146 ceph-mon[103278]: from='client.51699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/459689736' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:22 smithi078 ceph-mon[99959]: pgmap v3373: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:22 smithi078 ceph-mon[99959]: from='client.51699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/459689736' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:22 smithi078 ceph-mon[91136]: pgmap v3373: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:22 smithi078 ceph-mon[91136]: from='client.51699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/459689736' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:24 smithi078 ceph-mon[91136]: pgmap v3374: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:24 smithi078 ceph-mon[99959]: pgmap v3374: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:24.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:24.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:24.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:24 smithi146 ceph-mon[103278]: pgmap v3374: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:26 smithi146 ceph-mon[103278]: pgmap v3375: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:26 smithi078 ceph-mon[99959]: pgmap v3375: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:26 smithi078 ceph-mon[91136]: pgmap v3375: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:28 smithi146 ceph-mon[103278]: pgmap v3376: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:28 smithi078 ceph-mon[91136]: pgmap v3376: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:28 smithi078 ceph-mon[99959]: pgmap v3376: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:30 smithi146 ceph-mon[103278]: pgmap v3377: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[91136]: pgmap v3377: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[99959]: pgmap v3377: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:12:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:12:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:30.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:30.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:30.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:32 smithi146 ceph-mon[103278]: pgmap v3378: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:32 smithi078 ceph-mon[91136]: pgmap v3378: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:32 smithi078 ceph-mon[99959]: pgmap v3378: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:12:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:34.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:34.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:34.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:34.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:34 smithi146 ceph-mon[103278]: pgmap v3379: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[91136]: pgmap v3379: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[99959]: pgmap v3379: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:36 smithi078 ceph-mon[91136]: pgmap v3380: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:36 smithi078 ceph-mon[99959]: pgmap v3380: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:36 smithi146 ceph-mon[103278]: pgmap v3380: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:38 smithi078 ceph-mon[91136]: pgmap v3381: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:38 smithi078 ceph-mon[99959]: pgmap v3381: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:38 smithi146 ceph-mon[103278]: pgmap v3381: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:40 smithi078 ceph-mon[99959]: pgmap v3382: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:40 smithi078 ceph-mon[91136]: pgmap v3382: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:40.584Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:40.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:40.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:12:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:12:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:40 smithi146 ceph-mon[103278]: pgmap v3382: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:42 smithi078 ceph-mon[99959]: pgmap v3383: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:42 smithi078 ceph-mon[91136]: pgmap v3383: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:42 smithi146 ceph-mon[103278]: pgmap v3383: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:44.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:44.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:44.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:44.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:44.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:44.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:44 smithi078 ceph-mon[99959]: pgmap v3384: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:44 smithi078 ceph-mon[91136]: pgmap v3384: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:44 smithi146 ceph-mon[103278]: pgmap v3384: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:46.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:46 smithi078 ceph-mon[91136]: pgmap v3385: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:46 smithi078 ceph-mon[99959]: pgmap v3385: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:46 smithi146 ceph-mon[103278]: pgmap v3385: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:48.903 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:48 smithi078 ceph-mon[91136]: pgmap v3386: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:48.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:48 smithi078 ceph-mon[99959]: pgmap v3386: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:48 smithi146 ceph-mon[103278]: pgmap v3386: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:50 smithi146 ceph-mon[103278]: pgmap v3387: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[91136]: pgmap v3387: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:12:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:12:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[99959]: pgmap v3387: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:50.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:50.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:50.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:50.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:51.790 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:12:52.315 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 2h - - 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 2h - - 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (117m) 2s ago 118m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (118m) 18s ago 118m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (111m) 2s ago 116m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (109m) 18s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (111m) 2s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (110m) 2s ago 2h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (108m) 18s ago 2h 147M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (109m) 2s ago 2h 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (118m) 2s ago 118m 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (118m) 18s ago 118m 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:12:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (107m) 2s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (106m) 2s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (106m) 2s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (105m) 2s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (105m) 18s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (104m) 18s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (103m) 18s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (102m) 18s ago 119m 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (111m) 18s ago 119m 73.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (102m) 2s ago 116m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (101m) 18s ago 116m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (101m) 2s ago 116m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:12:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (101m) 18s ago 116m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:12:52.628 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:12:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:12:52.629 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:12:52.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:52 smithi078 ceph-mon[91136]: pgmap v3388: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:52.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:52 smithi078 ceph-mon[91136]: from='client.51711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:52.630 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:52 smithi078 ceph-mon[99959]: pgmap v3388: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:52.630 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:52 smithi078 ceph-mon[99959]: from='client.51711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:52.694 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:52 smithi146 ceph-mon[103278]: pgmap v3388: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:52.694 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:52 smithi146 ceph-mon[103278]: from='client.51711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:52.897 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:12:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:12:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:12:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:12:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:12:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:12:52.898 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:12:53.208 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:12:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:12:53.209 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:12:53.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:53 smithi146 ceph-mon[103278]: from='client.51717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:53.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:53 smithi146 ceph-mon[103278]: from='client.51723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:53.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3673202447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:53.465 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2605098866' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[91136]: from='client.51717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[91136]: from='client.51723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3673202447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2605098866' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[99959]: from='client.51717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[99959]: from='client.51723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3673202447' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:12:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2605098866' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:12:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:54.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:54.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:12:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:12:54.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:12:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:54 smithi146 ceph-mon[103278]: from='client.71654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:54 smithi146 ceph-mon[103278]: pgmap v3389: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[91136]: from='client.71654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[91136]: pgmap v3389: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[99959]: from='client.71654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:12:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[99959]: pgmap v3389: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:12:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:12:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:12:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:56 smithi146 ceph-mon[103278]: pgmap v3390: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:56 smithi078 ceph-mon[91136]: pgmap v3390: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:56 smithi078 ceph-mon[99959]: pgmap v3390: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:57 smithi146 ceph-mon[103278]: pgmap v3391: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:57 smithi078 ceph-mon[91136]: pgmap v3391: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:57 smithi078 ceph-mon[99959]: pgmap v3391: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:12:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:12:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:12:59 smithi146 ceph-mon[103278]: pgmap v3392: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:12:59 smithi078 ceph-mon[91136]: pgmap v3392: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:12:59 smithi078 ceph-mon[99959]: pgmap v3392: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:13:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:13:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:00.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:00.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:00.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:02 smithi146 ceph-mon[103278]: pgmap v3393: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:02 smithi078 ceph-mon[99959]: pgmap v3393: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:02 smithi078 ceph-mon[91136]: pgmap v3393: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:04 smithi078 ceph-mon[91136]: pgmap v3394: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:04 smithi078 ceph-mon[99959]: pgmap v3394: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:04.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:04.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:04.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:04 smithi146 ceph-mon[103278]: pgmap v3394: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:06 smithi146 ceph-mon[103278]: pgmap v3395: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:06 smithi078 ceph-mon[99959]: pgmap v3395: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:06 smithi078 ceph-mon[91136]: pgmap v3395: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:08.407 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:08 smithi078 ceph-mon[99959]: pgmap v3396: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:08.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:08 smithi078 ceph-mon[91136]: pgmap v3396: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:08 smithi146 ceph-mon[103278]: pgmap v3396: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:10.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[91136]: pgmap v3397: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:10.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.589 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[99959]: pgmap v3397: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:10.589 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.589 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.589 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:10 smithi146 ceph-mon[103278]: pgmap v3397: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:13:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:13:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:10.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:10.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:10.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:12 smithi146 ceph-mon[103278]: pgmap v3398: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:12 smithi078 ceph-mon[91136]: pgmap v3398: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:12 smithi078 ceph-mon[99959]: pgmap v3398: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:14.013Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:14.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:14.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:14.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:14 smithi146 ceph-mon[103278]: pgmap v3399: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:14.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.541 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[99959]: pgmap v3399: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[91136]: pgmap v3399: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:16 smithi146 ceph-mon[103278]: pgmap v3400: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:16 smithi078 ceph-mon[91136]: pgmap v3400: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:16 smithi078 ceph-mon[99959]: pgmap v3400: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:18 smithi078 ceph-mon[99959]: pgmap v3401: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:18 smithi078 ceph-mon[91136]: pgmap v3401: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:18 smithi146 ceph-mon[103278]: pgmap v3401: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:20 smithi078 ceph-mon[99959]: pgmap v3402: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:13:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:13:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:20 smithi078 ceph-mon[91136]: pgmap v3402: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:20.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:20.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:20.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:20 smithi146 ceph-mon[103278]: pgmap v3402: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:22 smithi078 ceph-mon[91136]: pgmap v3403: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:22 smithi078 ceph-mon[99959]: pgmap v3403: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:22 smithi146 ceph-mon[103278]: pgmap v3403: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:23.503 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:13:24.031 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 2h - - 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 2h - - 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (118m) 14s ago 119m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (119m) 9s ago 118m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (111m) 14s ago 116m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (110m) 9s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (112m) 14s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (110m) 14s ago 2h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (109m) 9s ago 2h 148M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (109m) 14s ago 2h 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (118m) 14s ago 118m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:13:24.032 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (118m) 9s ago 118m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (108m) 14s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (107m) 14s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (106m) 14s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (106m) 14s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (105m) 9s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (104m) 9s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (103m) 9s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (102m) 9s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (111m) 9s ago 119m 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (102m) 14s ago 117m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (102m) 9s ago 117m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (102m) 14s ago 117m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:13:24.033 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (102m) 9s ago 117m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:13:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:24.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:24.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:24.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:24.344 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:13:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:13:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:13:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:13:24.345 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:13:24.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[91136]: pgmap v3404: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:24.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[91136]: from='client.51747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:24.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[91136]: from='client.71672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:24.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1121670869' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:24.595 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[99959]: pgmap v3404: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:24.595 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[99959]: from='client.51747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:24.596 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[99959]: from='client.71672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:24.596 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1121670869' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:24.614 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:13:24.614 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:13:24.614 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:13:24.614 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:13:24.614 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:13:24.615 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:13:24.921 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:13:24.921 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:24.921 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:24.921 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:24.921 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:24 smithi146 ceph-mon[103278]: pgmap v3404: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:24 smithi146 ceph-mon[103278]: from='client.51747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:24 smithi146 ceph-mon[103278]: from='client.71672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1121670869' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:25 smithi146 ceph-mon[103278]: from='client.71678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:25 smithi146 ceph-mon[103278]: from='client.71690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/178958443' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[91136]: from='client.71678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[91136]: from='client.71690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/178958443' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[99959]: from='client.71678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[99959]: from='client.71690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/178958443' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:26 smithi146 ceph-mon[103278]: pgmap v3405: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:26 smithi078 ceph-mon[91136]: pgmap v3405: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:26 smithi078 ceph-mon[99959]: pgmap v3405: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:28 smithi078 ceph-mon[99959]: pgmap v3406: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:28 smithi078 ceph-mon[91136]: pgmap v3406: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:28 smithi146 ceph-mon[103278]: pgmap v3406: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:30 smithi146 ceph-mon[103278]: pgmap v3407: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:13:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[91136]: pgmap v3407: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[99959]: pgmap v3407: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:30.588Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:30.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:30.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:32 smithi146 ceph-mon[103278]: pgmap v3408: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:32 smithi078 ceph-mon[91136]: pgmap v3408: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:32 smithi078 ceph-mon[99959]: pgmap v3408: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:34.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:34.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:34.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:34.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:34 smithi146 ceph-mon[103278]: pgmap v3409: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:34.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:34.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:34.809 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[91136]: pgmap v3409: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[99959]: pgmap v3409: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:35.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:36 smithi146 ceph-mon[103278]: pgmap v3410: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:36 smithi078 ceph-mon[91136]: pgmap v3410: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:36 smithi078 ceph-mon[99959]: pgmap v3410: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:38 smithi146 ceph-mon[103278]: pgmap v3411: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:38 smithi078 ceph-mon[91136]: pgmap v3411: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:38 smithi078 ceph-mon[99959]: pgmap v3411: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:40.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:40 smithi078 ceph-mon[99959]: pgmap v3412: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:40.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:40 smithi078 ceph-mon[91136]: pgmap v3412: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:40.858 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:13:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:13:40.859 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:40.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:40.859 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:40.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:40.859 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:40.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:40 smithi146 ceph-mon[103278]: pgmap v3412: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:42 smithi146 ceph-mon[103278]: pgmap v3413: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:42 smithi078 ceph-mon[91136]: pgmap v3413: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:42 smithi078 ceph-mon[99959]: pgmap v3413: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:44.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:44.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:44.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:44.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:44.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:44 smithi146 ceph-mon[103278]: pgmap v3414: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:44 smithi078 ceph-mon[91136]: pgmap v3414: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:44 smithi078 ceph-mon[99959]: pgmap v3414: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:46 smithi146 ceph-mon[103278]: pgmap v3415: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:46 smithi078 ceph-mon[91136]: pgmap v3415: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:46 smithi078 ceph-mon[99959]: pgmap v3415: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:48.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:48 smithi078 ceph-mon[91136]: pgmap v3416: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:48.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:48 smithi078 ceph-mon[99959]: pgmap v3416: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:48 smithi146 ceph-mon[103278]: pgmap v3416: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:50 smithi146 ceph-mon[103278]: pgmap v3417: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:13:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:13:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[99959]: pgmap v3417: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[91136]: pgmap v3417: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:50.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:50.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:50.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:52 smithi146 ceph-mon[103278]: pgmap v3418: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:52 smithi078 ceph-mon[91136]: pgmap v3418: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:52 smithi078 ceph-mon[99959]: pgmap v3418: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:54.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:54.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:13:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:13:54.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:13:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:54 smithi146 ceph-mon[103278]: pgmap v3419: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[99959]: pgmap v3419: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[91136]: pgmap v3419: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:13:55.217 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:13:55.739 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 2h - - 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (118m) 5s ago 119m 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (119m) 1s ago 119m 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (112m) 5s ago 117m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (110m) 1s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (112m) 5s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (111m) 5s ago 2h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (109m) 1s ago 2h 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (110m) 5s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (119m) 5s ago 119m 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:13:55.740 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (119m) 1s ago 119m 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (108m) 5s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (107m) 5s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (107m) 5s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (106m) 5s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (106m) 1s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (105m) 1s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (104m) 1s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (103m) 1s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (112m) 1s ago 2h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (103m) 5s ago 117m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (102m) 1s ago 117m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (102m) 5s ago 117m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:13:55.741 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (102m) 1s ago 117m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:13:55.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:13:56.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:13:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:13:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:13:56.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:13:56.049 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:13:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:13:56.319 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:13:56.319 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:13:56.625 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:13:56.625 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:56.625 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:13:56.626 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:56.626 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[91136]: pgmap v3420: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[91136]: from='client.71702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[91136]: from='client.71708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[91136]: from='client.71714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3530562346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3505691727' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[99959]: pgmap v3420: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[99959]: from='client.71702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[99959]: from='client.71708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[99959]: from='client.71714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3530562346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3505691727' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:56 smithi146 ceph-mon[103278]: pgmap v3420: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:56 smithi146 ceph-mon[103278]: from='client.71702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:56 smithi146 ceph-mon[103278]: from='client.71708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:56 smithi146 ceph-mon[103278]: from='client.71714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3530562346' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:13:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3505691727' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:13:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:57 smithi078 ceph-mon[91136]: from='client.51807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:57 smithi078 ceph-mon[99959]: from='client.51807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:57 smithi146 ceph-mon[103278]: from='client.51807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:13:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:58 smithi078 ceph-mon[91136]: pgmap v3421: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:58 smithi078 ceph-mon[99959]: pgmap v3421: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:13:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:58 smithi146 ceph-mon[103278]: pgmap v3421: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:13:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:00 smithi078 ceph-mon[91136]: pgmap v3422: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:00 smithi078 ceph-mon[99959]: pgmap v3422: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:14:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:14:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:00.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:00.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:00.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:00 smithi146 ceph-mon[103278]: pgmap v3422: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:02 smithi078 ceph-mon[91136]: pgmap v3423: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:02 smithi078 ceph-mon[99959]: pgmap v3423: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:02 smithi146 ceph-mon[103278]: pgmap v3423: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:04.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:04.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:04.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:04 smithi078 ceph-mon[91136]: pgmap v3424: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:04 smithi078 ceph-mon[99959]: pgmap v3424: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:04 smithi146 ceph-mon[103278]: pgmap v3424: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:06 smithi146 ceph-mon[103278]: pgmap v3425: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:06 smithi078 ceph-mon[91136]: pgmap v3425: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:06 smithi078 ceph-mon[99959]: pgmap v3425: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:09.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:08 smithi078 ceph-mon[91136]: pgmap v3426: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:09.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:08 smithi078 ceph-mon[99959]: pgmap v3426: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:08 smithi146 ceph-mon[103278]: pgmap v3426: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[91136]: pgmap v3427: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:14:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[99959]: pgmap v3427: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:10.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:10.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:10.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:10 smithi146 ceph-mon[103278]: pgmap v3427: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:12 smithi146 ceph-mon[103278]: pgmap v3428: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:12 smithi078 ceph-mon[91136]: pgmap v3428: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:12 smithi078 ceph-mon[99959]: pgmap v3428: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:14.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:14.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:14.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:14 smithi146 ceph-mon[103278]: pgmap v3429: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[99959]: pgmap v3429: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[91136]: pgmap v3429: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:16.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:16.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:16 smithi146 ceph-mon[103278]: pgmap v3430: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:16 smithi078 ceph-mon[91136]: pgmap v3430: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:16 smithi078 ceph-mon[99959]: pgmap v3430: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:18 smithi146 ceph-mon[103278]: pgmap v3431: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:18 smithi078 ceph-mon[91136]: pgmap v3431: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:18 smithi078 ceph-mon[99959]: pgmap v3431: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:20 smithi078 ceph-mon[91136]: pgmap v3432: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:20 smithi078 ceph-mon[99959]: pgmap v3432: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:14:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:14:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:20.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:20.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:20.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:20 smithi146 ceph-mon[103278]: pgmap v3432: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:22 smithi146 ceph-mon[103278]: pgmap v3433: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:22 smithi078 ceph-mon[91136]: pgmap v3433: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:22 smithi078 ceph-mon[99959]: pgmap v3433: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:24.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:24.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:24.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:24 smithi078 ceph-mon[91136]: pgmap v3434: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:24 smithi078 ceph-mon[99959]: pgmap v3434: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:24 smithi146 ceph-mon[103278]: pgmap v3434: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 22:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T22:14:25.850208297Z level=info msg="Completed cleanup jobs" duration=14.230268ms 2024-09-20T22:14:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:26.929 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:14:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:26 smithi146 ceph-mon[103278]: pgmap v3435: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:26 smithi078 ceph-mon[99959]: pgmap v3435: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:26 smithi078 ceph-mon[91136]: pgmap v3435: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:27.453 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 2h - - 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 2h - - 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (119m) 17s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 13s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (112m) 17s ago 118m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (111m) 13s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (113m) 17s ago 2h 565M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (111m) 17s ago 2h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (110m) 13s ago 2h 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (110m) 17s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (119m) 17s ago 119m 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (119m) 13s ago 119m 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (109m) 17s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:14:27.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (108m) 17s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (107m) 17s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (107m) 17s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (106m) 13s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (105m) 13s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (104m) 13s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (103m) 13s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (112m) 13s ago 2h 74.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (103m) 17s ago 118m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (103m) 13s ago 118m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (103m) 17s ago 118m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:14:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (103m) 13s ago 118m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:14:27.764 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:27.765 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:14:27.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:14:27.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:14:27.766 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:14:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/222216443' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/222216443' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:14:28.039 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:14:28.040 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:14:28.040 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:14:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/222216443' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:28.351 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:14:28.351 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:14:28.351 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:14:28.351 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:14:28.351 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:28 smithi146 ceph-mon[103278]: from='client.51819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:28 smithi146 ceph-mon[103278]: pgmap v3436: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:28 smithi146 ceph-mon[103278]: from='client.71744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:28 smithi146 ceph-mon[103278]: from='client.71750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3092448890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[99959]: from='client.51819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[99959]: pgmap v3436: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[99959]: from='client.71744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[99959]: from='client.71750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3092448890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[91136]: from='client.51819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[91136]: pgmap v3436: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[91136]: from='client.71744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[91136]: from='client.71750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3092448890' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:14:30.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:29 smithi078 ceph-mon[99959]: from='client.71762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:30.069 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:29 smithi078 ceph-mon[91136]: from='client.71762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:29 smithi146 ceph-mon[103278]: from='client.71762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:30.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:14:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:14:30.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:30.594Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:30.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:30.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:30.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:30.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:31.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[91136]: pgmap v3437: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:31.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[99959]: pgmap v3437: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:31.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:30 smithi146 ceph-mon[103278]: pgmap v3437: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:32 smithi078 ceph-mon[91136]: pgmap v3438: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:32 smithi078 ceph-mon[99959]: pgmap v3438: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:32 smithi146 ceph-mon[103278]: pgmap v3438: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:34.020Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:34.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:34.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[99959]: pgmap v3439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[91136]: pgmap v3439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:34 smithi146 ceph-mon[103278]: pgmap v3439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:36.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:36 smithi146 ceph-mon[103278]: pgmap v3440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:36 smithi078 ceph-mon[91136]: pgmap v3440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:36 smithi078 ceph-mon[99959]: pgmap v3440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:38 smithi146 ceph-mon[103278]: pgmap v3441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:38 smithi078 ceph-mon[91136]: pgmap v3441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:38 smithi078 ceph-mon[99959]: pgmap v3441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:40.882 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:14:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:14:40.883 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:40.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:40.883 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:40.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:40.883 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:40.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:40 smithi146 ceph-mon[103278]: pgmap v3442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:40 smithi078 ceph-mon[99959]: pgmap v3442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:40 smithi078 ceph-mon[91136]: pgmap v3442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:42 smithi146 ceph-mon[103278]: pgmap v3443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:42 smithi078 ceph-mon[99959]: pgmap v3443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:42 smithi078 ceph-mon[91136]: pgmap v3443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:44.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:44.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:44.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:44 smithi078 ceph-mon[91136]: pgmap v3444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:44 smithi078 ceph-mon[99959]: pgmap v3444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:44 smithi146 ceph-mon[103278]: pgmap v3444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:46.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:46 smithi146 ceph-mon[103278]: pgmap v3445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:46 smithi078 ceph-mon[99959]: pgmap v3445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:46 smithi078 ceph-mon[91136]: pgmap v3445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:47 smithi146 ceph-mon[103278]: pgmap v3446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:47 smithi078 ceph-mon[91136]: pgmap v3446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:47 smithi078 ceph-mon[99959]: pgmap v3446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:50 smithi146 ceph-mon[103278]: pgmap v3447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[99959]: pgmap v3447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:50.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[91136]: pgmap v3447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:50.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:50.783 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:14:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:14:50.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:50.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:50.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:50.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:50.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:50.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:52.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:52 smithi078 ceph-mon[91136]: pgmap v3448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:52.396 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:52 smithi078 ceph-mon[99959]: pgmap v3448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:52 smithi146 ceph-mon[103278]: pgmap v3448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:54 smithi078 ceph-mon[91136]: pgmap v3449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:54 smithi078 ceph-mon[99959]: pgmap v3449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:54.022Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:54.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:14:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:14:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:14:54.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:14:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:54 smithi146 ceph-mon[103278]: pgmap v3449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:14:55.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:55.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:55.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:55.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:14:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:14:56.984 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:56 smithi146 ceph-mon[103278]: pgmap v3450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:56 smithi078 ceph-mon[91136]: pgmap v3450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:56 smithi078 ceph-mon[99959]: pgmap v3450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:58.644 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:14:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:58 smithi146 ceph-mon[103278]: pgmap v3451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:58 smithi078 ceph-mon[99959]: pgmap v3451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:58 smithi078 ceph-mon[91136]: pgmap v3451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:14:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 2h - - 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 2h - - 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (119m) 9s ago 2h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 4s ago 2h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (113m) 9s ago 118m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (111m) 4s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (113m) 9s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (112m) 9s ago 2h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (110m) 4s ago 2h 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (111m) 9s ago 2h 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 9s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:14:59.169 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 4s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (109m) 9s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (108m) 9s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (108m) 9s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (107m) 9s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (107m) 4s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (106m) 4s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (105m) 4s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (104m) 4s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (113m) 4s ago 2h 75.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (104m) 9s ago 118m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (103m) 4s ago 118m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (104m) 9s ago 118m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:14:59.170 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (103m) 4s ago 118m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:14:59.478 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:14:59.479 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:14:59.750 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:14:59.750 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:14:59.750 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:14:59.750 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:14:59.750 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:14:59.751 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:14:59.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:59 smithi078 ceph-mon[91136]: from='client.51855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:59.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:59 smithi078 ceph-mon[91136]: from='client.51861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:59.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:14:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1657484915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:14:59.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:59 smithi078 ceph-mon[99959]: from='client.51855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:59.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:59 smithi078 ceph-mon[99959]: from='client.51861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:14:59.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:14:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1657484915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:59 smithi146 ceph-mon[103278]: from='client.51855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:59 smithi146 ceph-mon[103278]: from='client.51861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:14:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1657484915' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:00.063 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:15:00.063 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:00.063 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:00 smithi146 ceph-mon[103278]: pgmap v3452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:00 smithi146 ceph-mon[103278]: from='client.51867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:00 smithi146 ceph-mon[103278]: from='client.51879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2551466459' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:15:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:15:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[91136]: pgmap v3452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[91136]: from='client.51867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[91136]: from='client.51879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2551466459' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[99959]: pgmap v3452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[99959]: from='client.51867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[99959]: from='client.51879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2551466459' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:00.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:00.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:00.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:02 smithi146 ceph-mon[103278]: pgmap v3453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:02 smithi078 ceph-mon[91136]: pgmap v3453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:02 smithi078 ceph-mon[99959]: pgmap v3453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:04.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:04.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:04.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:04 smithi146 ceph-mon[103278]: pgmap v3454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:04 smithi078 ceph-mon[91136]: pgmap v3454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:04 smithi078 ceph-mon[99959]: pgmap v3454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:06 smithi146 ceph-mon[103278]: pgmap v3455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:06 smithi078 ceph-mon[99959]: pgmap v3455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:06 smithi078 ceph-mon[91136]: pgmap v3455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:08 smithi146 ceph-mon[103278]: pgmap v3456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:08 smithi078 ceph-mon[99959]: pgmap v3456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:08 smithi078 ceph-mon[91136]: pgmap v3456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:10 smithi146 ceph-mon[103278]: pgmap v3457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[91136]: pgmap v3457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:15:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:15:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[99959]: pgmap v3457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:10.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:10.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:10.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:12 smithi146 ceph-mon[103278]: pgmap v3458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:12 smithi078 ceph-mon[91136]: pgmap v3458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:12 smithi078 ceph-mon[99959]: pgmap v3458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:13.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:13.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:14.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:14.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:14.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:14 smithi146 ceph-mon[103278]: pgmap v3459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:14 smithi078 ceph-mon[91136]: pgmap v3459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:14 smithi078 ceph-mon[99959]: pgmap v3459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:16.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:16.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:16 smithi146 ceph-mon[103278]: pgmap v3460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:16 smithi078 ceph-mon[91136]: pgmap v3460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:16 smithi078 ceph-mon[99959]: pgmap v3460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:18 smithi146 ceph-mon[103278]: pgmap v3461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:18 smithi078 ceph-mon[91136]: pgmap v3461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:18 smithi078 ceph-mon[99959]: pgmap v3461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:20 smithi078 ceph-mon[91136]: pgmap v3462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:20 smithi078 ceph-mon[99959]: pgmap v3462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:15:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:15:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:20.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:20.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:20.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:20 smithi146 ceph-mon[103278]: pgmap v3462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:23.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:22 smithi078 ceph-mon[99959]: pgmap v3463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:23.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:22 smithi078 ceph-mon[91136]: pgmap v3463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:22 smithi146 ceph-mon[103278]: pgmap v3463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:24.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:24.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:24.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:24 smithi078 ceph-mon[99959]: pgmap v3464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:24 smithi078 ceph-mon[91136]: pgmap v3464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:24 smithi146 ceph-mon[103278]: pgmap v3464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:26 smithi146 ceph-mon[103278]: pgmap v3465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:26 smithi078 ceph-mon[91136]: pgmap v3465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:26 smithi078 ceph-mon[99959]: pgmap v3465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:28 smithi146 ceph-mon[103278]: pgmap v3466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:28 smithi078 ceph-mon[99959]: pgmap v3466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:28 smithi078 ceph-mon[91136]: pgmap v3466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:30.359 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:15:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:30.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:30.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:30.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:30.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 2h - - 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 2h - - 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 0s ago 2h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 16s ago 2h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (113m) 0s ago 119m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (112m) 16s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (114m) 0s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (112m) 0s ago 2h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (111m) 16s ago 2h 149M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (111m) 0s ago 2h 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:15:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 0s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 16s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (110m) 0s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (109m) 0s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (108m) 0s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (108m) 0s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (107m) 16s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (106m) 16s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (105m) 16s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (104m) 16s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (113m) 16s ago 2h 75.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (104m) 0s ago 119m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (104m) 16s ago 119m 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (104m) 0s ago 119m 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:15:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (104m) 16s ago 119m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:15:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[99959]: pgmap v3467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[91136]: pgmap v3467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:30.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:30.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:15:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:15:31.217 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:15:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:30 smithi146 ceph-mon[103278]: pgmap v3467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:15:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:15:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:15:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:15:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:15:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:15:31.485 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:15:31.797 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:15:31.797 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:15:31.797 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:15:32.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[99959]: from='client.71810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.065 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[99959]: from='client.71816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[99959]: from='client.71822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3612303887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3241987709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[91136]: from='client.71810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[91136]: from='client.71816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[91136]: from='client.71822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3612303887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:32.066 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3241987709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:31 smithi146 ceph-mon[103278]: from='client.71810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:31 smithi146 ceph-mon[103278]: from='client.71816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:31 smithi146 ceph-mon[103278]: from='client.71822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3612303887' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:15:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3241987709' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:15:33.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:32 smithi078 ceph-mon[99959]: pgmap v3468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:33.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:32 smithi078 ceph-mon[99959]: from='client.71834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:32 smithi078 ceph-mon[91136]: pgmap v3468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:32 smithi078 ceph-mon[91136]: from='client.71834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:32 smithi146 ceph-mon[103278]: pgmap v3468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:32 smithi146 ceph-mon[103278]: from='client.71834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:15:34.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:34.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:34.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:34.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:34.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:34.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:34 smithi078 ceph-mon[91136]: pgmap v3469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:34 smithi078 ceph-mon[99959]: pgmap v3469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:34 smithi146 ceph-mon[103278]: pgmap v3469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:37.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:36 smithi146 ceph-mon[103278]: pgmap v3470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:36 smithi078 ceph-mon[99959]: pgmap v3470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:36 smithi078 ceph-mon[91136]: pgmap v3470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:39.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:38 smithi146 ceph-mon[103278]: pgmap v3471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:38 smithi078 ceph-mon[99959]: pgmap v3471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:38 smithi078 ceph-mon[91136]: pgmap v3471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:40.874 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:15:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:15:40.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:40.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:40.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:40.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:40.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:40.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:40 smithi146 ceph-mon[103278]: pgmap v3472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:40 smithi078 ceph-mon[91136]: pgmap v3472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:40 smithi078 ceph-mon[99959]: pgmap v3472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:42 smithi146 ceph-mon[103278]: pgmap v3473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:42 smithi078 ceph-mon[91136]: pgmap v3473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:42 smithi078 ceph-mon[99959]: pgmap v3473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:44.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:44.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:44.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:44 smithi078 ceph-mon[91136]: pgmap v3474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:44 smithi078 ceph-mon[99959]: pgmap v3474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:44 smithi146 ceph-mon[103278]: pgmap v3474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:46.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:47.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:46 smithi146 ceph-mon[103278]: pgmap v3475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:46 smithi078 ceph-mon[91136]: pgmap v3475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:46 smithi078 ceph-mon[99959]: pgmap v3475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:48 smithi146 ceph-mon[103278]: pgmap v3476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:48 smithi078 ceph-mon[91136]: pgmap v3476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:48 smithi078 ceph-mon[99959]: pgmap v3476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:50.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:15:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:15:50.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:50.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:50.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:50.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:50.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:50.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:50 smithi146 ceph-mon[103278]: pgmap v3477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[91136]: pgmap v3477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[99959]: pgmap v3477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:53.110 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:52 smithi078 ceph-mon[99959]: pgmap v3478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:53.110 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:52 smithi078 ceph-mon[91136]: pgmap v3478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:52 smithi146 ceph-mon[103278]: pgmap v3478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:54.197 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:54.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:54.197 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:54.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:15:54.197 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:15:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:15:54.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:15:55.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:54 smithi078 ceph-mon[91136]: pgmap v3479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:54 smithi078 ceph-mon[99959]: pgmap v3479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:54 smithi146 ceph-mon[103278]: pgmap v3479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:15:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:15:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:15:57.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:56 smithi146 ceph-mon[103278]: pgmap v3480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:56 smithi078 ceph-mon[91136]: pgmap v3480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:56 smithi078 ceph-mon[99959]: pgmap v3480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:58 smithi146 ceph-mon[103278]: pgmap v3481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:15:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:58 smithi078 ceph-mon[91136]: pgmap v3481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:15:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:15:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:58 smithi078 ceph-mon[99959]: pgmap v3481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:15:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:15:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:00.910 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:16:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:16:00.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:00.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:00.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:00.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:00.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:00.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:00 smithi146 ceph-mon[103278]: pgmap v3482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:00 smithi078 ceph-mon[99959]: pgmap v3482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:00 smithi078 ceph-mon[91136]: pgmap v3482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:02.093 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:16:02.617 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 2h - - 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 2h - - 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 12s ago 2h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 7s ago 2h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (114m) 12s ago 119m 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (112m) 7s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (114m) 12s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (113m) 12s ago 2h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (111m) 7s ago 2h 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (112m) 12s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 12s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:16:02.618 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 7s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (110m) 12s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (109m) 12s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (109m) 12s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (108m) 12s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (108m) 7s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (107m) 7s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (106m) 7s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (105m) 7s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (114m) 7s ago 2h 75.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (105m) 12s ago 119m 88.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (105m) 7s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (105m) 12s ago 119m 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:16:02.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (104m) 7s ago 119m 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:02.946 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:16:02.947 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:16:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:16:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:16:03.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:02 smithi078 ceph-mon[99959]: pgmap v3483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:03.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:02 smithi078 ceph-mon[91136]: pgmap v3483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:02 smithi146 ceph-mon[103278]: pgmap v3483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:03.527 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:16:03.528 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:16:03.528 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:16:03.528 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:16:03.528 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:16:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:03 smithi146 ceph-mon[103278]: from='client.71846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:03 smithi146 ceph-mon[103278]: from='client.51933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:03 smithi146 ceph-mon[103278]: from='client.51939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/161806180' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2488421854' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[99959]: from='client.71846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[99959]: from='client.51933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[99959]: from='client.51939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/161806180' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2488421854' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[91136]: from='client.71846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[91136]: from='client.51933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[91136]: from='client.51939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/161806180' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2488421854' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:04.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:04.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:04.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-mon[99959]: pgmap v3484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-mon[99959]: from='client.51951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-mon[91136]: pgmap v3484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-mon[91136]: from='client.51951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:04 smithi146 ceph-mon[103278]: pgmap v3484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:04 smithi146 ceph-mon[103278]: from='client.51951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:05 smithi078 ceph-mon[99959]: pgmap v3485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:05 smithi078 ceph-mon[91136]: pgmap v3485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:05 smithi146 ceph-mon[103278]: pgmap v3485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:08 smithi146 ceph-mon[103278]: pgmap v3486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:08 smithi078 ceph-mon[99959]: pgmap v3486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:08 smithi078 ceph-mon[91136]: pgmap v3486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:10 smithi146 ceph-mon[103278]: pgmap v3487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:10 smithi078 ceph-mon[99959]: pgmap v3487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:10 smithi078 ceph-mon[91136]: pgmap v3487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:16:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:16:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:10.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:10.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:10.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:12 smithi146 ceph-mon[103278]: pgmap v3488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:12 smithi078 ceph-mon[91136]: pgmap v3488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:12 smithi078 ceph-mon[99959]: pgmap v3488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:14.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:14.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:14.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:14.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:14.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:14.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:14.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:14 smithi078 ceph-mon[91136]: pgmap v3489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:14.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:14 smithi078 ceph-mon[99959]: pgmap v3489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:14 smithi146 ceph-mon[103278]: pgmap v3489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:16 smithi146 ceph-mon[103278]: pgmap v3490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:16 smithi078 ceph-mon[91136]: pgmap v3490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:16 smithi078 ceph-mon[99959]: pgmap v3490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:18 smithi146 ceph-mon[103278]: pgmap v3491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:18 smithi078 ceph-mon[91136]: pgmap v3491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:18 smithi078 ceph-mon[99959]: pgmap v3491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:20 smithi146 ceph-mon[103278]: pgmap v3492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:21.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:20 smithi078 ceph-mon[91136]: pgmap v3492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:21.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:20 smithi078 ceph-mon[99959]: pgmap v3492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:21.026 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:16:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:16:21.026 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:20.602Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:21.026 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:20.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:21.027 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:20.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:23.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:22 smithi146 ceph-mon[103278]: pgmap v3493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:22 smithi078 ceph-mon[91136]: pgmap v3493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:22 smithi078 ceph-mon[99959]: pgmap v3493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:24.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:24.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:24.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:24 smithi146 ceph-mon[103278]: pgmap v3494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:24 smithi078 ceph-mon[91136]: pgmap v3494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:24 smithi078 ceph-mon[99959]: pgmap v3494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:26 smithi146 ceph-mon[103278]: pgmap v3495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:26 smithi078 ceph-mon[91136]: pgmap v3495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:26 smithi078 ceph-mon[99959]: pgmap v3495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:28 smithi146 ceph-mon[103278]: pgmap v3496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:28 smithi078 ceph-mon[91136]: pgmap v3496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:28 smithi078 ceph-mon[99959]: pgmap v3496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:30 smithi146 ceph-mon[103278]: pgmap v3497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:30 smithi078 ceph-mon[91136]: pgmap v3497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:30 smithi078 ceph-mon[99959]: pgmap v3497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:16:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:16:31.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:30.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:30.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:30.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:32 smithi146 ceph-mon[103278]: pgmap v3498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:32 smithi078 ceph-mon[99959]: pgmap v3498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:32 smithi078 ceph-mon[91136]: pgmap v3498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:33.835 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:16:34.150 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:34.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:34.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:34.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:34.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:34.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 2h - - 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 2h - - 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 3s ago 2h 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 19s ago 2h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (114m) 3s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (113m) 19s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (115m) 3s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:16:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (113m) 3s ago 2h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (112m) 19s ago 2h 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (112m) 3s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 3s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 19s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (111m) 3s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (110m) 3s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (109m) 3s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (109m) 3s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (108m) 19s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (107m) 19s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (106m) 19s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (106m) 19s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (114m) 19s ago 2h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:16:34.373 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (105m) 3s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:16:34.374 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (105m) 19s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:16:34.374 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (105m) 3s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:16:34.374 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (105m) 19s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:34.690 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:16:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:16:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:16:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:16:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:16:34.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:16:34.691 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:16:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:16:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:16:34.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:16:34.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-mon[91136]: pgmap v3499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:34.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-mon[91136]: from='client.71882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:34.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3216318222' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:34.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-mon[99959]: pgmap v3499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:34.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-mon[99959]: from='client.71882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:34.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3216318222' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:34 smithi146 ceph-mon[103278]: pgmap v3499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:34 smithi146 ceph-mon[103278]: from='client.71882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3216318222' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:16:35.324 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:16:35.324 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:16:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:16:35.324 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:16:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:16:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='client.51969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='client.71894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:36.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4109389708' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='client.51969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='client.71894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4109389708' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='client.51969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='client.71894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4109389708' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:37.105 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:36 smithi078 ceph-mon[99959]: from='client.51987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:37.106 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:36 smithi078 ceph-mon[99959]: pgmap v3500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:37.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:36 smithi078 ceph-mon[91136]: from='client.51987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:37.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:36 smithi078 ceph-mon[91136]: pgmap v3500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:36 smithi146 ceph-mon[103278]: from='client.51987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:16:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:36 smithi146 ceph-mon[103278]: pgmap v3500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:38 smithi146 ceph-mon[103278]: pgmap v3501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:38 smithi078 ceph-mon[91136]: pgmap v3501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:38 smithi078 ceph-mon[99959]: pgmap v3501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:40 smithi078 ceph-mon[91136]: pgmap v3502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:40 smithi078 ceph-mon[99959]: pgmap v3502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:16:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:16:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:40.604Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:40.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:40.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:40 smithi146 ceph-mon[103278]: pgmap v3502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:42 smithi146 ceph-mon[103278]: pgmap v3503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:42 smithi078 ceph-mon[91136]: pgmap v3503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:42 smithi078 ceph-mon[99959]: pgmap v3503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:44.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:44.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:44.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:44 smithi078 ceph-mon[91136]: pgmap v3504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:44 smithi078 ceph-mon[99959]: pgmap v3504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:44 smithi146 ceph-mon[103278]: pgmap v3504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:46.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:46 smithi146 ceph-mon[103278]: pgmap v3505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:46 smithi078 ceph-mon[91136]: pgmap v3505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:46 smithi078 ceph-mon[99959]: pgmap v3505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:48 smithi146 ceph-mon[103278]: pgmap v3506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:48 smithi078 ceph-mon[91136]: pgmap v3506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:48 smithi078 ceph-mon[99959]: pgmap v3506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:50.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:16:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:16:50.862 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:50.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:50.862 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:50.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:50.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:50.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:50 smithi146 ceph-mon[103278]: pgmap v3507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:50 smithi078 ceph-mon[91136]: pgmap v3507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:50 smithi078 ceph-mon[99959]: pgmap v3507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:51 smithi146 ceph-mon[103278]: pgmap v3508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[99959]: pgmap v3508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:51 smithi078 ceph-mon[91136]: pgmap v3508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:54 smithi078 ceph-mon[91136]: pgmap v3509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:54 smithi078 ceph-mon[99959]: pgmap v3509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:54.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:54.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:16:54.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:16:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:16:54.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:16:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:54 smithi146 ceph-mon[103278]: pgmap v3509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:16:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:16:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:56 smithi146 ceph-mon[103278]: pgmap v3510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[91136]: pgmap v3510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[99959]: pgmap v3510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:16:58.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:58 smithi146 ceph-mon[103278]: pgmap v3511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:58.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:16:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:58 smithi078 ceph-mon[91136]: pgmap v3511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:16:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:58 smithi078 ceph-mon[99959]: pgmap v3511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:16:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:00.546 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:00 smithi146 ceph-mon[103278]: pgmap v3512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:00 smithi078 ceph-mon[91136]: pgmap v3512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:00 smithi078 ceph-mon[99959]: pgmap v3512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:17:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:17:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:00.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:00.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:00.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:02 smithi146 ceph-mon[103278]: pgmap v3513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:02 smithi078 ceph-mon[99959]: pgmap v3513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:02 smithi078 ceph-mon[91136]: pgmap v3513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:04.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:04.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:04.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:04.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:04 smithi146 ceph-mon[103278]: pgmap v3514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:04 smithi078 ceph-mon[99959]: pgmap v3514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:04 smithi078 ceph-mon[91136]: pgmap v3514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:05.620 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:17:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 2h - - 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 2h - - 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 15s ago 2h 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 10s ago 2h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (115m) 15s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (114m) 10s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (115m) 15s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (114m) 15s ago 2h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (112m) 10s ago 2h 150M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:17:06.150 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (113m) 15s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 15s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 10s ago 2h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (111m) 15s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (110m) 15s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (110m) 15s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (109m) 15s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (109m) 10s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (108m) 10s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (107m) 10s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (106m) 10s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (115m) 10s ago 2h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (106m) 15s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (106m) 10s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (106m) 15s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:17:06.151 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (105m) 10s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:17:06.462 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:17:06.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:17:06.464 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:17:06.733 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:17:06.733 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:17:06.733 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:17:06.733 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:17:06.733 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:17:06.734 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:17:06.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[99959]: pgmap v3515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:06.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[99959]: from='client.51999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:06.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[99959]: from='client.52005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:06.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1009980154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:06.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[91136]: pgmap v3515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:06.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[91136]: from='client.51999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:06.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[91136]: from='client.52005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:06.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1009980154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:06 smithi146 ceph-mon[103278]: pgmap v3515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:06 smithi146 ceph-mon[103278]: from='client.51999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:06 smithi146 ceph-mon[103278]: from='client.52005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1009980154' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:07.040 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:17:07.041 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:07.041 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:07 smithi146 ceph-mon[103278]: from='client.71930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:07 smithi146 ceph-mon[103278]: from='client.71942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4212001556' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:07 smithi078 ceph-mon[91136]: from='client.71930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:07 smithi078 ceph-mon[91136]: from='client.71942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4212001556' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:07 smithi078 ceph-mon[99959]: from='client.71930 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:07 smithi078 ceph-mon[99959]: from='client.71942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4212001556' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:08 smithi146 ceph-mon[103278]: pgmap v3516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:08 smithi078 ceph-mon[99959]: pgmap v3516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:08 smithi078 ceph-mon[91136]: pgmap v3516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:10 smithi146 ceph-mon[103278]: pgmap v3517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:10 smithi078 ceph-mon[91136]: pgmap v3517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:10 smithi078 ceph-mon[99959]: pgmap v3517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:17:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:17:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:10.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:10.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:10.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:12 smithi146 ceph-mon[103278]: pgmap v3518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[91136]: pgmap v3518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:12 smithi078 ceph-mon[99959]: pgmap v3518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:14.299 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:14.034Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:14.300 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:14.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:14.300 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:14.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:14 smithi146 ceph-mon[103278]: pgmap v3519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:14 smithi078 ceph-mon[91136]: pgmap v3519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:14 smithi078 ceph-mon[99959]: pgmap v3519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:15.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:15.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:15.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:16 smithi146 ceph-mon[103278]: pgmap v3520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[99959]: pgmap v3520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[91136]: pgmap v3520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:16.857 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:18 smithi146 ceph-mon[103278]: pgmap v3521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:18 smithi078 ceph-mon[91136]: pgmap v3521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:18 smithi078 ceph-mon[99959]: pgmap v3521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:20.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:20 smithi146 ceph-mon[103278]: pgmap v3522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:20 smithi078 ceph-mon[91136]: pgmap v3522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:20 smithi078 ceph-mon[99959]: pgmap v3522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:17:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:17:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:20.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:20.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:20.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:22 smithi146 ceph-mon[103278]: pgmap v3523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:22 smithi078 ceph-mon[91136]: pgmap v3523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:22 smithi078 ceph-mon[99959]: pgmap v3523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:24.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:24.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:24.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:24 smithi146 ceph-mon[103278]: pgmap v3524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:24 smithi078 ceph-mon[91136]: pgmap v3524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:24 smithi078 ceph-mon[99959]: pgmap v3524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:26 smithi146 ceph-mon[103278]: pgmap v3525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:26 smithi078 ceph-mon[91136]: pgmap v3525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:26 smithi078 ceph-mon[99959]: pgmap v3525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:28 smithi078 ceph-mon[91136]: pgmap v3526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:28 smithi078 ceph-mon[99959]: pgmap v3526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:28 smithi146 ceph-mon[103278]: pgmap v3526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:30 smithi146 ceph-mon[103278]: pgmap v3527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:30 smithi078 ceph-mon[91136]: pgmap v3527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:17:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:17:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:30 smithi078 ceph-mon[99959]: pgmap v3527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:30.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:30.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:30.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:32 smithi146 ceph-mon[103278]: pgmap v3528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[91136]: pgmap v3528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[99959]: pgmap v3528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:34 smithi146 ceph-mon[103278]: pgmap v3529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:34 smithi078 ceph-mon[91136]: pgmap v3529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:34 smithi078 ceph-mon[99959]: pgmap v3529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:34.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:34.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:34.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:36.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[99959]: pgmap v3530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:36.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:36.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:36.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:36.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[91136]: pgmap v3530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:36.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:36.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:36.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:36 smithi146 ceph-mon[103278]: pgmap v3530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:37.347 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:17:37.903 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 2h - - 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 2h - - 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 6s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 2s ago 2h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (115m) 6s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (114m) 2s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (116m) 6s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (114m) 6s ago 2h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (113m) 2s ago 2h 152M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (113m) 6s ago 2h 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 6s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 2s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (112m) 6s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:17:37.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (111m) 6s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (110m) 6s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (110m) 6s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (109m) 2s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (108m) 2s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (107m) 2s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (107m) 2s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (115m) 2s ago 2h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (106m) 6s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (106m) 2s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (106m) 6s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:17:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (106m) 2s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:17:38.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:17:38.222 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:17:38.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:17:38.493 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:17:38.493 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:17:38.493 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:17:38.494 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:17:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[99959]: pgmap v3531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[99959]: from='client.52035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[99959]: from='client.71960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[99959]: from='client.71966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1865173070' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[91136]: pgmap v3531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[91136]: from='client.52035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[91136]: from='client.71960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[91136]: from='client.71966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1865173070' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:38.807 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:17:38.807 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:17:38.808 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:38 smithi146 ceph-mon[103278]: pgmap v3531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:38 smithi146 ceph-mon[103278]: from='client.52035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:38 smithi146 ceph-mon[103278]: from='client.71960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:38 smithi146 ceph-mon[103278]: from='client.71966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1865173070' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:17:39.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:39 smithi146 ceph-mon[103278]: from='client.71978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:39.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2972335499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:39 smithi078 ceph-mon[91136]: from='client.71978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2972335499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:39 smithi078 ceph-mon[99959]: from='client.71978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:17:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2972335499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:17:40.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:40 smithi146 ceph-mon[103278]: pgmap v3532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:40 smithi078 ceph-mon[91136]: pgmap v3532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:40 smithi078 ceph-mon[99959]: pgmap v3532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:17:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:17:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:40.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:40.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:40.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:42 smithi146 ceph-mon[103278]: pgmap v3533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:42 smithi078 ceph-mon[99959]: pgmap v3533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:42 smithi078 ceph-mon[91136]: pgmap v3533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:44.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:44.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:44.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:44.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:44.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:44.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:44 smithi146 ceph-mon[103278]: pgmap v3534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:44 smithi078 ceph-mon[91136]: pgmap v3534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:44 smithi078 ceph-mon[99959]: pgmap v3534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:46 smithi146 ceph-mon[103278]: pgmap v3535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:46 smithi078 ceph-mon[99959]: pgmap v3535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:46 smithi078 ceph-mon[91136]: pgmap v3535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:48 smithi146 ceph-mon[103278]: pgmap v3536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:48 smithi078 ceph-mon[99959]: pgmap v3536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:48 smithi078 ceph-mon[91136]: pgmap v3536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:50 smithi146 ceph-mon[103278]: pgmap v3537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:50 smithi078 ceph-mon[99959]: pgmap v3537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:50 smithi078 ceph-mon[91136]: pgmap v3537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:17:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:17:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:50.610Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:50.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:50.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:52 smithi146 ceph-mon[103278]: pgmap v3538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[99959]: pgmap v3538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[91136]: pgmap v3538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:54.326 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:54.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:54.327 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:54.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:17:54.327 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:17:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:17:54.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:17:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:54 smithi146 ceph-mon[103278]: pgmap v3539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:54 smithi078 ceph-mon[91136]: pgmap v3539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:54 smithi078 ceph-mon[99959]: pgmap v3539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:17:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[99959]: pgmap v3540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[91136]: pgmap v3540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:56 smithi146 ceph-mon[103278]: pgmap v3540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:17:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:58 smithi078 ceph-mon[91136]: pgmap v3541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:58 smithi078 ceph-mon[99959]: pgmap v3541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:17:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:58 smithi146 ceph-mon[103278]: pgmap v3541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:17:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:00 smithi078 ceph-mon[91136]: pgmap v3542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:00 smithi078 ceph-mon[99959]: pgmap v3542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:18:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:18:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:00.610Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:00.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:00.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:01.120 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:00 smithi146 ceph-mon[103278]: pgmap v3542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:02 smithi146 ceph-mon[103278]: pgmap v3543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:02 smithi078 ceph-mon[91136]: pgmap v3543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:02 smithi078 ceph-mon[99959]: pgmap v3543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:04.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:04.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:04.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:04 smithi146 ceph-mon[103278]: pgmap v3544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:04 smithi078 ceph-mon[91136]: pgmap v3544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:04 smithi078 ceph-mon[99959]: pgmap v3544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:06 smithi146 ceph-mon[103278]: pgmap v3545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:06 smithi078 ceph-mon[91136]: pgmap v3545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:06 smithi078 ceph-mon[99959]: pgmap v3545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.110 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:18:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:08 smithi146 ceph-mon[103278]: pgmap v3546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:08 smithi078 ceph-mon[99959]: pgmap v3546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:08 smithi078 ceph-mon[91136]: pgmap v3546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 2h - - 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 2h - - 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 18s ago 2h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 13s ago 2h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (116m) 18s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (115m) 13s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (117m) 18s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (115m) 18s ago 2h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (113m) 13s ago 2h 152M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:18:09.641 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (114m) 18s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 18s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 13s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (113m) 18s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (112m) 18s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (111m) 18s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (110m) 18s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (110m) 13s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (109m) 13s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (108m) 13s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (107m) 13s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (116m) 13s ago 2h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (107m) 18s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (107m) 13s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (107m) 18s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:18:09.642 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (107m) 13s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:18:09.947 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:09.948 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:18:09.949 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:18:09.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:18:09.949 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:18:10.217 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:18:10.217 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:18:10.217 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:18:10.217 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:18:10.217 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:18:10.218 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:18:10.530 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:18:10.530 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:10.530 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:10.530 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:10.531 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:18:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:18:10.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:10.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:10.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:10.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:10.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:10.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:10 smithi146 ceph-mon[103278]: from='client.52071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:10 smithi146 ceph-mon[103278]: pgmap v3547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:10 smithi146 ceph-mon[103278]: from='client.71996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:10 smithi146 ceph-mon[103278]: from='client.52083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2004206146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1934118161' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[91136]: from='client.52071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[91136]: pgmap v3547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[91136]: from='client.71996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[91136]: from='client.52083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2004206146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1934118161' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[99959]: from='client.52071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[99959]: pgmap v3547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[99959]: from='client.71996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[99959]: from='client.52083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2004206146' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1934118161' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:11 smithi146 ceph-mon[103278]: from='client.72014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[91136]: from='client.72014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[99959]: from='client.72014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:12 smithi078 ceph-mon[99959]: pgmap v3548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:12 smithi078 ceph-mon[91136]: pgmap v3548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:12 smithi146 ceph-mon[103278]: pgmap v3548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:14.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:14.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:14.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:14 smithi146 ceph-mon[103278]: pgmap v3549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:14 smithi078 ceph-mon[91136]: pgmap v3549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:14 smithi078 ceph-mon[99959]: pgmap v3549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:16.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:17.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:16 smithi146 ceph-mon[103278]: pgmap v3550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:17.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[91136]: pgmap v3550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:17.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[99959]: pgmap v3550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:17.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:17.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:18 smithi146 ceph-mon[103278]: pgmap v3551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:18 smithi078 ceph-mon[91136]: pgmap v3551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:18 smithi078 ceph-mon[99959]: pgmap v3551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:20.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:18:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:18:20.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:20.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:20.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:20.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:20.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:20.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:21.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:20 smithi146 ceph-mon[103278]: pgmap v3552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:20 smithi078 ceph-mon[91136]: pgmap v3552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:20 smithi078 ceph-mon[99959]: pgmap v3552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:22 smithi146 ceph-mon[103278]: pgmap v3553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:22 smithi078 ceph-mon[91136]: pgmap v3553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:22 smithi078 ceph-mon[99959]: pgmap v3553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:24.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:24.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:24.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:24 smithi146 ceph-mon[103278]: pgmap v3554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:24 smithi078 ceph-mon[91136]: pgmap v3554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:24 smithi078 ceph-mon[99959]: pgmap v3554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:26 smithi146 ceph-mon[103278]: pgmap v3555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:26 smithi078 ceph-mon[91136]: pgmap v3555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:26 smithi078 ceph-mon[99959]: pgmap v3555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:28 smithi146 ceph-mon[103278]: pgmap v3556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:28 smithi078 ceph-mon[91136]: pgmap v3556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:28 smithi078 ceph-mon[99959]: pgmap v3556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:30.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:18:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:18:30.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:30.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:30.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:30.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:30.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:30.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:30 smithi146 ceph-mon[103278]: pgmap v3557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:30 smithi078 ceph-mon[91136]: pgmap v3557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:30 smithi078 ceph-mon[99959]: pgmap v3557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:32 smithi146 ceph-mon[103278]: pgmap v3558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[91136]: pgmap v3558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[99959]: pgmap v3558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:34.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:34.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:34.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:34.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:34 smithi078 ceph-mon[99959]: pgmap v3559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:34.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:34 smithi078 ceph-mon[91136]: pgmap v3559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:34 smithi146 ceph-mon[103278]: pgmap v3559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:36 smithi146 ceph-mon[103278]: pgmap v3560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[91136]: pgmap v3560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[99959]: pgmap v3560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:38 smithi078 ceph-mon[91136]: pgmap v3561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:38 smithi078 ceph-mon[99959]: pgmap v3561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:38 smithi146 ceph-mon[103278]: pgmap v3561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:40.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:40 smithi078 ceph-mon[91136]: pgmap v3562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:40.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:40 smithi078 ceph-mon[99959]: pgmap v3562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:40.828 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:40.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:40.828 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:40.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:40.828 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:40.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:40.829 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:18:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:18:40.829 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:18:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:40 smithi146 ceph-mon[103278]: pgmap v3562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 2h - - 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 2h - - 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 9s ago 2h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 5s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (116m) 9s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (115m) 5s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:18:41.376 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (117m) 9s ago 2h 566M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (115m) 9s ago 2h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (114m) 5s ago 2h 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (114m) 9s ago 2h 160M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 9s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 5s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (113m) 9s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (112m) 9s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (111m) 9s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (111m) 9s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (110m) 5s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (109m) 5s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (109m) 5s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:18:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (108m) 5s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:18:41.378 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (117m) 5s ago 2h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:18:41.378 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (107m) 9s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:18:41.378 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (107m) 5s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:18:41.378 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (107m) 9s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:18:41.378 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (107m) 5s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:18:41.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:41 smithi146 ceph-mon[103278]: from='client.72026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:18:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:18:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:18:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:18:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:18:41.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:18:41.982 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:18:41.983 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:18:41.984 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:18:41.984 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:18:41.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:41 smithi078 ceph-mon[99959]: from='client.72026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:41.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:41 smithi078 ceph-mon[91136]: from='client.72026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:42.299 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:18:42.299 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:18:42.299 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:42.299 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:18:42.749 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:42 smithi146 ceph-mon[103278]: from='client.52113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:42.749 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:42 smithi146 ceph-mon[103278]: pgmap v3563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:42.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:42 smithi146 ceph-mon[103278]: from='client.72038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:42.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/41688576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:42.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/285601718' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[91136]: from='client.52113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[91136]: pgmap v3563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[91136]: from='client.72038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/41688576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/285601718' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[99959]: from='client.52113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[99959]: pgmap v3563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[99959]: from='client.72038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/41688576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:18:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/285601718' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:18:43.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:43 smithi146 ceph-mon[103278]: from='client.52131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:43.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:43 smithi078 ceph-mon[91136]: from='client.52131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:43 smithi078 ceph-mon[99959]: from='client.52131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:18:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:44.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:44.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:44.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:44.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:44.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:44.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:44.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:44 smithi078 ceph-mon[91136]: pgmap v3564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:44.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:44 smithi078 ceph-mon[99959]: pgmap v3564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:44 smithi146 ceph-mon[103278]: pgmap v3564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:46.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:46 smithi146 ceph-mon[103278]: pgmap v3565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:46 smithi078 ceph-mon[91136]: pgmap v3565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:46 smithi078 ceph-mon[99959]: pgmap v3565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:48 smithi146 ceph-mon[103278]: pgmap v3566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:48 smithi078 ceph-mon[91136]: pgmap v3566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:48 smithi078 ceph-mon[99959]: pgmap v3566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:50 smithi146 ceph-mon[103278]: pgmap v3567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:50 smithi078 ceph-mon[91136]: pgmap v3567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:50 smithi078 ceph-mon[99959]: pgmap v3567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:18:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:18:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:50.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:50.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:50.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:52 smithi146 ceph-mon[103278]: pgmap v3568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[91136]: pgmap v3568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[99959]: pgmap v3568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:54.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:54.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:18:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:18:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:18:54.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:18:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:54 smithi146 ceph-mon[103278]: pgmap v3569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:54 smithi078 ceph-mon[91136]: pgmap v3569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:54 smithi078 ceph-mon[99959]: pgmap v3569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:18:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:18:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:56 smithi146 ceph-mon[103278]: pgmap v3570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[91136]: pgmap v3570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[99959]: pgmap v3570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:18:58.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:58 smithi078 ceph-mon[91136]: pgmap v3571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:58.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:58.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:58 smithi078 ceph-mon[99959]: pgmap v3571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:58.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:18:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:58 smithi146 ceph-mon[103278]: pgmap v3571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:18:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:00 smithi078 ceph-mon[91136]: pgmap v3572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:19:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:19:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:00 smithi078 ceph-mon[99959]: pgmap v3572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:00.616Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:00.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:00.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:00 smithi146 ceph-mon[103278]: pgmap v3572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:02 smithi146 ceph-mon[103278]: pgmap v3573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:02 smithi078 ceph-mon[91136]: pgmap v3573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:02 smithi078 ceph-mon[99959]: pgmap v3573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:04.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:04.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:04.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:04 smithi146 ceph-mon[103278]: pgmap v3574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:04 smithi078 ceph-mon[91136]: pgmap v3574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:04 smithi078 ceph-mon[99959]: pgmap v3574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:06.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:06 smithi146 ceph-mon[103278]: pgmap v3575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:06 smithi078 ceph-mon[99959]: pgmap v3575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:06 smithi078 ceph-mon[91136]: pgmap v3575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:08 smithi146 ceph-mon[103278]: pgmap v3576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:08 smithi078 ceph-mon[91136]: pgmap v3576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:08 smithi078 ceph-mon[99959]: pgmap v3576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:10 smithi146 ceph-mon[103278]: pgmap v3577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:10 smithi078 ceph-mon[91136]: pgmap v3577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:19:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:19:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:10 smithi078 ceph-mon[99959]: pgmap v3577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:10.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:10.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:10.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:12.585 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 2h - - 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 2h - - 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 1s ago 2h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 16s ago 2h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (117m) 1s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (116m) 16s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (118m) 1s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (116m) 1s ago 2h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (115m) 16s ago 2h 154M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (115m) 1s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 1s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:19:13.126 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 16s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (114m) 1s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (113m) 1s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (112m) 1s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (111m) 1s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (111m) 16s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (110m) 16s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (109m) 16s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (108m) 16s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (117m) 16s ago 2h 75.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (108m) 1s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (108m) 16s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (108m) 1s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:19:13.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (108m) 16s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[99959]: pgmap v3578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[91136]: pgmap v3578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:12 smithi146 ceph-mon[103278]: pgmap v3578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:19:13.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:19:13.462 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:19:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:19:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:19:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:19:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:19:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:19:13.762 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[99959]: from='client.52143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[99959]: from='client.52149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2484312213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[91136]: from='client.52143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[91136]: from='client.52149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2484312213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:14.084 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:19:14.084 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:14.085 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:14.085 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:13 smithi146 ceph-mon[103278]: from='client.52143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:13 smithi146 ceph-mon[103278]: from='client.52149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2484312213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:14.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:14.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:14.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:14 smithi146 ceph-mon[103278]: from='client.52155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:14 smithi146 ceph-mon[103278]: pgmap v3579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:14 smithi146 ceph-mon[103278]: from='client.52167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3601975938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[91136]: from='client.52155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[91136]: pgmap v3579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[91136]: from='client.52167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3601975938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[99959]: from='client.52155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[99959]: pgmap v3579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[99959]: from='client.52167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3601975938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:16 smithi146 ceph-mon[103278]: pgmap v3580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[91136]: pgmap v3580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[99959]: pgmap v3580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:18 smithi146 ceph-mon[103278]: pgmap v3581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:19.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:18 smithi078 ceph-mon[91136]: pgmap v3581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:19.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:18 smithi078 ceph-mon[99959]: pgmap v3581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:20.666 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:19:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:19:20.667 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:20.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:20.667 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:20.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:20.667 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:20.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:20.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:20 smithi078 ceph-mon[91136]: pgmap v3582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:20.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:20 smithi078 ceph-mon[99959]: pgmap v3582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:20 smithi146 ceph-mon[103278]: pgmap v3582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:23.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:22 smithi146 ceph-mon[103278]: pgmap v3583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:22 smithi078 ceph-mon[99959]: pgmap v3583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:22 smithi078 ceph-mon[91136]: pgmap v3583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:24.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:24.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:24.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:24 smithi146 ceph-mon[103278]: pgmap v3584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:24 smithi078 ceph-mon[91136]: pgmap v3584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:24 smithi078 ceph-mon[99959]: pgmap v3584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:26 smithi146 ceph-mon[103278]: pgmap v3585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:26 smithi078 ceph-mon[91136]: pgmap v3585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:26 smithi078 ceph-mon[99959]: pgmap v3585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:28 smithi146 ceph-mon[103278]: pgmap v3586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:28 smithi078 ceph-mon[91136]: pgmap v3586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:28 smithi078 ceph-mon[99959]: pgmap v3586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:30.895 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:19:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:19:30.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:30.618Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:30.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:30.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:30.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:30.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:31.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:30 smithi078 ceph-mon[99959]: pgmap v3587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:31.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:30 smithi078 ceph-mon[91136]: pgmap v3587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:30 smithi146 ceph-mon[103278]: pgmap v3587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:32 smithi146 ceph-mon[103278]: pgmap v3588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[91136]: pgmap v3588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[99959]: pgmap v3588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:34.044Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:34.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:34.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:34 smithi146 ceph-mon[103278]: pgmap v3589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:34 smithi078 ceph-mon[99959]: pgmap v3589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:34 smithi078 ceph-mon[91136]: pgmap v3589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:36 smithi146 ceph-mon[103278]: pgmap v3590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[99959]: pgmap v3590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[91136]: pgmap v3590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:38 smithi146 ceph-mon[103278]: pgmap v3591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:38 smithi078 ceph-mon[99959]: pgmap v3591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:38 smithi078 ceph-mon[91136]: pgmap v3591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:40.916 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:19:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:19:40.917 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:40.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:40.917 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:40.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:40.917 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:40.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:41.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:40 smithi078 ceph-mon[91136]: pgmap v3592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:41.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:40 smithi078 ceph-mon[99959]: pgmap v3592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:40 smithi146 ceph-mon[103278]: pgmap v3592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:43.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:42 smithi146 ceph-mon[103278]: pgmap v3593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:43.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:42 smithi078 ceph-mon[99959]: pgmap v3593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:43.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:42 smithi078 ceph-mon[91136]: pgmap v3593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:44.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:44.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:44.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:44.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:44.396 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 2h - - 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 12s ago 2h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (117m) 12s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:19:44.922 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (116m) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (118m) 12s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (116m) 12s ago 2h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (115m) 8s ago 2h 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (115m) 12s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 12s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (114m) 12s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (113m) 12s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (112m) 12s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (112m) 12s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (111m) 8s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (110m) 8s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (110m) 8s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:19:44.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (109m) 8s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:19:44.924 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (118m) 8s ago 2h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:19:44.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (108m) 12s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:19:44.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (108m) 8s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:19:44.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (108m) 12s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:19:44.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (108m) 8s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:19:45.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:44 smithi078 ceph-mon[99959]: pgmap v3594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:45.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:44 smithi078 ceph-mon[91136]: pgmap v3594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:45.232 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:19:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:19:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:19:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:19:45.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:19:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:44 smithi146 ceph-mon[103278]: pgmap v3594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:45.501 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:19:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:19:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:19:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:19:45.502 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:19:45.809 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:19:45.809 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:19:45.809 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:19:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:45 smithi146 ceph-mon[103278]: from='client.72098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:45 smithi146 ceph-mon[103278]: from='client.52185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:45 smithi146 ceph-mon[103278]: from='client.52191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3447727372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1197568238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[91136]: from='client.72098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[91136]: from='client.52185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[91136]: from='client.52191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3447727372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1197568238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[99959]: from='client.72098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[99959]: from='client.52185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[99959]: from='client.52191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3447727372' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:19:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1197568238' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:19:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:46 smithi146 ceph-mon[103278]: pgmap v3595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:46 smithi146 ceph-mon[103278]: from='client.72122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:46 smithi078 ceph-mon[91136]: pgmap v3595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:46 smithi078 ceph-mon[91136]: from='client.72122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:46 smithi078 ceph-mon[99959]: pgmap v3595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:46 smithi078 ceph-mon[99959]: from='client.72122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:19:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:48 smithi146 ceph-mon[103278]: pgmap v3596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:48 smithi078 ceph-mon[91136]: pgmap v3596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:48 smithi078 ceph-mon[99959]: pgmap v3596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:50.941 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:19:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:19:50.941 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:50.620Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:50.941 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:50.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:50.941 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:50.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:50 smithi146 ceph-mon[103278]: pgmap v3597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:50 smithi078 ceph-mon[91136]: pgmap v3597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:50 smithi078 ceph-mon[99959]: pgmap v3597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:52 smithi146 ceph-mon[103278]: pgmap v3598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[91136]: pgmap v3598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[99959]: pgmap v3598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:54.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:54.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:19:54.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:19:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:19:54.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:19:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:54 smithi146 ceph-mon[103278]: pgmap v3599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:54 smithi078 ceph-mon[99959]: pgmap v3599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:54 smithi078 ceph-mon[91136]: pgmap v3599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:19:57.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:57.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:56 smithi146 ceph-mon[103278]: pgmap v3600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:57.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[99959]: pgmap v3600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:19:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[91136]: pgmap v3600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:58 smithi146 ceph-mon[103278]: pgmap v3601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:58 smithi078 ceph-mon[91136]: pgmap v3601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:19:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:58 smithi078 ceph-mon[99959]: pgmap v3601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:19:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:00 smithi146 ceph-mon[103278]: pgmap v3602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-mon[99959]: pgmap v3602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T22:19:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-mon[91136]: pgmap v3602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:00.879 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:20:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:20:00.879 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:00.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:00.879 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:00.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:00.879 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:00.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:02.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:02 smithi078 ceph-mon[91136]: pgmap v3603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:02.559 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:02 smithi078 ceph-mon[99959]: pgmap v3603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:02 smithi146 ceph-mon[103278]: pgmap v3603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:04.300 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:04.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:04.301 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:04.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:04.301 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:04.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:04 smithi146 ceph-mon[103278]: pgmap v3604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:04 smithi078 ceph-mon[91136]: pgmap v3604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:04 smithi078 ceph-mon[99959]: pgmap v3604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:06 smithi146 ceph-mon[103278]: pgmap v3605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:06 smithi078 ceph-mon[91136]: pgmap v3605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:06 smithi078 ceph-mon[99959]: pgmap v3605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:08 smithi146 ceph-mon[103278]: pgmap v3606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:08 smithi078 ceph-mon[91136]: pgmap v3606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:08 smithi078 ceph-mon[99959]: pgmap v3606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:10 smithi078 ceph-mon[91136]: pgmap v3607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:10 smithi078 ceph-mon[99959]: pgmap v3607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:10 smithi146 ceph-mon[103278]: pgmap v3607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:20:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:20:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:10.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:10.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:10.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[99959]: pgmap v3608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[91136]: pgmap v3608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:12 smithi146 ceph-mon[103278]: pgmap v3608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:13.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:13.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:14.323 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:14.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:14.323 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:14.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:14.323 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:14.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:14 smithi146 ceph-mon[103278]: pgmap v3609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:14 smithi078 ceph-mon[99959]: pgmap v3609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:14 smithi078 ceph-mon[91136]: pgmap v3609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:16.103 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:20:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:16 smithi078 ceph-mon[99959]: pgmap v3610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:16 smithi078 ceph-mon[91136]: pgmap v3610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:16.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:16.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:16 smithi146 ceph-mon[103278]: pgmap v3610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 2h - - 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 2h - - 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 4s ago 2h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 19s ago 2h 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (118m) 4s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (117m) 19s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (119m) 4s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (117m) 4s ago 2h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (116m) 19s ago 2h 153M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:20:16.626 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (116m) 4s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 4s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 19s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (115m) 4s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (114m) 4s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (113m) 4s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (113m) 4s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (112m) 19s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (111m) 19s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (110m) 19s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (109m) 19s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (118m) 19s ago 2h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (109m) 4s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:20:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (109m) 19s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:20:16.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (109m) 4s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:20:16.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (109m) 19s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:20:16.943 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:20:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:20:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:20:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:20:16.944 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:20:17.214 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:20:17.215 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:20:17.215 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:20:17.215 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:20:17.525 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:17.525 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:17.525 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='client.72134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='client.72140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='client.72146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3782950803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/326207373' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='client.72134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='client.72140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='client.72146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3782950803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/326207373' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='client.72134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='client.72140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='client.72146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3782950803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/326207373' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:18 smithi078 ceph-mon[91136]: from='client.52239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:18 smithi078 ceph-mon[91136]: pgmap v3611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:18 smithi078 ceph-mon[99959]: from='client.52239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:18 smithi078 ceph-mon[99959]: pgmap v3611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:18 smithi146 ceph-mon[103278]: from='client.52239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:18 smithi146 ceph-mon[103278]: pgmap v3611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:20 smithi078 ceph-mon[91136]: pgmap v3612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:20 smithi078 ceph-mon[99959]: pgmap v3612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:20:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:20:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:20.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:20.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:20.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:20 smithi146 ceph-mon[103278]: pgmap v3612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:22.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:22 smithi078 ceph-mon[91136]: pgmap v3613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:22.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:22 smithi078 ceph-mon[99959]: pgmap v3613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:22 smithi146 ceph-mon[103278]: pgmap v3613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:24.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:24.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:24.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:24 smithi078 ceph-mon[91136]: pgmap v3614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:24 smithi078 ceph-mon[99959]: pgmap v3614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:25.215 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:24 smithi146 ceph-mon[103278]: pgmap v3614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:26 smithi078 ceph-mon[91136]: pgmap v3615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:26 smithi078 ceph-mon[99959]: pgmap v3615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:26 smithi146 ceph-mon[103278]: pgmap v3615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:28 smithi078 ceph-mon[91136]: pgmap v3616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:28 smithi078 ceph-mon[99959]: pgmap v3616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:28 smithi146 ceph-mon[103278]: pgmap v3616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:30 smithi078 ceph-mon[91136]: pgmap v3617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:30 smithi078 ceph-mon[99959]: pgmap v3617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:20:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:20:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:30.623Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:30.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:30.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:30 smithi146 ceph-mon[103278]: pgmap v3617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:32 smithi146 ceph-mon[103278]: pgmap v3618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[99959]: pgmap v3618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[91136]: pgmap v3618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:34.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:34.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:34.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:34 smithi146 ceph-mon[103278]: pgmap v3619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:34 smithi078 ceph-mon[99959]: pgmap v3619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:34 smithi078 ceph-mon[91136]: pgmap v3619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:36 smithi146 ceph-mon[103278]: pgmap v3620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:36 smithi078 ceph-mon[91136]: pgmap v3620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:36 smithi078 ceph-mon[99959]: pgmap v3620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:38 smithi146 ceph-mon[103278]: pgmap v3621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:38 smithi078 ceph-mon[91136]: pgmap v3621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:38 smithi078 ceph-mon[99959]: pgmap v3621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:40.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:20:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:20:40.905 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:40.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:40.905 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:40.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:40.905 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:40.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:40 smithi146 ceph-mon[103278]: pgmap v3622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:40 smithi078 ceph-mon[91136]: pgmap v3622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:40 smithi078 ceph-mon[99959]: pgmap v3622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:42 smithi078 ceph-mon[91136]: pgmap v3623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:42 smithi078 ceph-mon[99959]: pgmap v3623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:42 smithi146 ceph-mon[103278]: pgmap v3623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:44.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:44.052 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:44.385 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:44.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:44.385 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:44.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:44.385 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:44.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:45.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:44 smithi146 ceph-mon[103278]: pgmap v3624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:44 smithi078 ceph-mon[91136]: pgmap v3624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:44 smithi078 ceph-mon[99959]: pgmap v3624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:46 smithi146 ceph-mon[103278]: pgmap v3625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:46 smithi078 ceph-mon[99959]: pgmap v3625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:46 smithi078 ceph-mon[91136]: pgmap v3625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:47.823 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:20:48.352 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 2h - - 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 2h - - 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 15s ago 2h 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 11s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (118m) 15s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (117m) 11s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (119m) 15s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (117m) 15s ago 2h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (116m) 11s ago 2h 155M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (116m) 15s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 15s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 11s ago 2h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (115m) 15s ago 2h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:20:48.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (114m) 15s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (114m) 15s ago 2h 66.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (113m) 15s ago 2h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (113m) 11s ago 2h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (111m) 11s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (111m) 11s ago 2h 70.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (110m) 11s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (119m) 11s ago 2h 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (109m) 15s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (109m) 11s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (109m) 15s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:20:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (109m) 11s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:20:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:20:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:20:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:20:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:20:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:20:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:20:48.665 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:20:48.933 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:20:48.934 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:20:49.242 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:20:49.243 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:20:49.243 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:20:49.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:48 smithi078 ceph-mon[99959]: pgmap v3626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:49.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:48 smithi078 ceph-mon[99959]: from='client.52251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:49.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3201168042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:49.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:48 smithi078 ceph-mon[91136]: pgmap v3626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:49.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:48 smithi078 ceph-mon[91136]: from='client.52251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:49.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3201168042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:48 smithi146 ceph-mon[103278]: pgmap v3626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:48 smithi146 ceph-mon[103278]: from='client.52251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3201168042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:20:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:49 smithi146 ceph-mon[103278]: from='client.52257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:49 smithi146 ceph-mon[103278]: from='client.52263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:49 smithi146 ceph-mon[103278]: from='client.52275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/616538856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[99959]: from='client.52257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[99959]: from='client.52263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[99959]: from='client.52275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/616538856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[91136]: from='client.52257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[91136]: from='client.52263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[91136]: from='client.52275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:20:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/616538856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:20:50.933 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:20:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:20:50.933 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:50.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:50.933 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:50.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:50.933 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:50.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:50 smithi146 ceph-mon[103278]: pgmap v3627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:50 smithi078 ceph-mon[91136]: pgmap v3627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:50 smithi078 ceph-mon[99959]: pgmap v3627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:52 smithi146 ceph-mon[103278]: pgmap v3628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[91136]: pgmap v3628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[99959]: pgmap v3628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:20:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:54.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:54.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:20:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:20:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:20:54.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:20:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:54 smithi146 ceph-mon[103278]: pgmap v3629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:20:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:54 smithi078 ceph-mon[91136]: pgmap v3629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:20:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:54 smithi078 ceph-mon[99959]: pgmap v3629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:20:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:56 smithi146 ceph-mon[103278]: pgmap v3630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:20:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:56 smithi078 ceph-mon[91136]: pgmap v3630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:20:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:20:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:56 smithi078 ceph-mon[99959]: pgmap v3630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:20:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[91136]: pgmap v3631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:58 smithi078 ceph-mon[99959]: pgmap v3631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:20:58.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.310 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.310 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:20:58.310 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:58 smithi146 ceph-mon[103278]: pgmap v3631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:20:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:20:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:00 smithi146 ceph-mon[103278]: pgmap v3632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:21:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:00 smithi078 ceph-mon[91136]: pgmap v3632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:21:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:00 smithi078 ceph-mon[99959]: pgmap v3632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:21:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:21:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:21:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:00.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:00.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:00.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:02.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:02 smithi078 ceph-mon[99959]: pgmap v3633: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-20T22:21:02.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:02 smithi078 ceph-mon[91136]: pgmap v3633: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-20T22:21:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:02 smithi146 ceph-mon[103278]: pgmap v3633: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-20T22:21:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:04.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:04.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:04.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:04.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:04 smithi078 ceph-mon[99959]: pgmap v3634: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-20T22:21:04.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:04 smithi078 ceph-mon[91136]: pgmap v3634: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-20T22:21:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:04 smithi146 ceph-mon[103278]: pgmap v3634: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-20T22:21:06.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:06.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:06 smithi146 ceph-mon[103278]: pgmap v3635: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:21:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:06 smithi078 ceph-mon[91136]: pgmap v3635: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:21:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:06 smithi078 ceph-mon[99959]: pgmap v3635: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:21:08.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:08 smithi146 ceph-mon[103278]: pgmap v3636: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:21:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:08 smithi078 ceph-mon[91136]: pgmap v3636: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:21:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:08 smithi078 ceph-mon[99959]: pgmap v3636: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-20T22:21:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:10 smithi078 ceph-mon[99959]: pgmap v3637: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T22:21:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:10 smithi078 ceph-mon[91136]: pgmap v3637: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T22:21:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:10 smithi146 ceph-mon[103278]: pgmap v3637: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-20T22:21:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:21:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:21:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:10.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:10.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:10.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:12.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:12 smithi078 ceph-mon[99959]: pgmap v3638: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:21:12.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:12 smithi078 ceph-mon[91136]: pgmap v3638: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:21:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:12 smithi146 ceph-mon[103278]: pgmap v3638: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T22:21:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:14.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:14.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:14.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:14.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:14.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:14 smithi146 ceph-mon[103278]: pgmap v3639: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T22:21:15.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:14 smithi078 ceph-mon[99959]: pgmap v3639: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T22:21:15.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:14 smithi078 ceph-mon[91136]: pgmap v3639: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 123 op/s 2024-09-20T22:21:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:16 smithi146 ceph-mon[103278]: pgmap v3640: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T22:21:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:16 smithi078 ceph-mon[99959]: pgmap v3640: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T22:21:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:16 smithi078 ceph-mon[91136]: pgmap v3640: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-20T22:21:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:18 smithi146 ceph-mon[103278]: pgmap v3641: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:21:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[99959]: pgmap v3641: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:21:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:18 smithi078 ceph-mon[91136]: pgmap v3641: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T22:21:19.539 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:21:20.061 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:21:20.061 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 2h - - 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 2h - - 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 2s ago 2h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (119m) 7s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (118m) 2s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (118m) 7s ago 2h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (117m) 2s ago 2h 156M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (117m) 7s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 2s ago 2h 15.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (116m) 7s ago 2h 72.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:21:20.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (115m) 7s ago 2h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (114m) 7s ago 2h 66.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (114m) 7s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (113m) 2s ago 2h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (112m) 2s ago 2h 67.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (111m) 2s ago 2h 71.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (110m) 2s ago 2h 67.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (119m) 2s ago 2h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (110m) 7s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (110m) 2s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (110m) 7s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:21:20.063 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (110m) 2s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:21:20.374 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:21:20.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:21:20.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:21:20.376 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:21:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-mon[99959]: pgmap v3642: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T22:21:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-mon[99959]: from='client.52287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-mon[99959]: from='client.72212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-mon[91136]: pgmap v3642: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T22:21:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-mon[91136]: from='client.52287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-mon[91136]: from='client.72212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:21:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:21:20.646 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:21:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:20 smithi146 ceph-mon[103278]: pgmap v3642: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-20T22:21:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:20 smithi146 ceph-mon[103278]: from='client.52287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:20 smithi146 ceph-mon[103278]: from='client.72212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:20.950 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:21:20.950 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:20.950 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:20.951 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:20.951 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:20.951 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:21:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:21:20.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:20.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:20.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:20.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:20.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:20.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:21 smithi146 ceph-mon[103278]: from='client.52299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4288150959' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:21 smithi146 ceph-mon[103278]: from='client.72230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4165544606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[91136]: from='client.52299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4288150959' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[91136]: from='client.72230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4165544606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[99959]: from='client.52299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4288150959' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[99959]: from='client.72230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4165544606' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:22 smithi146 ceph-mon[103278]: pgmap v3643: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T22:21:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:22 smithi078 ceph-mon[91136]: pgmap v3643: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T22:21:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:22 smithi078 ceph-mon[99959]: pgmap v3643: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T22:21:24.196 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:24.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:24.196 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:24.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:24.196 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:24.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:24 smithi078 ceph-mon[99959]: pgmap v3644: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:21:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:24 smithi078 ceph-mon[91136]: pgmap v3644: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:21:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:24 smithi146 ceph-mon[103278]: pgmap v3644: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:21:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:26 smithi146 ceph-mon[103278]: pgmap v3645: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:21:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:26 smithi078 ceph-mon[91136]: pgmap v3645: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:21:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:26 smithi078 ceph-mon[99959]: pgmap v3645: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:21:28.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:28 smithi146 ceph-mon[103278]: pgmap v3646: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T22:21:28.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:28 smithi078 ceph-mon[91136]: pgmap v3646: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T22:21:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:28 smithi078 ceph-mon[99959]: pgmap v3646: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-20T22:21:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:30 smithi078 ceph-mon[91136]: pgmap v3647: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:21:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:30 smithi078 ceph-mon[99959]: pgmap v3647: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:21:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:30 smithi146 ceph-mon[103278]: pgmap v3647: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:21:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:21:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:21:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:30.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:30.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:30.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:32.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:32 smithi078 ceph-mon[91136]: pgmap v3648: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T22:21:32.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:32 smithi078 ceph-mon[99959]: pgmap v3648: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T22:21:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:32 smithi146 ceph-mon[103278]: pgmap v3648: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T22:21:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:34.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:34.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:34.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:34 smithi146 ceph-mon[103278]: pgmap v3649: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:21:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:34 smithi078 ceph-mon[91136]: pgmap v3649: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:21:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:34 smithi078 ceph-mon[99959]: pgmap v3649: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:21:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:36 smithi146 ceph-mon[103278]: pgmap v3650: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:21:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:36 smithi078 ceph-mon[91136]: pgmap v3650: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:21:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:36 smithi078 ceph-mon[99959]: pgmap v3650: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:21:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[91136]: pgmap v3651: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[99959]: pgmap v3651: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:38 smithi146 ceph-mon[103278]: pgmap v3651: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:21:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:40 smithi078 ceph-mon[91136]: pgmap v3652: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:21:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:40 smithi078 ceph-mon[99959]: pgmap v3652: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:21:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:40 smithi146 ceph-mon[103278]: pgmap v3652: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:21:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:21:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:21:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:40.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:40.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:40.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:42 smithi146 ceph-mon[103278]: pgmap v3653: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:21:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:42 smithi078 ceph-mon[91136]: pgmap v3653: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:21:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:42 smithi078 ceph-mon[99959]: pgmap v3653: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:21:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:44.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:44.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:44.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:44.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:44.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:44.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:44 smithi078 ceph-mon[99959]: pgmap v3654: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:44 smithi078 ceph-mon[91136]: pgmap v3654: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:44 smithi146 ceph-mon[103278]: pgmap v3654: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:46.354 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:46.354 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:46 smithi078 ceph-mon[99959]: pgmap v3655: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:46.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:46.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:46 smithi078 ceph-mon[91136]: pgmap v3655: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:46.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:46.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:46 smithi146 ceph-mon[103278]: pgmap v3655: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:48 smithi146 ceph-mon[103278]: pgmap v3656: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:48 smithi078 ceph-mon[91136]: pgmap v3656: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:48 smithi078 ceph-mon[99959]: pgmap v3656: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:50 smithi078 ceph-mon[91136]: pgmap v3657: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:50 smithi078 ceph-mon[99959]: pgmap v3657: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:50 smithi146 ceph-mon[103278]: pgmap v3657: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:21:50] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:21:51.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:50.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:51.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:50.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:50.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:51.250 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 2h - - 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 2h - - 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 19s ago 2h 30.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 14s ago 2h 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (119m) 19s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (118m) 14s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 19s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (119m) 19s ago 2h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (117m) 14s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:21:51.781 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (118m) 19s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 19s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 14s ago 2h 15.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (116m) 19s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (115m) 19s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (115m) 19s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (114m) 19s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (114m) 14s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (113m) 14s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (112m) 14s ago 2h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (111m) 14s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 14s ago 2h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (111m) 19s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (110m) 14s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:21:51.782 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (110m) 19s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:21:51.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (110m) 14s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:21:52.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:21:52.091 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:21:52.362 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:21:52.363 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:21:52.363 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[99959]: from='client.52323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.363 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[99959]: pgmap v3658: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:52.363 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[99959]: from='client.72248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.363 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[99959]: from='client.72254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/948678238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:52.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[91136]: from='client.52323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[91136]: pgmap v3658: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:52.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[91136]: from='client.72248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[91136]: from='client.72254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/948678238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:52.672 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:21:52.673 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:52.673 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:21:52.673 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:52.673 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:21:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:52 smithi146 ceph-mon[103278]: from='client.52323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:52 smithi146 ceph-mon[103278]: pgmap v3658: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:52 smithi146 ceph-mon[103278]: from='client.72248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:52 smithi146 ceph-mon[103278]: from='client.72254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/948678238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:21:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:53 smithi146 ceph-mon[103278]: from='client.52347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4168464384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[91136]: from='client.52347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4168464384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[99959]: from='client.52347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:21:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4168464384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:21:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:54.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:54.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:21:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:21:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:21:54.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:21:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:54 smithi146 ceph-mon[103278]: pgmap v3659: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:54 smithi078 ceph-mon[91136]: pgmap v3659: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:54 smithi078 ceph-mon[99959]: pgmap v3659: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:21:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:56 smithi146 ceph-mon[103278]: pgmap v3660: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:56 smithi078 ceph-mon[91136]: pgmap v3660: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:21:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:56 smithi078 ceph-mon[99959]: pgmap v3660: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:58 smithi146 ceph-mon[103278]: pgmap v3661: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:21:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[99959]: pgmap v3661: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:21:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[91136]: pgmap v3661: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:21:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:21:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:21:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:00 smithi146 ceph-mon[103278]: pgmap v3662: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:00 smithi078 ceph-mon[91136]: pgmap v3662: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:00 smithi078 ceph-mon[99959]: pgmap v3662: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:22:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:22:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:00.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:00.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:00.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:00.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:02 smithi146 ceph-mon[103278]: pgmap v3663: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:02 smithi078 ceph-mon[99959]: pgmap v3663: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:02 smithi078 ceph-mon[91136]: pgmap v3663: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:04.422 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:04.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:04.422 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:04.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:04.422 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:04.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:04.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:04 smithi078 ceph-mon[91136]: pgmap v3664: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:04.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:04 smithi078 ceph-mon[99959]: pgmap v3664: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:04 smithi146 ceph-mon[103278]: pgmap v3664: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:06 smithi146 ceph-mon[103278]: pgmap v3665: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:06 smithi078 ceph-mon[99959]: pgmap v3665: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:06 smithi078 ceph-mon[91136]: pgmap v3665: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:08.575 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:08 smithi146 ceph-mon[103278]: pgmap v3666: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:08 smithi078 ceph-mon[91136]: pgmap v3666: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:08 smithi078 ceph-mon[99959]: pgmap v3666: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:10 smithi146 ceph-mon[103278]: pgmap v3667: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:10 smithi078 ceph-mon[99959]: pgmap v3667: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:10 smithi078 ceph-mon[91136]: pgmap v3667: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:22:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:22:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:10.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:10.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:10.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:12 smithi146 ceph-mon[103278]: pgmap v3668: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:12 smithi078 ceph-mon[91136]: pgmap v3668: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:12 smithi078 ceph-mon[99959]: pgmap v3668: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:13 smithi146 ceph-mon[103278]: pgmap v3669: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[99959]: pgmap v3669: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:13 smithi078 ceph-mon[91136]: pgmap v3669: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:14.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:14.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:14.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:16 smithi146 ceph-mon[103278]: pgmap v3670: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:16 smithi078 ceph-mon[99959]: pgmap v3670: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:16 smithi078 ceph-mon[91136]: pgmap v3670: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:18 smithi146 ceph-mon[103278]: pgmap v3671: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[99959]: pgmap v3671: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[91136]: pgmap v3671: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:20 smithi146 ceph-mon[103278]: pgmap v3672: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:20 smithi078 ceph-mon[99959]: pgmap v3672: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:20 smithi078 ceph-mon[91136]: pgmap v3672: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:22:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:22:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:20.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:20.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:20.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:22.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:22 smithi078 ceph-mon[99959]: pgmap v3673: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:22.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:22 smithi078 ceph-mon[91136]: pgmap v3673: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:22.971 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:22:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:22 smithi146 ceph-mon[103278]: pgmap v3673: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 2h - - 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 2h - - 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 10s ago 2h 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 5s ago 2h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 10s ago 2h 62.3M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (119m) 5s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 10s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (119m) 10s ago 2h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:22:23.497 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (118m) 5s ago 2h 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (118m) 10s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 10s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 5s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (117m) 10s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (116m) 10s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (115m) 10s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (115m) 10s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (114m) 5s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (113m) 5s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:22:23.498 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (112m) 5s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:22:23.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (111m) 5s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:22:23.499 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 5s ago 2h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:22:23.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (111m) 10s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:22:23.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (111m) 5s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:22:23.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (111m) 10s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:22:23.499 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (111m) 5s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:22:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:23 smithi078 ceph-mon[99959]: from='client.72278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:23 smithi078 ceph-mon[91136]: from='client.72278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:23.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:22:23.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:22:23.808 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:22:23.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:22:23.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:22:23.810 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:22:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:23 smithi146 ceph-mon[103278]: from='client.72278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:24.059 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:24.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:24.080 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:22:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:22:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:22:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:22:24.081 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:22:24.391 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:22:24.391 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:24.392 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:24.392 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:24.392 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:24.392 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:24.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:24.392 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:24.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[99959]: from='client.52365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[99959]: pgmap v3674: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[99959]: from='client.52371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2338280247' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2515755625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[91136]: from='client.52365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[91136]: pgmap v3674: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[91136]: from='client.52371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2338280247' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2515755625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:24 smithi146 ceph-mon[103278]: from='client.52365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:24 smithi146 ceph-mon[103278]: pgmap v3674: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:24 smithi146 ceph-mon[103278]: from='client.52371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2338280247' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2515755625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:25 smithi078 ceph-mon[91136]: from='client.72302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:25 smithi078 ceph-mon[99959]: from='client.72302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:25 smithi146 ceph-mon[103278]: from='client.72302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:26.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:26.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:26 smithi078 ceph-mon[99959]: pgmap v3675: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:26.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:26.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:26 smithi078 ceph-mon[91136]: pgmap v3675: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:26 smithi146 ceph-mon[103278]: pgmap v3675: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:28 smithi146 ceph-mon[103278]: pgmap v3676: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:28 smithi078 ceph-mon[99959]: pgmap v3676: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:28 smithi078 ceph-mon[91136]: pgmap v3676: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:30.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:30 smithi146 ceph-mon[103278]: pgmap v3677: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:30 smithi078 ceph-mon[91136]: pgmap v3677: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:30 smithi078 ceph-mon[99959]: pgmap v3677: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:22:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:22:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:30.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:30.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:30.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:32 smithi146 ceph-mon[103278]: pgmap v3678: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:32 smithi078 ceph-mon[99959]: pgmap v3678: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:32 smithi078 ceph-mon[91136]: pgmap v3678: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:34 smithi146 ceph-mon[103278]: pgmap v3679: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[99959]: pgmap v3679: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-mon[91136]: pgmap v3679: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:34.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:34.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:34.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:34.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:36 smithi146 ceph-mon[103278]: pgmap v3680: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:36 smithi078 ceph-mon[91136]: pgmap v3680: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:36 smithi078 ceph-mon[99959]: pgmap v3680: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:38 smithi146 ceph-mon[103278]: pgmap v3681: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[91136]: pgmap v3681: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[99959]: pgmap v3681: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:40 smithi146 ceph-mon[103278]: pgmap v3682: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:40 smithi078 ceph-mon[99959]: pgmap v3682: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:40 smithi078 ceph-mon[91136]: pgmap v3682: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:22:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:22:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:40.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:40.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:40.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:42 smithi146 ceph-mon[103278]: pgmap v3683: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:42 smithi078 ceph-mon[91136]: pgmap v3683: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:42 smithi078 ceph-mon[99959]: pgmap v3683: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:44.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:44.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:44.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:44.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:44.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:44 smithi146 ceph-mon[103278]: pgmap v3684: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:44 smithi078 ceph-mon[99959]: pgmap v3684: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:44 smithi078 ceph-mon[91136]: pgmap v3684: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:46 smithi146 ceph-mon[103278]: pgmap v3685: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:46 smithi078 ceph-mon[91136]: pgmap v3685: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:46 smithi078 ceph-mon[99959]: pgmap v3685: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:48.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:48 smithi078 ceph-mon[91136]: pgmap v3686: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:48.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:48 smithi078 ceph-mon[99959]: pgmap v3686: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:48 smithi146 ceph-mon[103278]: pgmap v3686: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:50 smithi146 ceph-mon[103278]: pgmap v3687: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:50 smithi078 ceph-mon[91136]: pgmap v3687: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:50 smithi078 ceph-mon[99959]: pgmap v3687: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:22:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:22:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:50.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:50.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:50.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:52 smithi146 ceph-mon[103278]: pgmap v3688: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:52 smithi078 ceph-mon[91136]: pgmap v3688: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:52 smithi078 ceph-mon[99959]: pgmap v3688: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:54 smithi146 ceph-mon[103278]: pgmap v3689: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[91136]: pgmap v3689: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-mon[99959]: pgmap v3689: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:22:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:54.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:54.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:22:54.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:22:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:22:54.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:22:54.687 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:22:55.213 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:22:55.213 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 2h - - 2024-09-20T22:22:55.213 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 2h - - 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 1s ago 2h 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 17s ago 2h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 1s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (119m) 17s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 1s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 1s ago 2h 252M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (118m) 17s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (119m) 1s ago 2h 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 1s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 17s ago 2h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (117m) 1s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (116m) 1s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (116m) 1s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:22:55.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (115m) 1s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (115m) 17s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (114m) 17s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (113m) 17s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (112m) 17s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 17s ago 2h 75.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (112m) 1s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (111m) 17s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (111m) 1s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:22:55.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (111m) 17s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:22:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:55 smithi146 ceph-mon[103278]: from='client.72314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:55 smithi146 ceph-mon[103278]: from='client.52401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:22:55.525 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:22:55.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:55 smithi078 ceph-mon[99959]: from='client.72314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:55.525 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:55 smithi078 ceph-mon[99959]: from='client.52401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:55.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:55 smithi078 ceph-mon[91136]: from='client.72314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:55.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:55 smithi078 ceph-mon[91136]: from='client.52401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:55.793 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:22:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:22:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:22:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:22:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:22:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:22:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:22:55.811 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:22:56.100 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:22:56.100 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:56.101 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:22:56.101 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:56.101 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:22:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[91136]: from='client.72326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[91136]: pgmap v3690: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3315668880' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[91136]: from='client.72338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/434362109' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[99959]: from='client.72326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[99959]: pgmap v3690: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3315668880' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[99959]: from='client.72338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/434362109' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:56 smithi146 ceph-mon[103278]: from='client.72326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:56 smithi146 ceph-mon[103278]: pgmap v3690: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3315668880' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:56 smithi146 ceph-mon[103278]: from='client.72338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/434362109' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:58 smithi146 ceph-mon[103278]: pgmap v3691: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[91136]: pgmap v3691: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[99959]: pgmap v3691: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:22:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:00 smithi078 ceph-mon[91136]: pgmap v3692: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:00 smithi078 ceph-mon[99959]: pgmap v3692: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:23:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:23:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:00.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:00.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:00.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:00 smithi146 ceph-mon[103278]: pgmap v3692: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:02 smithi146 ceph-mon[103278]: pgmap v3693: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:02 smithi078 ceph-mon[91136]: pgmap v3693: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:02 smithi078 ceph-mon[99959]: pgmap v3693: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:04.060Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:04.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:04.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:05.115 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:04 smithi078 ceph-mon[99959]: pgmap v3694: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:05.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:04 smithi078 ceph-mon[91136]: pgmap v3694: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:04 smithi146 ceph-mon[103278]: pgmap v3694: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:06.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:06.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:06 smithi078 ceph-mon[99959]: pgmap v3695: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:06.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:06.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:06 smithi078 ceph-mon[91136]: pgmap v3695: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:06 smithi146 ceph-mon[103278]: pgmap v3695: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:09.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:08 smithi146 ceph-mon[103278]: pgmap v3696: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:09.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:08 smithi078 ceph-mon[99959]: pgmap v3696: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:09.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:08 smithi078 ceph-mon[91136]: pgmap v3696: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:10 smithi078 ceph-mon[91136]: pgmap v3697: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:10 smithi078 ceph-mon[99959]: pgmap v3697: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:23:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:23:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:10.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:10.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:10.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:11.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:10 smithi146 ceph-mon[103278]: pgmap v3697: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:12 smithi146 ceph-mon[103278]: pgmap v3698: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:12 smithi078 ceph-mon[91136]: pgmap v3698: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:12 smithi078 ceph-mon[99959]: pgmap v3698: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:14.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:14.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:14.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:14.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:14.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:14.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:15.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:14 smithi078 ceph-mon[91136]: pgmap v3699: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:15.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:14 smithi078 ceph-mon[99959]: pgmap v3699: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:14 smithi146 ceph-mon[103278]: pgmap v3699: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:16 smithi146 ceph-mon[103278]: pgmap v3700: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:16 smithi078 ceph-mon[91136]: pgmap v3700: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:16 smithi078 ceph-mon[99959]: pgmap v3700: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:18 smithi146 ceph-mon[103278]: pgmap v3701: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[91136]: pgmap v3701: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[99959]: pgmap v3701: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:20.954 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:23:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:23:20.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:20.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:20.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:20.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:20.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:20.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:20 smithi146 ceph-mon[103278]: pgmap v3702: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:20 smithi078 ceph-mon[91136]: pgmap v3702: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:20 smithi078 ceph-mon[99959]: pgmap v3702: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:22 smithi146 ceph-mon[103278]: pgmap v3703: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:22 smithi078 ceph-mon[91136]: pgmap v3703: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:22 smithi078 ceph-mon[99959]: pgmap v3703: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:24.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:24.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:24.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:24.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:24 smithi146 ceph-mon[103278]: pgmap v3704: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:24 smithi078 ceph-mon[99959]: pgmap v3704: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:24 smithi078 ceph-mon[91136]: pgmap v3704: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:26.396 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:23:26.922 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:23:26.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 2h - - 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 13s ago 2h 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 13s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 13s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 13s ago 2h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (119m) 8s ago 2h 157M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (119m) 13s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 13s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (118m) 13s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (117m) 13s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (116m) 13s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (116m) 13s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:23:26.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (115m) 8s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (114m) 8s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (113m) 8s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (112m) 8s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 8s ago 2h 76.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (112m) 13s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (112m) 8s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (112m) 13s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:23:26.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (112m) 8s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:23:27.230 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:27.231 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:23:27.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:23:27.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:27.232 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:23:27.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:23:27.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:23:27.232 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:23:27.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:27.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:26 smithi078 ceph-mon[99959]: pgmap v3705: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:27.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:27.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:26 smithi078 ceph-mon[91136]: pgmap v3705: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:26 smithi146 ceph-mon[103278]: pgmap v3705: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:27.505 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:23:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:23:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:23:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:23:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:23:27.506 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:23:27.831 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:23:27.831 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:27.831 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:27.831 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:27.831 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:27 smithi146 ceph-mon[103278]: from='client.72350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:27 smithi146 ceph-mon[103278]: from='client.72356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:27 smithi146 ceph-mon[103278]: from='client.72362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2819046665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1552124906' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[91136]: from='client.72350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[91136]: from='client.72356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[91136]: from='client.72362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2819046665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1552124906' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[99959]: from='client.72350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[99959]: from='client.72356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[99959]: from='client.72362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2819046665' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:23:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1552124906' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:23:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:28 smithi078 ceph-mon[91136]: pgmap v3706: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:28 smithi078 ceph-mon[91136]: from='client.72374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:28 smithi078 ceph-mon[99959]: pgmap v3706: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:28 smithi078 ceph-mon[99959]: from='client.72374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:28 smithi146 ceph-mon[103278]: pgmap v3706: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:28 smithi146 ceph-mon[103278]: from='client.72374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:23:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:30.972 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:23:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:23:30.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:30.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:30.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:30.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:30.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:30.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:30 smithi146 ceph-mon[103278]: pgmap v3707: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:30 smithi078 ceph-mon[91136]: pgmap v3707: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:30 smithi078 ceph-mon[99959]: pgmap v3707: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:32 smithi146 ceph-mon[103278]: pgmap v3708: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:32 smithi078 ceph-mon[91136]: pgmap v3708: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:32 smithi078 ceph-mon[99959]: pgmap v3708: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:34.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:34.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:34.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:34.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:34.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:34.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:34 smithi146 ceph-mon[103278]: pgmap v3709: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[91136]: pgmap v3709: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[99959]: pgmap v3709: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:36 smithi146 ceph-mon[103278]: pgmap v3710: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:36 smithi078 ceph-mon[91136]: pgmap v3710: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:36 smithi078 ceph-mon[99959]: pgmap v3710: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[99959]: pgmap v3711: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[91136]: pgmap v3711: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:38 smithi146 ceph-mon[103278]: pgmap v3711: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:23:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:40 smithi078 ceph-mon[91136]: pgmap v3712: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:40 smithi078 ceph-mon[99959]: pgmap v3712: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:40 smithi146 ceph-mon[103278]: pgmap v3712: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:23:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:23:41.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:40.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:40.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:40.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:42 smithi078 ceph-mon[91136]: pgmap v3713: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:42 smithi078 ceph-mon[99959]: pgmap v3713: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:42 smithi146 ceph-mon[103278]: pgmap v3713: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:44.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:44.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:44.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:44 smithi078 ceph-mon[99959]: pgmap v3714: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:44 smithi078 ceph-mon[91136]: pgmap v3714: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:44 smithi146 ceph-mon[103278]: pgmap v3714: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:46 smithi078 ceph-mon[91136]: pgmap v3715: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:46 smithi078 ceph-mon[99959]: pgmap v3715: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:46 smithi146 ceph-mon[103278]: pgmap v3715: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:48.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:48 smithi078 ceph-mon[91136]: pgmap v3716: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:48.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:48 smithi078 ceph-mon[99959]: pgmap v3716: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:48.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:48 smithi146 ceph-mon[103278]: pgmap v3716: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:50.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:50 smithi146 ceph-mon[103278]: pgmap v3717: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:50 smithi078 ceph-mon[91136]: pgmap v3717: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:23:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:23:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:50 smithi078 ceph-mon[99959]: pgmap v3717: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:50.638Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:50.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:50.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:52.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:52 smithi146 ceph-mon[103278]: pgmap v3718: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:52 smithi078 ceph-mon[91136]: pgmap v3718: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:52 smithi078 ceph-mon[99959]: pgmap v3718: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:54.377 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:54.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:54.377 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:54.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:23:54.378 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:23:54.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:23:54.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[99959]: pgmap v3719: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[91136]: pgmap v3719: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:54.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:54 smithi146 ceph-mon[103278]: pgmap v3719: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:56 smithi146 ceph-mon[103278]: pgmap v3720: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:56 smithi078 ceph-mon[91136]: pgmap v3720: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:23:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:56 smithi078 ceph-mon[99959]: pgmap v3720: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:58.125 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:23:58.673 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:23:58.673 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:23:58.673 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 2h - - 2024-09-20T22:23:58.673 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 5s ago 2h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:23:58.673 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 0s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:23:58.673 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 5s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 0s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 5s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 5s ago 2h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (119m) 0s ago 2h 158M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 5s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 5s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 0s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (118m) 5s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (117m) 5s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (117m) 5s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:23:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (116m) 5s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (116m) 0s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (115m) 0s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (114m) 0s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (113m) 0s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 0s ago 2h 73.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (113m) 5s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (112m) 0s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (113m) 5s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:23:58.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (112m) 0s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:23:58.984 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:23:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:23:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:23:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:58.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:23:58.985 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[99959]: pgmap v3721: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[91136]: pgmap v3721: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:58.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:58 smithi146 ceph-mon[103278]: pgmap v3721: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:23:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:23:59.255 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:23:59.255 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:23:59.255 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:23:59.565 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:23:59.565 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:23:59.565 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:23:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:59 smithi146 ceph-mon[103278]: from='client.72386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:59 smithi146 ceph-mon[103278]: from='client.72392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:59 smithi146 ceph-mon[103278]: from='client.72398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/507388639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:23:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3517885386' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[91136]: from='client.72386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[91136]: from='client.72392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[91136]: from='client.72398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/507388639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3517885386' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[99959]: from='client.72386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[99959]: from='client.72392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[99959]: from='client.72398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/507388639' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:23:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3517885386' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:00 smithi146 ceph-mon[103278]: from='client.52491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:00 smithi146 ceph-mon[103278]: pgmap v3722: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-mon[91136]: from='client.52491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-mon[91136]: pgmap v3722: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:24:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:24:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-mon[99959]: from='client.52491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-mon[99959]: pgmap v3722: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:00.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:00.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:00.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:02 smithi146 ceph-mon[103278]: pgmap v3723: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:02 smithi078 ceph-mon[91136]: pgmap v3723: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:02 smithi078 ceph-mon[99959]: pgmap v3723: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:04.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:04.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:04.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:04 smithi146 ceph-mon[103278]: pgmap v3724: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:04 smithi078 ceph-mon[91136]: pgmap v3724: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:04 smithi078 ceph-mon[99959]: pgmap v3724: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:06 smithi146 ceph-mon[103278]: pgmap v3725: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:06 smithi078 ceph-mon[91136]: pgmap v3725: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:06 smithi078 ceph-mon[99959]: pgmap v3725: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:08.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:08 smithi078 ceph-mon[91136]: pgmap v3726: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:08.974 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:08 smithi078 ceph-mon[99959]: pgmap v3726: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:08 smithi146 ceph-mon[103278]: pgmap v3726: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:10.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:10 smithi146 ceph-mon[103278]: pgmap v3727: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:10 smithi078 ceph-mon[91136]: pgmap v3727: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:24:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:24:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:10 smithi078 ceph-mon[99959]: pgmap v3727: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:10.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:10.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:10.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:12 smithi146 ceph-mon[103278]: pgmap v3728: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:12 smithi078 ceph-mon[91136]: pgmap v3728: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:12 smithi078 ceph-mon[99959]: pgmap v3728: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:14.065Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:14.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:14.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[91136]: pgmap v3729: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[99959]: pgmap v3729: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:14 smithi146 ceph-mon[103278]: pgmap v3729: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:16 smithi078 ceph-mon[91136]: pgmap v3730: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:16 smithi078 ceph-mon[99959]: pgmap v3730: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:16 smithi146 ceph-mon[103278]: pgmap v3730: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:19.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:18 smithi146 ceph-mon[103278]: pgmap v3731: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:19.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[91136]: pgmap v3731: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[99959]: pgmap v3731: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:20 smithi078 ceph-mon[91136]: pgmap v3732: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:20 smithi078 ceph-mon[99959]: pgmap v3732: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:24:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:24:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:20.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:20.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:20.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:20 smithi146 ceph-mon[103278]: pgmap v3732: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:23.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:22 smithi078 ceph-mon[99959]: pgmap v3733: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:23.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:22 smithi078 ceph-mon[91136]: pgmap v3733: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:22 smithi146 ceph-mon[103278]: pgmap v3733: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:24.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:24.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:24.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:25.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:24 smithi078 ceph-mon[91136]: pgmap v3734: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:25.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:24 smithi078 ceph-mon[99959]: pgmap v3734: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:24 smithi146 ceph-mon[103278]: pgmap v3734: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 22:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T22:24:25.847200852Z level=info msg="Completed cleanup jobs" duration=12.099367ms 2024-09-20T22:24:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:26 smithi146 ceph-mon[103278]: pgmap v3735: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:26 smithi078 ceph-mon[91136]: pgmap v3735: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:26 smithi078 ceph-mon[99959]: pgmap v3735: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:28 smithi146 ceph-mon[103278]: pgmap v3736: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:28 smithi078 ceph-mon[99959]: pgmap v3736: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:28 smithi078 ceph-mon[91136]: pgmap v3736: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:29.862 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 2h - - 2024-09-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 2h - - 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 16s ago 2h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 12s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 16s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 12s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 16s ago 2h 567M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 16s ago 2h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 12s ago 2h 159M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 16s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 16s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 12s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (119m) 16s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (118m) 16s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (117m) 16s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (117m) 16s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (116m) 12s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (115m) 12s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (114m) 12s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (113m) 12s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 12s ago 2h 73.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (113m) 16s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (113m) 12s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (113m) 16s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (113m) 12s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:24:30.662 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:24:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:24:30.662 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:30.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:30.663 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:30.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:30.663 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:30.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:30.716 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:24:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:24:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:24:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:24:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:24:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:24:30.717 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:24:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-mon[99959]: pgmap v3737: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-mon[99959]: from='client.72422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:30.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1958154163' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:30.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-mon[91136]: pgmap v3737: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:30.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-mon[91136]: from='client.72422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:30.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1958154163' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:30.999 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:24:31.000 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:24:31.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:30 smithi146 ceph-mon[103278]: pgmap v3737: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:31.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:30 smithi146 ceph-mon[103278]: from='client.72422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:31.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1958154163' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:24:31.361 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:24:31.362 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:24:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:24:31.362 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:24:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:31 smithi078 ceph-mon[91136]: from='client.52509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:31 smithi078 ceph-mon[91136]: from='client.52515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2177906589' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:31 smithi078 ceph-mon[99959]: from='client.52509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:31 smithi078 ceph-mon[99959]: from='client.52515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2177906589' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:32.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:31 smithi146 ceph-mon[103278]: from='client.52509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:32.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:31 smithi146 ceph-mon[103278]: from='client.52515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:32.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2177906589' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:24:33.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:32 smithi146 ceph-mon[103278]: from='client.52527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:33.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:32 smithi146 ceph-mon[103278]: pgmap v3738: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:32 smithi078 ceph-mon[91136]: from='client.52527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:32 smithi078 ceph-mon[91136]: pgmap v3738: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:32 smithi078 ceph-mon[99959]: from='client.52527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:24:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:32 smithi078 ceph-mon[99959]: pgmap v3738: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:34.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:34.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:34.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:34.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:35.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[99959]: pgmap v3739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:35.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[91136]: pgmap v3739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:35.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:34 smithi146 ceph-mon[103278]: pgmap v3739: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:36 smithi146 ceph-mon[103278]: pgmap v3740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:36 smithi078 ceph-mon[99959]: pgmap v3740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:36 smithi078 ceph-mon[91136]: pgmap v3740: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:38 smithi146 ceph-mon[103278]: pgmap v3741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[99959]: pgmap v3741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[91136]: pgmap v3741: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:40.915 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:24:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:24:40.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:40.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:40.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:40.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:40.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:40.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:40 smithi146 ceph-mon[103278]: pgmap v3742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:40 smithi078 ceph-mon[91136]: pgmap v3742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:40 smithi078 ceph-mon[99959]: pgmap v3742: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:42 smithi146 ceph-mon[103278]: pgmap v3743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:42 smithi078 ceph-mon[91136]: pgmap v3743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:42 smithi078 ceph-mon[99959]: pgmap v3743: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:44.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:44.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:44.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:45.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:44 smithi078 ceph-mon[91136]: pgmap v3744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:45.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:44 smithi078 ceph-mon[99959]: pgmap v3744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:44 smithi146 ceph-mon[103278]: pgmap v3744: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:46 smithi146 ceph-mon[103278]: pgmap v3745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:46 smithi078 ceph-mon[99959]: pgmap v3745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:46 smithi078 ceph-mon[91136]: pgmap v3745: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:48 smithi146 ceph-mon[103278]: pgmap v3746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:48 smithi078 ceph-mon[99959]: pgmap v3746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:48 smithi078 ceph-mon[91136]: pgmap v3746: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:50.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:24:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:24:50.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:50.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:50.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:50.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:50.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:50.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:51.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:50 smithi078 ceph-mon[91136]: pgmap v3747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:51.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:50 smithi078 ceph-mon[99959]: pgmap v3747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:50 smithi146 ceph-mon[103278]: pgmap v3747: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:53.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:52 smithi146 ceph-mon[103278]: pgmap v3748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:52 smithi078 ceph-mon[91136]: pgmap v3748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:52 smithi078 ceph-mon[99959]: pgmap v3748: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:54.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:54.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:24:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:24:54.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:24:55.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[99959]: pgmap v3749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[91136]: pgmap v3749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:54 smithi146 ceph-mon[103278]: pgmap v3749: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:24:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:56 smithi146 ceph-mon[103278]: pgmap v3750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:56 smithi078 ceph-mon[99959]: pgmap v3750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:24:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:56 smithi078 ceph-mon[91136]: pgmap v3750: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:58 smithi146 ceph-mon[103278]: pgmap v3751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[99959]: pgmap v3751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[91136]: pgmap v3751: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:24:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:24:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:24:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:00.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:25:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:25:00.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:00.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:00.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:00.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:00.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:00.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:00 smithi146 ceph-mon[103278]: pgmap v3752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:00 smithi078 ceph-mon[99959]: pgmap v3752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:00 smithi078 ceph-mon[91136]: pgmap v3752: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:01.655 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:25:02.182 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:25:02.182 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 2h - - 2024-09-20T22:25:02.182 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 2h - - 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 8s ago 2h 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 3s ago 2h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 8s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 3s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 8s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 8s ago 2h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 3s ago 2h 160M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 8s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 8s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 3s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (119m) 8s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:25:02.183 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (118m) 8s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (118m) 8s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (117m) 8s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (117m) 3s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (116m) 3s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (115m) 3s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (114m) 3s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 3s ago 2h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (114m) 8s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (114m) 3s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (114m) 8s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:25:02.184 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (113m) 3s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:25:02.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:25:02.494 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:25:02.760 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:25:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:25:02.762 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:25:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[91136]: pgmap v3753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[91136]: from='client.52539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[91136]: from='client.72464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3976442017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[99959]: pgmap v3753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[99959]: from='client.52539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[99959]: from='client.72464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3976442017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:03.080 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:25:03.080 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:25:03.080 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:25:03.080 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:25:03.080 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:25:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:02 smithi146 ceph-mon[103278]: pgmap v3753: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:02 smithi146 ceph-mon[103278]: from='client.52539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:02 smithi146 ceph-mon[103278]: from='client.72464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3976442017' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:03 smithi146 ceph-mon[103278]: from='client.72470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:03 smithi146 ceph-mon[103278]: from='client.52563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2436624609' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:03 smithi078 ceph-mon[91136]: from='client.72470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:03 smithi078 ceph-mon[91136]: from='client.52563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2436624609' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:03 smithi078 ceph-mon[99959]: from='client.72470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:03 smithi078 ceph-mon[99959]: from='client.52563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2436624609' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:04.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:04.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:04.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:04.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:05.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:04 smithi078 ceph-mon[91136]: pgmap v3754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:05.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:04 smithi078 ceph-mon[99959]: pgmap v3754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:04 smithi146 ceph-mon[103278]: pgmap v3754: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:06 smithi146 ceph-mon[103278]: pgmap v3755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:06 smithi078 ceph-mon[91136]: pgmap v3755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:06 smithi078 ceph-mon[99959]: pgmap v3755: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:08 smithi146 ceph-mon[103278]: pgmap v3756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:08 smithi078 ceph-mon[99959]: pgmap v3756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:08 smithi078 ceph-mon[91136]: pgmap v3756: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:10.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:25:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:25:10.905 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:10.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:10.905 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:10.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:10.905 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:10.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:10 smithi146 ceph-mon[103278]: pgmap v3757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:10 smithi078 ceph-mon[91136]: pgmap v3757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:10 smithi078 ceph-mon[99959]: pgmap v3757: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:13.242 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:12 smithi078 ceph-mon[99959]: pgmap v3758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:13.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:12 smithi078 ceph-mon[91136]: pgmap v3758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:12 smithi146 ceph-mon[103278]: pgmap v3758: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:14.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:14.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:14.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:14.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:14.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:14.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:14.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:14.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:14 smithi146 ceph-mon[103278]: pgmap v3759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[91136]: pgmap v3759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[99959]: pgmap v3759: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:16 smithi146 ceph-mon[103278]: pgmap v3760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:16 smithi078 ceph-mon[99959]: pgmap v3760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:16 smithi078 ceph-mon[91136]: pgmap v3760: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:18 smithi146 ceph-mon[103278]: pgmap v3761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[91136]: pgmap v3761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:19.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.310 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.310 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[99959]: pgmap v3761: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:19.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:19.311 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:20.995 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:25:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:25:20.996 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:20.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:20.996 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:20.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:20.996 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:20.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:20 smithi146 ceph-mon[103278]: pgmap v3762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:20 smithi078 ceph-mon[91136]: pgmap v3762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:20 smithi078 ceph-mon[99959]: pgmap v3762: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:22 smithi146 ceph-mon[103278]: pgmap v3763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:22 smithi078 ceph-mon[91136]: pgmap v3763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:22 smithi078 ceph-mon[99959]: pgmap v3763: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:24.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:24.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:24.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:25.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:25 smithi078 ceph-mon[91136]: pgmap v3764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:25.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:25 smithi078 ceph-mon[99959]: pgmap v3764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:25 smithi146 ceph-mon[103278]: pgmap v3764: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:26 smithi146 ceph-mon[103278]: pgmap v3765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:26 smithi078 ceph-mon[91136]: pgmap v3765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:26 smithi078 ceph-mon[99959]: pgmap v3765: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:28 smithi146 ceph-mon[103278]: pgmap v3766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:28 smithi078 ceph-mon[91136]: pgmap v3766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:28 smithi078 ceph-mon[99959]: pgmap v3766: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:30 smithi146 ceph-mon[103278]: pgmap v3767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:30 smithi078 ceph-mon[91136]: pgmap v3767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:30 smithi078 ceph-mon[99959]: pgmap v3767: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:25:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:25:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:30.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:30.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:30.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:32 smithi146 ceph-mon[103278]: pgmap v3768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:32.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:32 smithi078 ceph-mon[99959]: pgmap v3768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:32.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:32 smithi078 ceph-mon[91136]: pgmap v3768: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:33.378 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:25:33.934 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:25:33.934 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 2h - - 2024-09-20T22:25:33.934 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 2h - - 2024-09-20T22:25:33.934 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 19s ago 2h 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 15s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 19s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 15s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 19s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 19s ago 2h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 15s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 19s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 19s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 15s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 19s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (119m) 19s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (118m) 19s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (118m) 19s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (117m) 15s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:25:33.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (116m) 15s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (115m) 15s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (115m) 15s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 15s ago 2h 75.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (114m) 19s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (114m) 15s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (114m) 19s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:25:33.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (114m) 15s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:25:34.257 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:25:34.257 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:25:34.257 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:25:34.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:25:34.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:25:34.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:34.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:34.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:34.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:34.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:34.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: pgmap v3769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='client.52575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='client.52581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='client.52587 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3435759195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: pgmap v3769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='client.52575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='client.52581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='client.52587 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3435759195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:25:34.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:25:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:25:34.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: pgmap v3769: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:34.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='client.52575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='client.52581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='client.52587 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:34.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3435759195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:25:34.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:34.854 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:25:34.854 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:25:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:25:34.854 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:25:34.854 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:25:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:35 smithi078 ceph-mon[91136]: from='client.72518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3607797445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:35 smithi078 ceph-mon[99959]: from='client.72518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3607797445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:35 smithi146 ceph-mon[103278]: from='client.72518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:25:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3607797445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:25:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:36 smithi146 ceph-mon[103278]: pgmap v3770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:36 smithi078 ceph-mon[91136]: pgmap v3770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:36 smithi078 ceph-mon[99959]: pgmap v3770: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:38 smithi146 ceph-mon[103278]: pgmap v3771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:38 smithi078 ceph-mon[91136]: pgmap v3771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:38 smithi078 ceph-mon[99959]: pgmap v3771: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:40.876 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:25:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:25:40.877 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:40.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:40.877 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:40.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:40.877 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:40.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:40 smithi146 ceph-mon[103278]: pgmap v3772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:40 smithi078 ceph-mon[91136]: pgmap v3772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:40 smithi078 ceph-mon[99959]: pgmap v3772: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:42 smithi146 ceph-mon[103278]: pgmap v3773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:42 smithi078 ceph-mon[91136]: pgmap v3773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:42 smithi078 ceph-mon[99959]: pgmap v3773: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:44.073Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:44.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:44.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:44 smithi078 ceph-mon[91136]: pgmap v3774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:44 smithi078 ceph-mon[99959]: pgmap v3774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:44 smithi146 ceph-mon[103278]: pgmap v3774: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:46 smithi146 ceph-mon[103278]: pgmap v3775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:46 smithi078 ceph-mon[99959]: pgmap v3775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:46 smithi078 ceph-mon[91136]: pgmap v3775: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:48 smithi146 ceph-mon[103278]: pgmap v3776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:48 smithi078 ceph-mon[91136]: pgmap v3776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:48 smithi078 ceph-mon[99959]: pgmap v3776: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:50.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:25:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:25:50.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:50.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:50.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:50.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:50.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:50.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:50 smithi146 ceph-mon[103278]: pgmap v3777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:50 smithi078 ceph-mon[91136]: pgmap v3777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:50 smithi078 ceph-mon[99959]: pgmap v3777: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:52 smithi078 ceph-mon[91136]: pgmap v3778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:52 smithi078 ceph-mon[99959]: pgmap v3778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:52 smithi146 ceph-mon[103278]: pgmap v3778: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:54.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:54.074Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:54.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:54.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:25:54.216 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:25:54.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:25:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:54 smithi146 ceph-mon[103278]: pgmap v3779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[91136]: pgmap v3779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:55.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[99959]: pgmap v3779: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:25:55.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:55.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:25:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:56 smithi146 ceph-mon[103278]: pgmap v3780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:56 smithi078 ceph-mon[91136]: pgmap v3780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:25:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:56 smithi078 ceph-mon[99959]: pgmap v3780: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:58 smithi146 ceph-mon[103278]: pgmap v3781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:58 smithi078 ceph-mon[91136]: pgmap v3781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:25:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:58 smithi078 ceph-mon[99959]: pgmap v3781: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:25:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:01.005 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:26:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:26:01.005 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:00.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:01.005 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:00.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:01.005 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:00.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:01 smithi146 ceph-mon[103278]: pgmap v3782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:01 smithi078 ceph-mon[91136]: pgmap v3782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:01 smithi078 ceph-mon[99959]: pgmap v3782: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:03 smithi146 ceph-mon[103278]: pgmap v3783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:03 smithi078 ceph-mon[91136]: pgmap v3783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:03 smithi078 ceph-mon[99959]: pgmap v3783: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:04 smithi146 ceph-mon[103278]: pgmap v3784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:04 smithi078 ceph-mon[99959]: pgmap v3784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:04 smithi078 ceph-mon[91136]: pgmap v3784: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:04.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:04.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:04.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:05.147 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:26:05.675 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:26:05.675 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 2h - - 2024-09-20T22:26:05.675 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 2h - - 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 11s ago 2h 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 6s ago 2h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 11s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 6s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 11s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 11s ago 2h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 6s ago 2h 162M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 11s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 11s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 6s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 11s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (119m) 11s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:26:05.676 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (119m) 11s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (118m) 11s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (118m) 6s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (117m) 6s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (116m) 6s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (115m) 6s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 6s ago 2h 76.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (115m) 11s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (115m) 6s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (115m) 11s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:26:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (114m) 6s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:26:05.983 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:26:06.252 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:26:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:26:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:26:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:26:06.252 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:26:06.253 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:26:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[91136]: from='client.72530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[91136]: pgmap v3785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[91136]: from='client.52617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[91136]: from='client.52623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4079598374' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[99959]: from='client.72530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[99959]: pgmap v3785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[99959]: from='client.52617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[99959]: from='client.52623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4079598374' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:06.586 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:26:06.586 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:06.586 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:06 smithi146 ceph-mon[103278]: from='client.72530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:06 smithi146 ceph-mon[103278]: pgmap v3785: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:06 smithi146 ceph-mon[103278]: from='client.52617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:06 smithi146 ceph-mon[103278]: from='client.52623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4079598374' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:07 smithi146 ceph-mon[103278]: from='client.72554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3362203642' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:07 smithi078 ceph-mon[91136]: from='client.72554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3362203642' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:07 smithi078 ceph-mon[99959]: from='client.72554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3362203642' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:08 smithi146 ceph-mon[103278]: pgmap v3786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:08 smithi078 ceph-mon[99959]: pgmap v3786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:08 smithi078 ceph-mon[91136]: pgmap v3786: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:10 smithi146 ceph-mon[103278]: pgmap v3787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:10 smithi078 ceph-mon[99959]: pgmap v3787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:26:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:26:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:10 smithi078 ceph-mon[91136]: pgmap v3787: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:10.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:10.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:10.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:10.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:12 smithi146 ceph-mon[103278]: pgmap v3788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:12.771 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:12 smithi078 ceph-mon[99959]: pgmap v3788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:12.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:12 smithi078 ceph-mon[91136]: pgmap v3788: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:14.362 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:14.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:14.362 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:14.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:14.363 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:14.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:14.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:14 smithi146 ceph-mon[103278]: pgmap v3789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:14.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:14 smithi078 ceph-mon[99959]: pgmap v3789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:14.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:14 smithi078 ceph-mon[91136]: pgmap v3789: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:16 smithi146 ceph-mon[103278]: pgmap v3790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:16 smithi078 ceph-mon[91136]: pgmap v3790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:16 smithi078 ceph-mon[99959]: pgmap v3790: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:18 smithi146 ceph-mon[103278]: pgmap v3791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:18 smithi078 ceph-mon[91136]: pgmap v3791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:18 smithi078 ceph-mon[99959]: pgmap v3791: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:20 smithi146 ceph-mon[103278]: pgmap v3792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[91136]: pgmap v3792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-mon[99959]: pgmap v3792: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:26:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:26:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:20.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:20.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:20.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:22 smithi146 ceph-mon[103278]: pgmap v3793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:22 smithi078 ceph-mon[91136]: pgmap v3793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:22 smithi078 ceph-mon[99959]: pgmap v3793: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:24.352 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:24.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:24.353 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:24.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:24.353 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:24.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:24 smithi146 ceph-mon[103278]: pgmap v3794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:24 smithi078 ceph-mon[91136]: pgmap v3794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:24 smithi078 ceph-mon[99959]: pgmap v3794: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:26 smithi146 ceph-mon[103278]: pgmap v3795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:26 smithi078 ceph-mon[99959]: pgmap v3795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:26 smithi078 ceph-mon[91136]: pgmap v3795: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:28 smithi146 ceph-mon[103278]: pgmap v3796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:28 smithi078 ceph-mon[99959]: pgmap v3796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:28 smithi078 ceph-mon[91136]: pgmap v3796: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:30 smithi146 ceph-mon[103278]: pgmap v3797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:30 smithi078 ceph-mon[91136]: pgmap v3797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:30 smithi078 ceph-mon[99959]: pgmap v3797: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:26:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:26:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:30.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:30.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:30.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:32 smithi146 ceph-mon[103278]: pgmap v3798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:32 smithi078 ceph-mon[91136]: pgmap v3798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:32 smithi078 ceph-mon[99959]: pgmap v3798: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:34.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:34.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:34.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:34.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:34.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:34.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:34.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:34 smithi078 ceph-mon[91136]: pgmap v3799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:34.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:34 smithi078 ceph-mon[99959]: pgmap v3799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:34.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:34 smithi146 ceph-mon[103278]: pgmap v3799: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:35.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:35.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:35.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:36.882 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:26:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:36 smithi146 ceph-mon[103278]: pgmap v3800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:36 smithi078 ceph-mon[99959]: pgmap v3800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:36 smithi078 ceph-mon[91136]: pgmap v3800: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 2h - - 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 2h - - 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 2s ago 2h 30.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 18s ago 2h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 2s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 18s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 2s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 2s ago 2h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:26:37.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 18s ago 2h 160M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 2s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 2s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 18s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 2s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 2s ago 2h 75.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (119m) 2s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (119m) 2s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (118m) 18s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (117m) 18s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (116m) 18s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (116m) 18s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 18s ago 2h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (115m) 2s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (115m) 18s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:26:37.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (115m) 2s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:26:37.410 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (115m) 18s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:26:37.726 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:26:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:26:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:26:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:26:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:26:37.727 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:26:37.997 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:26:37.998 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:26:37.999 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:26:37.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:37 smithi078 ceph-mon[91136]: from='client.52647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:37.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:37 smithi078 ceph-mon[99959]: from='client.52647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:37 smithi146 ceph-mon[103278]: from='client.52647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:38.310 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:26:38.310 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:38.311 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:26:38.311 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:38.311 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:38 smithi146 ceph-mon[103278]: from='client.52653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:38 smithi146 ceph-mon[103278]: pgmap v3801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:38 smithi146 ceph-mon[103278]: from='client.72578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2499080955' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:38 smithi146 ceph-mon[103278]: from='client.52671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3592235402' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[91136]: from='client.52653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[91136]: pgmap v3801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[91136]: from='client.72578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2499080955' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[91136]: from='client.52671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3592235402' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[99959]: from='client.52653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[99959]: pgmap v3801: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[99959]: from='client.72578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2499080955' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[99959]: from='client.52671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3592235402' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:26:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:40 smithi146 ceph-mon[103278]: pgmap v3802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[91136]: pgmap v3802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-mon[99959]: pgmap v3802: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:26:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:26:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:40.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:40.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:40.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:42 smithi146 ceph-mon[103278]: pgmap v3803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:42 smithi078 ceph-mon[91136]: pgmap v3803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:42 smithi078 ceph-mon[99959]: pgmap v3803: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:44.441 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:44.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:44.441 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:44.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:44.441 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:44.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:44.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:44 smithi078 ceph-mon[91136]: pgmap v3804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:44.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:44 smithi078 ceph-mon[99959]: pgmap v3804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:44 smithi146 ceph-mon[103278]: pgmap v3804: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:46 smithi146 ceph-mon[103278]: pgmap v3805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:46 smithi078 ceph-mon[99959]: pgmap v3805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:46 smithi078 ceph-mon[91136]: pgmap v3805: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:48 smithi146 ceph-mon[103278]: pgmap v3806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:48 smithi078 ceph-mon[91136]: pgmap v3806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:48 smithi078 ceph-mon[99959]: pgmap v3806: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:50 smithi146 ceph-mon[103278]: pgmap v3807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:50 smithi078 ceph-mon[91136]: pgmap v3807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:50 smithi078 ceph-mon[99959]: pgmap v3807: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:26:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:26:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:50.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:50.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:50.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:50.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:52.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:52 smithi146 ceph-mon[103278]: pgmap v3808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:52 smithi078 ceph-mon[91136]: pgmap v3808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:52 smithi078 ceph-mon[99959]: pgmap v3808: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:54.384 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:54.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:54.384 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:54.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:26:54.384 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:26:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:26:54.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:26:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:54 smithi146 ceph-mon[103278]: pgmap v3809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:54 smithi078 ceph-mon[91136]: pgmap v3809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:54 smithi078 ceph-mon[99959]: pgmap v3809: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:26:55.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:55.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:55.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:55.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:55.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:55.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:56.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:26:57.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:57.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:56 smithi146 ceph-mon[103278]: pgmap v3810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:56 smithi078 ceph-mon[91136]: pgmap v3810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:26:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:56 smithi078 ceph-mon[99959]: pgmap v3810: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:58 smithi146 ceph-mon[103278]: pgmap v3811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:58 smithi078 ceph-mon[91136]: pgmap v3811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:26:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:58 smithi078 ceph-mon[99959]: pgmap v3811: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:26:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:00.653 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:27:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[91136]: pgmap v3812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[99959]: pgmap v3812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:00.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.655 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:00 smithi146 ceph-mon[103278]: pgmap v3812: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:00.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:00.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:00.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:02 smithi146 ceph-mon[103278]: pgmap v3813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:02 smithi078 ceph-mon[91136]: pgmap v3813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:02 smithi078 ceph-mon[99959]: pgmap v3813: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:04.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:04.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:04.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:04.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:04.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:04.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:04 smithi146 ceph-mon[103278]: pgmap v3814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:04 smithi078 ceph-mon[91136]: pgmap v3814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:04 smithi078 ceph-mon[99959]: pgmap v3814: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:06 smithi146 ceph-mon[103278]: pgmap v3815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:06 smithi078 ceph-mon[99959]: pgmap v3815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:06 smithi078 ceph-mon[91136]: pgmap v3815: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:08.610 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:27:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:08 smithi146 ceph-mon[103278]: pgmap v3816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:08 smithi078 ceph-mon[99959]: pgmap v3816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:08 smithi078 ceph-mon[91136]: pgmap v3816: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:09.135 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:27:09.135 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 2h - - 2024-09-20T22:27:09.135 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 2h - - 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 14s ago 2h 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 9s ago 2h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 9s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 2h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 9s ago 2h 161M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 14s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 9s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (119m) 14s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:27:09.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (119m) 9s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (118m) 9s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (117m) 9s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (116m) 9s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 9s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (116m) 14s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (116m) 9s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (116m) 14s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:27:09.137 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (116m) 9s ago 2h 88.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:27:09.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:27:09.451 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:27:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:27:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:27:09.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:09 smithi078 ceph-mon[91136]: from='client.52683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:09.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:09 smithi078 ceph-mon[91136]: from='client.52689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:09.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:09 smithi078 ceph-mon[99959]: from='client.52683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:09.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:09 smithi078 ceph-mon[99959]: from='client.52689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:09 smithi146 ceph-mon[103278]: from='client.52683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:09 smithi146 ceph-mon[103278]: from='client.52689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.029 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:27:10.029 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:10.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:10.029 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:10.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:10 smithi146 ceph-mon[103278]: from='client.72614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:10 smithi146 ceph-mon[103278]: pgmap v3817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1024857126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:10 smithi146 ceph-mon[103278]: from='client.52707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/725218254' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:27:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[99959]: from='client.72614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[99959]: pgmap v3817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1024857126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[99959]: from='client.52707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/725218254' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[91136]: from='client.72614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[91136]: pgmap v3817: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1024857126' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[91136]: from='client.52707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:10.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/725218254' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:10.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:10.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:10.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:10.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:12 smithi146 ceph-mon[103278]: pgmap v3818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:12 smithi078 ceph-mon[91136]: pgmap v3818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:12 smithi078 ceph-mon[99959]: pgmap v3818: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:14.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:14.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:14.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:14.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:14.471 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:14.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:14 smithi146 ceph-mon[103278]: pgmap v3819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:14 smithi078 ceph-mon[91136]: pgmap v3819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:14 smithi078 ceph-mon[99959]: pgmap v3819: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:16.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.165 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:16 smithi078 ceph-mon[91136]: pgmap v3820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:16 smithi078 ceph-mon[99959]: pgmap v3820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:17.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:17.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:16 smithi146 ceph-mon[103278]: pgmap v3820: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:18 smithi146 ceph-mon[103278]: pgmap v3821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:18 smithi078 ceph-mon[99959]: pgmap v3821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:18 smithi078 ceph-mon[91136]: pgmap v3821: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:20 smithi146 ceph-mon[103278]: pgmap v3822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[91136]: pgmap v3822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:27:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:27:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[99959]: pgmap v3822: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:20.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:20.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:20.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:20.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:22 smithi146 ceph-mon[103278]: pgmap v3823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:22 smithi078 ceph-mon[99959]: pgmap v3823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:22 smithi078 ceph-mon[91136]: pgmap v3823: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:24.516 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:24.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:24.516 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:24.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:24.517 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:24.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:24 smithi078 ceph-mon[91136]: pgmap v3824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:24 smithi078 ceph-mon[99959]: pgmap v3824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:24 smithi146 ceph-mon[103278]: pgmap v3824: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:26 smithi078 ceph-mon[99959]: pgmap v3825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:26 smithi078 ceph-mon[91136]: pgmap v3825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:26 smithi146 ceph-mon[103278]: pgmap v3825: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:28 smithi078 ceph-mon[99959]: pgmap v3826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:28 smithi078 ceph-mon[91136]: pgmap v3826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:28 smithi146 ceph-mon[103278]: pgmap v3826: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:30 smithi078 ceph-mon[99959]: pgmap v3827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:30 smithi078 ceph-mon[91136]: pgmap v3827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:27:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:27:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:30.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:30.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:30.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:30 smithi146 ceph-mon[103278]: pgmap v3827: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:32 smithi078 ceph-mon[91136]: pgmap v3828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:32 smithi078 ceph-mon[99959]: pgmap v3828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:32 smithi146 ceph-mon[103278]: pgmap v3828: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:34.532 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:34.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:34.532 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:34.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:34.532 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:34.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:34 smithi078 ceph-mon[91136]: pgmap v3829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:34 smithi078 ceph-mon[99959]: pgmap v3829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:34 smithi146 ceph-mon[103278]: pgmap v3829: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[91136]: pgmap v3830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:36.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:36 smithi078 ceph-mon[99959]: pgmap v3830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:36.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:36.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:36.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:36 smithi146 ceph-mon[103278]: pgmap v3830: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:38 smithi078 ceph-mon[91136]: pgmap v3831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:38 smithi078 ceph-mon[99959]: pgmap v3831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:38.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:38 smithi146 ceph-mon[103278]: pgmap v3831: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:40.326 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 2h - - 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 5s ago 2h 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 1s ago 2h 87.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 5s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:27:40.855 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 1s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 5s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 5s ago 2h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 1s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 5s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 5s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 1s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 5s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 5s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 5s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 5s ago 2h 75.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (119m) 1s ago 2h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (118m) 1s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (118m) 1s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (117m) 1s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:27:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 1s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:27:40.857 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (116m) 5s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:27:40.857 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (116m) 1s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:27:40.857 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (116m) 5s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:27:40.857 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (116m) 1s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:27:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:40 smithi146 ceph-mon[103278]: pgmap v3832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:27:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:27:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[99959]: pgmap v3832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[91136]: pgmap v3832: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:40.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:40.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:40.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:27:41.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:27:41.165 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:27:41.436 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:27:41.437 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:27:41.437 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:27:41.437 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:27:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:27:41.749 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:27:41.749 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:41.749 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:27:41.749 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:41.749 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[99959]: from='client.52719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[99959]: from='client.72644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[99959]: from='client.52731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2946891153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[91136]: from='client.52719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[91136]: from='client.72644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[91136]: from='client.52731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:41.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2946891153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:41 smithi146 ceph-mon[103278]: from='client.52719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:41 smithi146 ceph-mon[103278]: from='client.72644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:41 smithi146 ceph-mon[103278]: from='client.52731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2946891153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:27:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:42 smithi146 ceph-mon[103278]: pgmap v3833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:42 smithi146 ceph-mon[103278]: from='client.52743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/987908952' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:42 smithi078 ceph-mon[91136]: pgmap v3833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:42 smithi078 ceph-mon[91136]: from='client.52743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/987908952' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:42 smithi078 ceph-mon[99959]: pgmap v3833: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:42 smithi078 ceph-mon[99959]: from='client.52743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:27:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/987908952' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:27:43.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:43.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:44.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:44.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:44.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:44.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:44 smithi146 ceph-mon[103278]: pgmap v3834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:44 smithi078 ceph-mon[91136]: pgmap v3834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:44 smithi078 ceph-mon[99959]: pgmap v3834: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:46 smithi146 ceph-mon[103278]: pgmap v3835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:46 smithi078 ceph-mon[91136]: pgmap v3835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:46 smithi078 ceph-mon[99959]: pgmap v3835: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:48 smithi146 ceph-mon[103278]: pgmap v3836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:48 smithi078 ceph-mon[99959]: pgmap v3836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:48 smithi078 ceph-mon[91136]: pgmap v3836: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:50 smithi146 ceph-mon[103278]: pgmap v3837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:50 smithi078 ceph-mon[91136]: pgmap v3837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:50 smithi078 ceph-mon[99959]: pgmap v3837: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:27:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:27:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:50.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:50.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:50.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:52 smithi146 ceph-mon[103278]: pgmap v3838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:52 smithi078 ceph-mon[91136]: pgmap v3838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:52 smithi078 ceph-mon[99959]: pgmap v3838: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:54.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:54.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:27:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:27:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:27:54.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:27:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:54 smithi146 ceph-mon[103278]: pgmap v3839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:54 smithi078 ceph-mon[99959]: pgmap v3839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:54 smithi078 ceph-mon[91136]: pgmap v3839: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:27:56.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:56.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[99959]: pgmap v3840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:56.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:56.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:56 smithi078 ceph-mon[91136]: pgmap v3840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:27:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:27:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:56 smithi146 ceph-mon[103278]: pgmap v3840: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:58.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:58 smithi146 ceph-mon[103278]: pgmap v3841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:58.583 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:58 smithi078 ceph-mon[91136]: pgmap v3841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:27:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:58 smithi078 ceph-mon[99959]: pgmap v3841: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:27:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:00 smithi146 ceph-mon[103278]: pgmap v3842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[91136]: pgmap v3842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:28:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:28:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[99959]: pgmap v3842: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:01.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:00.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:01.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:00.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:01.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:00.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:02 smithi078 ceph-mon[91136]: pgmap v3843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:02 smithi078 ceph-mon[99959]: pgmap v3843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:02 smithi146 ceph-mon[103278]: pgmap v3843: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:04.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:04.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:04.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:04 smithi078 ceph-mon[91136]: pgmap v3844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:04 smithi078 ceph-mon[99959]: pgmap v3844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:04 smithi146 ceph-mon[103278]: pgmap v3844: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:06 smithi078 ceph-mon[91136]: pgmap v3845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:28:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:06 smithi078 ceph-mon[99959]: pgmap v3845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:28:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:06 smithi146 ceph-mon[103278]: pgmap v3845: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:28:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:08 smithi078 ceph-mon[91136]: pgmap v3846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:08 smithi078 ceph-mon[99959]: pgmap v3846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:08 smithi146 ceph-mon[103278]: pgmap v3846: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:10 smithi078 ceph-mon[91136]: pgmap v3847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:10 smithi078 ceph-mon[99959]: pgmap v3847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:28:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:28:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:10.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:10.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:10.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:10 smithi146 ceph-mon[103278]: pgmap v3847: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:12.042 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:28:12.569 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 2h - - 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 2h - - 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 17s ago 2h 31.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 12s ago 2h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 12s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:28:12.570 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:28:12.571 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 2h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:28:12.571 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 12s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:28:12.571 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:28:12.571 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 17s ago 2h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:28:12.571 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 12s ago 2h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:28:12.571 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 12s ago 2h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (119m) 12s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (118m) 12s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:28:12.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (117m) 12s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:28:12.573 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 12s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:28:12.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (117m) 17s ago 2h 88.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:28:12.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (117m) 12s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:28:12.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (117m) 17s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:28:12.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (117m) 12s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:28:12.882 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:28:12.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:12.882 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:28:12.882 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:28:12.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:28:12.882 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:28:12.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:12 smithi078 ceph-mon[99959]: pgmap v3848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:12.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:12 smithi078 ceph-mon[91136]: pgmap v3848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:13.149 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:28:13.149 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:28:13.149 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:28:13.149 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:28:13.149 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:28:13.150 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:28:13.151 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:28:13.151 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:28:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:12 smithi146 ceph-mon[103278]: pgmap v3848: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:13.463 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:13.464 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[99959]: from='client.52755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[99959]: from='client.72680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[99959]: from='client.72686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3402836612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1224012514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[91136]: from='client.52755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[91136]: from='client.72680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[91136]: from='client.72686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3402836612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:14.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:14.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1224012514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:13 smithi146 ceph-mon[103278]: from='client.52755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:13 smithi146 ceph-mon[103278]: from='client.72680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:13 smithi146 ceph-mon[103278]: from='client.72686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3402836612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1224012514' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:14.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:14.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:14.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:14.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:14.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-mon[91136]: from='client.72698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-mon[91136]: pgmap v3849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-mon[99959]: from='client.72698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:14 smithi078 ceph-mon[99959]: pgmap v3849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:14 smithi146 ceph-mon[103278]: from='client.72698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:14 smithi146 ceph-mon[103278]: pgmap v3849: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:16 smithi146 ceph-mon[103278]: pgmap v3850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[99959]: pgmap v3850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:16 smithi078 ceph-mon[91136]: pgmap v3850: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:18.627 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:18 smithi078 ceph-mon[99959]: pgmap v3851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:18.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:18 smithi078 ceph-mon[91136]: pgmap v3851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:18.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:18 smithi146 ceph-mon[103278]: pgmap v3851: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:20.884 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:28:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:28:20.884 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:20.659Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:20.884 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:20.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:20.884 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:20.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:20 smithi146 ceph-mon[103278]: pgmap v3852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[91136]: pgmap v3852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[99959]: pgmap v3852: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:22 smithi146 ceph-mon[103278]: pgmap v3853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:22 smithi078 ceph-mon[91136]: pgmap v3853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:22 smithi078 ceph-mon[99959]: pgmap v3853: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:24.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:24.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:24.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:24 smithi146 ceph-mon[103278]: pgmap v3854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:24 smithi078 ceph-mon[91136]: pgmap v3854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:24 smithi078 ceph-mon[99959]: pgmap v3854: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:26 smithi146 ceph-mon[103278]: pgmap v3855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:26 smithi078 ceph-mon[91136]: pgmap v3855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:26 smithi078 ceph-mon[99959]: pgmap v3855: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:28 smithi146 ceph-mon[103278]: pgmap v3856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:28 smithi078 ceph-mon[91136]: pgmap v3856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:28 smithi078 ceph-mon[99959]: pgmap v3856: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:30.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:28:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:28:30.900 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:30.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:30.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:30.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:30.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:30.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:30 smithi146 ceph-mon[103278]: pgmap v3857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:30 smithi078 ceph-mon[91136]: pgmap v3857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:30 smithi078 ceph-mon[99959]: pgmap v3857: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:32 smithi146 ceph-mon[103278]: pgmap v3858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:32 smithi078 ceph-mon[91136]: pgmap v3858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:32 smithi078 ceph-mon[99959]: pgmap v3858: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:34.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:34.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:34.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:34.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:34 smithi146 ceph-mon[103278]: pgmap v3859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:34 smithi078 ceph-mon[91136]: pgmap v3859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:34 smithi078 ceph-mon[99959]: pgmap v3859: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:36 smithi146 ceph-mon[103278]: pgmap v3860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[99959]: pgmap v3860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[91136]: pgmap v3860: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:38 smithi146 ceph-mon[103278]: pgmap v3861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:38 smithi078 ceph-mon[91136]: pgmap v3861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:38.820 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:38 smithi078 ceph-mon[99959]: pgmap v3861: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:41.006 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:28:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:28:41.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:40.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:41.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:40.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:41.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:40.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:41 smithi146 ceph-mon[103278]: pgmap v3862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[99959]: pgmap v3862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[91136]: pgmap v3862: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:43 smithi146 ceph-mon[103278]: pgmap v3863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:43 smithi078 ceph-mon[91136]: pgmap v3863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:43 smithi078 ceph-mon[99959]: pgmap v3863: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:43.760 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:28:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:44.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:44.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:44.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 2h - - 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 2h - - 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 8s ago 2h 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 4s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 8s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 4s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 8s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 8s ago 2h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:28:44.319 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 4s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 8s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 8s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 4s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 8s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 8s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 8s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 8s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 4s ago 2h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (119m) 4s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (119m) 4s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:28:44.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (118m) 4s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:28:44.321 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 4s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:28:44.321 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (117m) 8s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:28:44.321 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (117m) 4s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:28:44.321 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (117m) 8s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:28:44.321 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (117m) 4s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:28:44.633 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:28:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:28:44.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:28:44.635 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:28:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:28:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:28:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:28:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:28:44.905 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:28:45.215 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:28:45.215 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:28:45.215 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[99959]: pgmap v3864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[99959]: from='client.72710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[99959]: from='client.72716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2689304124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[91136]: pgmap v3864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[91136]: from='client.72710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[91136]: from='client.72716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:45.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2689304124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:45 smithi146 ceph-mon[103278]: pgmap v3864: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:45 smithi146 ceph-mon[103278]: from='client.72710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:45 smithi146 ceph-mon[103278]: from='client.72716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2689304124' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:28:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:46 smithi078 ceph-mon[99959]: from='client.52803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:46 smithi078 ceph-mon[99959]: from='client.52815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1681296750' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:46 smithi078 ceph-mon[91136]: from='client.52803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:46 smithi078 ceph-mon[91136]: from='client.52815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1681296750' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:46 smithi146 ceph-mon[103278]: from='client.52803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:46 smithi146 ceph-mon[103278]: from='client.52815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:28:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1681296750' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:28:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:47 smithi078 ceph-mon[91136]: pgmap v3865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:47 smithi078 ceph-mon[99959]: pgmap v3865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:47 smithi146 ceph-mon[103278]: pgmap v3865: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:49 smithi078 ceph-mon[91136]: pgmap v3866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:49 smithi078 ceph-mon[99959]: pgmap v3866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:49 smithi146 ceph-mon[103278]: pgmap v3866: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:51.029 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:28:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:28:51.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:50.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:51.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:50.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:51.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:50.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:51 smithi078 ceph-mon[91136]: pgmap v3867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:51 smithi078 ceph-mon[99959]: pgmap v3867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:51.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:51 smithi146 ceph-mon[103278]: pgmap v3867: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:53 smithi078 ceph-mon[99959]: pgmap v3868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:53 smithi078 ceph-mon[91136]: pgmap v3868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:53 smithi146 ceph-mon[103278]: pgmap v3868: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:54.392 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:54 smithi078 ceph-mon[99959]: pgmap v3869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:54.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:54 smithi078 ceph-mon[91136]: pgmap v3869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:54.393 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:54.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:54.393 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:54.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:28:54.393 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:28:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:28:54.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:28:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:54 smithi146 ceph-mon[103278]: pgmap v3869: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:28:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:56 smithi146 ceph-mon[103278]: pgmap v3870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[91136]: pgmap v3870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[99959]: pgmap v3870: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:28:58.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:58 smithi146 ceph-mon[103278]: pgmap v3871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:58.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:58 smithi078 ceph-mon[91136]: pgmap v3871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:28:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:58 smithi078 ceph-mon[99959]: pgmap v3871: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:28:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:01.000 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:29:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[99959]: pgmap v3872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[91136]: pgmap v3872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.002 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:00.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:01.002 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:00.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:01.002 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:00.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:00 smithi146 ceph-mon[103278]: pgmap v3872: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:02 smithi146 ceph-mon[103278]: pgmap v3873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:02 smithi078 ceph-mon[91136]: pgmap v3873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:02 smithi078 ceph-mon[99959]: pgmap v3873: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:04.089Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:04.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:04.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:04 smithi146 ceph-mon[103278]: pgmap v3874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:04 smithi078 ceph-mon[91136]: pgmap v3874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:04 smithi078 ceph-mon[99959]: pgmap v3874: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:06 smithi146 ceph-mon[103278]: pgmap v3875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:06 smithi078 ceph-mon[91136]: pgmap v3875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:06 smithi078 ceph-mon[99959]: pgmap v3875: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:08 smithi146 ceph-mon[103278]: pgmap v3876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:08 smithi078 ceph-mon[91136]: pgmap v3876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:08 smithi078 ceph-mon[99959]: pgmap v3876: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:10.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:10 smithi078 ceph-mon[99959]: pgmap v3877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:10.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:10 smithi078 ceph-mon[91136]: pgmap v3877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:10.978 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:29:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:29:10.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:10.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:10.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:10.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:10.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:10.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:10 smithi146 ceph-mon[103278]: pgmap v3877: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:12 smithi146 ceph-mon[103278]: pgmap v3878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:12 smithi078 ceph-mon[91136]: pgmap v3878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:12 smithi078 ceph-mon[99959]: pgmap v3878: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:13.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:13.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:14.090Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:14.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:14.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:14.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:14.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:14 smithi146 ceph-mon[103278]: pgmap v3879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:14 smithi078 ceph-mon[91136]: pgmap v3879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:14 smithi078 ceph-mon[99959]: pgmap v3879: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:15.496 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:29:16.030 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 2h - - 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 2h - - 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 0s ago 2h 31.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 15s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 0s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 15s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 0s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 0s ago 2h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 15s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 0s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 0s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:29:16.031 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 15s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 0s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 0s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 0s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 0s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 15s ago 2h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 15s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (119m) 15s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (118m) 15s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 15s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (118m) 0s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (118m) 15s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (118m) 0s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:29:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (118m) 15s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:29:16.342 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:29:16.342 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:29:16.342 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:29:16.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:16.342 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:29:16.342 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:29:16.343 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:29:16.611 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:29:16.614 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:29:16.615 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:29:16.615 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:29:16.615 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:29:16.615 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:29:16.615 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:29:16.615 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:29:16.923 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:29:16.923 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:16.923 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:16.923 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:16.923 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:16.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:16.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: pgmap v3880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:16.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='client.52827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:16.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:16.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:16.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:16.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='client.52833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:16.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='client.52839 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:16.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3530711283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:16.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:16.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: pgmap v3880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:16.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='client.52827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:16.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:16.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:16.926 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:16.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='client.52833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:16.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='client.52839 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:16.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3530711283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: pgmap v3880: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='client.52827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='client.52833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='client.52839 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3530711283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:17 smithi146 ceph-mon[103278]: from='client.72770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1057221093' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:17 smithi078 ceph-mon[99959]: from='client.72770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1057221093' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:17 smithi078 ceph-mon[91136]: from='client.72770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1057221093' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:18 smithi146 ceph-mon[103278]: pgmap v3881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:18 smithi078 ceph-mon[99959]: pgmap v3881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:18 smithi078 ceph-mon[91136]: pgmap v3881: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:20.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[91136]: pgmap v3882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:20.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:20.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:20.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:20.815 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:29:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:29:20.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[99959]: pgmap v3882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:20.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:20.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:20.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:20.816 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:20.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:20.816 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:20.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:20.816 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:20.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:20 smithi146 ceph-mon[103278]: pgmap v3882: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:22 smithi146 ceph-mon[103278]: pgmap v3883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:22 smithi078 ceph-mon[91136]: pgmap v3883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:22 smithi078 ceph-mon[99959]: pgmap v3883: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:24.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:24.090Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:24.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:24.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:24.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:24 smithi146 ceph-mon[103278]: pgmap v3884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:24 smithi078 ceph-mon[91136]: pgmap v3884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:24 smithi078 ceph-mon[99959]: pgmap v3884: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:26 smithi146 ceph-mon[103278]: pgmap v3885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:26 smithi078 ceph-mon[91136]: pgmap v3885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:26 smithi078 ceph-mon[99959]: pgmap v3885: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:28 smithi146 ceph-mon[103278]: pgmap v3886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:28 smithi078 ceph-mon[91136]: pgmap v3886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:28 smithi078 ceph-mon[99959]: pgmap v3886: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:30 smithi146 ceph-mon[103278]: pgmap v3887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:30 smithi078 ceph-mon[91136]: pgmap v3887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:30 smithi078 ceph-mon[99959]: pgmap v3887: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:29:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:29:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:30.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:30.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:30.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:33.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:32 smithi146 ceph-mon[103278]: pgmap v3888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:32 smithi078 ceph-mon[91136]: pgmap v3888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:32 smithi078 ceph-mon[99959]: pgmap v3888: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:34.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:34.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:34.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:34 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:34.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:34 smithi078 ceph-mon[91136]: pgmap v3889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:34 smithi078 ceph-mon[99959]: pgmap v3889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:34 smithi146 ceph-mon[103278]: pgmap v3889: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:36 smithi146 ceph-mon[103278]: pgmap v3890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[91136]: pgmap v3890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[99959]: pgmap v3890: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:38 smithi146 ceph-mon[103278]: pgmap v3891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:38 smithi078 ceph-mon[91136]: pgmap v3891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:38 smithi078 ceph-mon[99959]: pgmap v3891: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:40.665 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:29:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:29:40.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:40 smithi146 ceph-mon[103278]: pgmap v3892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:40.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:40.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:40.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[99959]: pgmap v3892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[91136]: pgmap v3892: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:40.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:40.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:40.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:43.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:42 smithi078 ceph-mon[91136]: pgmap v3893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:43.175 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:42 smithi078 ceph-mon[99959]: pgmap v3893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:42 smithi146 ceph-mon[103278]: pgmap v3893: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:44.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:44.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:44.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:44 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:44.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:44 smithi146 ceph-mon[103278]: pgmap v3894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:44 smithi078 ceph-mon[99959]: pgmap v3894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:44 smithi078 ceph-mon[91136]: pgmap v3894: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:47.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:46 smithi078 ceph-mon[99959]: pgmap v3895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:47.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:47.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:46 smithi078 ceph-mon[91136]: pgmap v3895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:47.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:47.217 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:29:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:46 smithi146 ceph-mon[103278]: pgmap v3895: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 2h - - 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 2h - - 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 11s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 7s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 11s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 7s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:29:47.740 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 11s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 11s ago 2h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 7s ago 2h 163M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 11s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 11s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 7s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 11s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 11s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 11s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 11s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 7s ago 2h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 7s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 7s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (119m) 7s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:29:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 7s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:29:47.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (118m) 11s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:29:47.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (118m) 7s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:29:47.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (118m) 11s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:29:47.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (118m) 7s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:29:48.054 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:29:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:29:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:29:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:29:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:29:48.055 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:29:48.323 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:29:48.324 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:29:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:48 smithi146 ceph-mon[103278]: from='client.72782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:48 smithi146 ceph-mon[103278]: pgmap v3896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:48 smithi146 ceph-mon[103278]: from='client.72788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:48 smithi146 ceph-mon[103278]: from='client.52875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3388145296' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:49.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3106704711' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[91136]: from='client.72782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[91136]: pgmap v3896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[91136]: from='client.72788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[91136]: from='client.52875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3388145296' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3106704711' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[99959]: from='client.72782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[99959]: pgmap v3896: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[99959]: from='client.72788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[99959]: from='client.52875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3388145296' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:29:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3106704711' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:29:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:49 smithi146 ceph-mon[103278]: from='client.72806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:49 smithi078 ceph-mon[91136]: from='client.72806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:49 smithi078 ceph-mon[99959]: from='client.72806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:50.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:29:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:29:50.903 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:50.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:50.903 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:50.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:50.903 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:50.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:51.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:50 smithi146 ceph-mon[103278]: pgmap v3897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:50 smithi078 ceph-mon[91136]: pgmap v3897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:50 smithi078 ceph-mon[99959]: pgmap v3897: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:52 smithi146 ceph-mon[103278]: pgmap v3898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:52 smithi078 ceph-mon[99959]: pgmap v3898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:52 smithi078 ceph-mon[91136]: pgmap v3898: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:54.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:54.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:54.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:29:54.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:29:54 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:29:54.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:29:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:54 smithi146 ceph-mon[103278]: pgmap v3899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:54 smithi078 ceph-mon[91136]: pgmap v3899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:54 smithi078 ceph-mon[99959]: pgmap v3899: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:29:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:57 smithi146 ceph-mon[103278]: pgmap v3900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[91136]: pgmap v3900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[99959]: pgmap v3900: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:29:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:29:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:59 smithi146 ceph-mon[103278]: pgmap v3901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:29:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:59 smithi078 ceph-mon[99959]: pgmap v3901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:29:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:29:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:59 smithi078 ceph-mon[91136]: pgmap v3901: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:29:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:29:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:00.209 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T22:29:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:00.878 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:30:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:30:00.878 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:00.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:00.878 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:00.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:00.878 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:00.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:01 smithi146 ceph-mon[103278]: pgmap v3902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[91136]: pgmap v3902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[99959]: pgmap v3902: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:03 smithi146 ceph-mon[103278]: pgmap v3903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:03 smithi078 ceph-mon[91136]: pgmap v3903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:03 smithi078 ceph-mon[99959]: pgmap v3903: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:04.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:04.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:04.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:04.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:04 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:04.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:05 smithi078 ceph-mon[99959]: pgmap v3904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:05 smithi078 ceph-mon[91136]: pgmap v3904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:05 smithi146 ceph-mon[103278]: pgmap v3904: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:07 smithi078 ceph-mon[91136]: pgmap v3905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:07 smithi078 ceph-mon[99959]: pgmap v3905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:07 smithi146 ceph-mon[103278]: pgmap v3905: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:09 smithi078 ceph-mon[91136]: pgmap v3906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:09 smithi078 ceph-mon[99959]: pgmap v3906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:09 smithi146 ceph-mon[103278]: pgmap v3906: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:11.031 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:30:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:30:11.032 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:10.666Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:11.032 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:10.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:11.032 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:10.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:11 smithi078 ceph-mon[91136]: pgmap v3907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:11 smithi078 ceph-mon[99959]: pgmap v3907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:11 smithi146 ceph-mon[103278]: pgmap v3907: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:13 smithi078 ceph-mon[91136]: pgmap v3908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:13 smithi078 ceph-mon[99959]: pgmap v3908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:13 smithi146 ceph-mon[103278]: pgmap v3908: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:14.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:14.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:14.330 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:14.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:14.330 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:14.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:14.330 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:14 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:14.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:15.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:15 smithi078 ceph-mon[91136]: pgmap v3909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:15.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:15 smithi078 ceph-mon[99959]: pgmap v3909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:15 smithi146 ceph-mon[103278]: pgmap v3909: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:16 smithi146 ceph-mon[103278]: pgmap v3910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:16 smithi078 ceph-mon[99959]: pgmap v3910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:16 smithi078 ceph-mon[91136]: pgmap v3910: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:18 smithi146 ceph-mon[103278]: pgmap v3911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:18 smithi078 ceph-mon[91136]: pgmap v3911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:18 smithi078 ceph-mon[99959]: pgmap v3911: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:18.927 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:30:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:19 smithi078 ceph-mon[91136]: from='client.52899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:19 smithi078 ceph-mon[99959]: from='client.52899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:19.453 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:30:19.453 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 2h - - 2024-09-20T22:30:19.453 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 2h - - 2024-09-20T22:30:19.453 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 3s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:30:19.453 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 18s ago 2h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 3s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 18s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 3s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 3s ago 2h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 18s ago 2h 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 3s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 3s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 18s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 3s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 3s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 3s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 3s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:30:19.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 18s ago 2h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 18s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 18s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (119m) 18s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 18s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (119m) 3s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (119m) 18s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (119m) 3s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:30:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (119m) 18s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:30:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:19 smithi146 ceph-mon[103278]: from='client.52899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:30:19.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:30:19.765 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:30:20.032 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:30:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:30:20.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:30:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[99959]: from='client.52905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[99959]: from='client.72830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[99959]: pgmap v3912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/951807794' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[99959]: from='client.52923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[91136]: from='client.52905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[91136]: from='client.72830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[91136]: pgmap v3912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/951807794' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-mon[91136]: from='client.52923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.348 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:20.348 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:20.348 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:20.348 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:20.349 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:20 smithi146 ceph-mon[103278]: from='client.52905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:20 smithi146 ceph-mon[103278]: from='client.72830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:20 smithi146 ceph-mon[103278]: pgmap v3912: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/951807794' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:20 smithi146 ceph-mon[103278]: from='client.52923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:30:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:30:21.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:20.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:20.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:20.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:21.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3360732152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:21.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3360732152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:21.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3360732152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:21.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:21.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:22.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:22 smithi078 ceph-mon[91136]: pgmap v3913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:22.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:22 smithi078 ceph-mon[99959]: pgmap v3913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:22 smithi146 ceph-mon[103278]: pgmap v3913: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:24.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:24 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:24.095Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:24 smithi146 ceph-mon[103278]: pgmap v3914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:24 smithi078 ceph-mon[99959]: pgmap v3914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:24 smithi078 ceph-mon[91136]: pgmap v3914: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:26 smithi146 ceph-mon[103278]: pgmap v3915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:26 smithi078 ceph-mon[91136]: pgmap v3915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:26 smithi078 ceph-mon[99959]: pgmap v3915: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:28 smithi146 ceph-mon[103278]: pgmap v3916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:28 smithi078 ceph-mon[99959]: pgmap v3916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:28 smithi078 ceph-mon[91136]: pgmap v3916: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:30 smithi078 ceph-mon[91136]: pgmap v3917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:30 smithi078 ceph-mon[99959]: pgmap v3917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:30:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:30:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:30.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:30.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:30.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:30.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:30 smithi146 ceph-mon[103278]: pgmap v3917: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:32 smithi078 ceph-mon[91136]: pgmap v3918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:32 smithi078 ceph-mon[99959]: pgmap v3918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:32.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:32 smithi146 ceph-mon[103278]: pgmap v3918: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:34 smithi078 ceph-mon[91136]: pgmap v3919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:34 smithi078 ceph-mon[99959]: pgmap v3919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:34 smithi146 ceph-mon[103278]: pgmap v3919: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[91136]: pgmap v3920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[99959]: pgmap v3920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:36 smithi146 ceph-mon[103278]: pgmap v3920: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:38 smithi078 ceph-mon[91136]: pgmap v3921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:38 smithi078 ceph-mon[99959]: pgmap v3921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:38 smithi146 ceph-mon[103278]: pgmap v3921: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:40 smithi078 ceph-mon[91136]: pgmap v3922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:40 smithi078 ceph-mon[99959]: pgmap v3922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:30:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:30:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:40.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:40.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:40.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:40.806 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:40 smithi146 ceph-mon[103278]: pgmap v3922: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:42.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:42.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:42 smithi078 ceph-mon[91136]: pgmap v3923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:42 smithi078 ceph-mon[99959]: pgmap v3923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:43.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:42 smithi146 ceph-mon[103278]: pgmap v3923: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:43.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:43.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:44 smithi146 ceph-mon[103278]: pgmap v3924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:44 smithi078 ceph-mon[91136]: pgmap v3924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:44 smithi078 ceph-mon[99959]: pgmap v3924: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:46 smithi146 ceph-mon[103278]: pgmap v3925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:46 smithi078 ceph-mon[99959]: pgmap v3925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:46 smithi078 ceph-mon[91136]: pgmap v3925: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:48 smithi146 ceph-mon[103278]: pgmap v3926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:48 smithi078 ceph-mon[91136]: pgmap v3926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:48 smithi078 ceph-mon[99959]: pgmap v3926: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:50.647 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:30:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:50 smithi078 ceph-mon[99959]: pgmap v3927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:50 smithi078 ceph-mon[91136]: pgmap v3927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:30:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:30:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:50.669Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:50.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:30:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:30:50.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:30:51.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:50 smithi146 ceph-mon[103278]: pgmap v3927: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 2h - - 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 2h - - 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 14s ago 2h 30.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 10s ago 2h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:30:51.173 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 10s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 2h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 10s ago 2h 164M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 14s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 10s ago 2h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 2h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 2h 67.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 14s ago 2h 75.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 10s ago 2h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 10s ago 2h 67.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:30:51.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 10s ago 2h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:30:51.175 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 10s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:30:51.175 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 10s ago 2h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:30:51.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:30:51.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (119m) 10s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:30:51.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (119m) 14s ago 2h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:30:51.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (119m) 10s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:30:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:30:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:30:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:30:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:30:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:30:51.491 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:30:51.774 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:30:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:30:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:30:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:30:51.775 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:30:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:51 smithi078 ceph-mon[99959]: from='client.72854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:51 smithi078 ceph-mon[99959]: from='client.72860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1930519895' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:51 smithi078 ceph-mon[91136]: from='client.72854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:51 smithi078 ceph-mon[91136]: from='client.72860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1930519895' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:52.091 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:30:52.091 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:52.092 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:30:52.092 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:52.092 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:30:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:51 smithi146 ceph-mon[103278]: from='client.72854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:51 smithi146 ceph-mon[103278]: from='client.72860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1930519895' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:30:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:52 smithi146 ceph-mon[103278]: from='client.72866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:52 smithi146 ceph-mon[103278]: pgmap v3928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:52 smithi146 ceph-mon[103278]: from='client.72878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3388655763' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[99959]: from='client.72866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[99959]: pgmap v3928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[99959]: from='client.72878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3388655763' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[91136]: from='client.72866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[91136]: pgmap v3928: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[91136]: from='client.72878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3388655763' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:30:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:54 smithi146 ceph-mon[103278]: pgmap v3929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:30:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:54 smithi078 ceph-mon[91136]: pgmap v3929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:54 smithi078 ceph-mon[99959]: pgmap v3929: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:30:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:56 smithi146 ceph-mon[103278]: pgmap v3930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:30:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[99959]: pgmap v3930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:30:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[91136]: pgmap v3930: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:30:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:58 smithi146 ceph-mon[103278]: pgmap v3931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:30:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:58 smithi078 ceph-mon[91136]: pgmap v3931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:30:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:30:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:58 smithi078 ceph-mon[99959]: pgmap v3931: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-20T22:30:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:00 smithi078 ceph-mon[91136]: pgmap v3932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:31:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:00 smithi078 ceph-mon[99959]: pgmap v3932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:31:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:31:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:31:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:00.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:00.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:00.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:01.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:00 smithi146 ceph-mon[103278]: pgmap v3932: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:31:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:02 smithi146 ceph-mon[103278]: pgmap v3933: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T22:31:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:02 smithi078 ceph-mon[91136]: pgmap v3933: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T22:31:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:02 smithi078 ceph-mon[99959]: pgmap v3933: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T22:31:05.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:04 smithi078 ceph-mon[91136]: pgmap v3934: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:31:05.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:04 smithi078 ceph-mon[99959]: pgmap v3934: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:31:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:04 smithi146 ceph-mon[103278]: pgmap v3934: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T22:31:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:06 smithi146 ceph-mon[103278]: pgmap v3935: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-20T22:31:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:06 smithi078 ceph-mon[91136]: pgmap v3935: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-20T22:31:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:06 smithi078 ceph-mon[99959]: pgmap v3935: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-20T22:31:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:08 smithi146 ceph-mon[103278]: pgmap v3936: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T22:31:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:08 smithi078 ceph-mon[99959]: pgmap v3936: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T22:31:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:08 smithi078 ceph-mon[91136]: pgmap v3936: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-20T22:31:10.932 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:31:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:31:10.932 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:10.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:10.932 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:10.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:10.932 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:10.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:10 smithi146 ceph-mon[103278]: pgmap v3937: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-20T22:31:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:10 smithi078 ceph-mon[91136]: pgmap v3937: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-20T22:31:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:10 smithi078 ceph-mon[99959]: pgmap v3937: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-20T22:31:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:12 smithi146 ceph-mon[103278]: pgmap v3938: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 177 op/s 2024-09-20T22:31:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:12 smithi078 ceph-mon[91136]: pgmap v3938: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 177 op/s 2024-09-20T22:31:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:12 smithi078 ceph-mon[99959]: pgmap v3938: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 177 op/s 2024-09-20T22:31:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:14 smithi146 ceph-mon[103278]: pgmap v3939: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T22:31:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:14 smithi078 ceph-mon[91136]: pgmap v3939: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T22:31:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:14 smithi078 ceph-mon[99959]: pgmap v3939: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T22:31:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:16 smithi146 ceph-mon[103278]: pgmap v3940: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 175 op/s 2024-09-20T22:31:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[91136]: pgmap v3940: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 175 op/s 2024-09-20T22:31:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[99959]: pgmap v3940: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 175 op/s 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:18 smithi146 ceph-mon[103278]: pgmap v3941: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T22:31:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:18 smithi078 ceph-mon[91136]: pgmap v3941: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T22:31:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:18 smithi078 ceph-mon[99959]: pgmap v3941: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-20T22:31:20.954 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:31:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:31:20.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:20.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:20.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:20.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:20.955 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:20.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:20 smithi146 ceph-mon[103278]: pgmap v3942: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T22:31:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:20 smithi078 ceph-mon[91136]: pgmap v3942: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T22:31:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:20 smithi078 ceph-mon[99959]: pgmap v3942: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T22:31:22.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:22.384 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 2h - - 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 2h - - 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 6s ago 2h 30.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 1s ago 2h 91.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 6s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 1s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 6s ago 2h 568M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 6s ago 2h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 1s ago 2h 165M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:31:22.914 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 6s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 6s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 1s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 6s ago 2h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 6s ago 2h 77.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 6s ago 2h 66.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 6s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 1s ago 2h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 1s ago 2h 67.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 1s ago 2h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 1s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 1s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 6s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 6s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:31:22.915 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 1s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:31:23.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:23 smithi078 ceph-mon[91136]: pgmap v3943: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T22:31:23.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:23 smithi078 ceph-mon[99959]: pgmap v3943: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:31:23.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:31:23.233 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:31:23.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:23 smithi146 ceph-mon[103278]: pgmap v3943: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:31:23.508 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:31:23.509 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:31:23.509 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:31:23.824 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:31:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:23.825 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:23.825 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[91136]: from='client.72890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[91136]: from='client.72896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[91136]: from='client.72902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2953108254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/564099066' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[99959]: from='client.72890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[99959]: from='client.72896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[99959]: from='client.72902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2953108254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/564099066' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:24.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:24 smithi146 ceph-mon[103278]: from='client.72890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:24 smithi146 ceph-mon[103278]: from='client.72896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:24 smithi146 ceph-mon[103278]: from='client.72902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:24.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2953108254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:24.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/564099066' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:25 smithi078 ceph-mon[99959]: pgmap v3944: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:31:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:25 smithi078 ceph-mon[99959]: from='client.52995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:25 smithi078 ceph-mon[91136]: pgmap v3944: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:31:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:25 smithi078 ceph-mon[91136]: from='client.52995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:25 smithi146 ceph-mon[103278]: pgmap v3944: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T22:31:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:25 smithi146 ceph-mon[103278]: from='client.52995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:26 smithi078 ceph-mon[99959]: pgmap v3945: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:31:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:26 smithi078 ceph-mon[91136]: pgmap v3945: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:31:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:26 smithi146 ceph-mon[103278]: pgmap v3945: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:31:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:28 smithi078 ceph-mon[99959]: pgmap v3946: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:28 smithi078 ceph-mon[91136]: pgmap v3946: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:28 smithi146 ceph-mon[103278]: pgmap v3946: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:30 smithi146 ceph-mon[103278]: pgmap v3947: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:30 smithi078 ceph-mon[91136]: pgmap v3947: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:30 smithi078 ceph-mon[99959]: pgmap v3947: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:31:30] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:31:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:30.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:30.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:30.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:32 smithi146 ceph-mon[103278]: pgmap v3948: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:32 smithi078 ceph-mon[91136]: pgmap v3948: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:32 smithi078 ceph-mon[99959]: pgmap v3948: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:34 smithi146 ceph-mon[103278]: pgmap v3949: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:34 smithi078 ceph-mon[91136]: pgmap v3949: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:34 smithi078 ceph-mon[99959]: pgmap v3949: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:36 smithi146 ceph-mon[103278]: pgmap v3950: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:36 smithi078 ceph-mon[91136]: pgmap v3950: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:36 smithi078 ceph-mon[99959]: pgmap v3950: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:38 smithi146 ceph-mon[103278]: pgmap v3951: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:38 smithi078 ceph-mon[91136]: pgmap v3951: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:38 smithi078 ceph-mon[99959]: pgmap v3951: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:40 smithi146 ceph-mon[103278]: pgmap v3952: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:40 smithi078 ceph-mon[91136]: pgmap v3952: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:40 smithi078 ceph-mon[99959]: pgmap v3952: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:31:40] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:31:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:40.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:40.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:40.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:42.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:42 smithi146 ceph-mon[103278]: pgmap v3953: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[91136]: pgmap v3953: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:42 smithi078 ceph-mon[99959]: pgmap v3953: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:43.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:44.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:44.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:45.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:44 smithi078 ceph-mon[91136]: pgmap v3954: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:45.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:44 smithi078 ceph-mon[99959]: pgmap v3954: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:44 smithi146 ceph-mon[103278]: pgmap v3954: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:46.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:46 smithi078 ceph-mon[99959]: pgmap v3955: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:46.919 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:46.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:46 smithi078 ceph-mon[91136]: pgmap v3955: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:46.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:46 smithi146 ceph-mon[103278]: pgmap v3955: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:48 smithi146 ceph-mon[103278]: pgmap v3956: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:48 smithi078 ceph-mon[91136]: pgmap v3956: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:48 smithi078 ceph-mon[99959]: pgmap v3956: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:50 smithi078 ceph-mon[91136]: pgmap v3957: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:50 smithi078 ceph-mon[99959]: pgmap v3957: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:31:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:31:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:50.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:50.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:31:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:31:50.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:31:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:50 smithi146 ceph-mon[103278]: pgmap v3957: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:53.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:52 smithi146 ceph-mon[103278]: pgmap v3958: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:52 smithi078 ceph-mon[91136]: pgmap v3958: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:52 smithi078 ceph-mon[99959]: pgmap v3958: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:54.122 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 2h - - 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 2h - - 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 17s ago 2h 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 13s ago 2h 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:31:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 13s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 2h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 13s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 17s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 13s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 13s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 13s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 13s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 13s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:31:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 13s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:31:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:31:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:31:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:31:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 13s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:31:54.959 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:31:54.960 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:31:54.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:31:54.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:31:54.961 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:31:54.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:54 smithi078 ceph-mon[99959]: pgmap v3959: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:54.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:54 smithi078 ceph-mon[91136]: pgmap v3959: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:31:55.226 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:31:55.227 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:31:55.227 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:31:55.227 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:31:55.227 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:31:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:54 smithi146 ceph-mon[103278]: pgmap v3959: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:31:55.535 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:31:55.536 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:55.536 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:31:55.536 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:55.536 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:31:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:55 smithi146 ceph-mon[103278]: from='client.53007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:55 smithi146 ceph-mon[103278]: from='client.72932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:55 smithi146 ceph-mon[103278]: from='client.53019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3919414041' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/991924515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[91136]: from='client.53007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[91136]: from='client.72932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[91136]: from='client.53019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3919414041' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/991924515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[99959]: from='client.53007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[99959]: from='client.72932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[99959]: from='client.53019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3919414041' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:31:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/991924515' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:31:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:56 smithi146 ceph-mon[103278]: from='client.72950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:56 smithi146 ceph-mon[103278]: pgmap v3960: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[91136]: from='client.72950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[91136]: pgmap v3960: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[99959]: from='client.72950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[99959]: pgmap v3960: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:31:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:58 smithi146 ceph-mon[103278]: pgmap v3961: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:31:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:58 smithi078 ceph-mon[91136]: pgmap v3961: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:31:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:31:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:58 smithi078 ceph-mon[99959]: pgmap v3961: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:31:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:31:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:00 smithi078 ceph-mon[91136]: pgmap v3962: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:32:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:32:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:00 smithi078 ceph-mon[99959]: pgmap v3962: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:00.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:00.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:00.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:00 smithi146 ceph-mon[103278]: pgmap v3962: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:02.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:02 smithi146 ceph-mon[103278]: pgmap v3963: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[91136]: pgmap v3963: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:02 smithi078 ceph-mon[99959]: pgmap v3963: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:04.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:04 smithi078 ceph-mon[91136]: pgmap v3964: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:04.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:04 smithi078 ceph-mon[99959]: pgmap v3964: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:05.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:04 smithi146 ceph-mon[103278]: pgmap v3964: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:06.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:06 smithi078 ceph-mon[99959]: pgmap v3965: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:06.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:06.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:06 smithi078 ceph-mon[91136]: pgmap v3965: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:06.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:06.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:06 smithi146 ceph-mon[103278]: pgmap v3965: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:06.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:08 smithi146 ceph-mon[103278]: pgmap v3966: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:08 smithi078 ceph-mon[99959]: pgmap v3966: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:08 smithi078 ceph-mon[91136]: pgmap v3966: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:10 smithi146 ceph-mon[103278]: pgmap v3967: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:10 smithi078 ceph-mon[91136]: pgmap v3967: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:10 smithi078 ceph-mon[99959]: pgmap v3967: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:32:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:32:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:10.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:10.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:10.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:12 smithi146 ceph-mon[103278]: pgmap v3968: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:12 smithi078 ceph-mon[91136]: pgmap v3968: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:12 smithi078 ceph-mon[99959]: pgmap v3968: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:14.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:14 smithi078 ceph-mon[99959]: pgmap v3969: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:14.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:14 smithi078 ceph-mon[91136]: pgmap v3969: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:14 smithi146 ceph-mon[103278]: pgmap v3969: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:16 smithi146 ceph-mon[103278]: pgmap v3970: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:32:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:16 smithi078 ceph-mon[91136]: pgmap v3970: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:32:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:16 smithi078 ceph-mon[99959]: pgmap v3970: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:32:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:18 smithi146 ceph-mon[103278]: pgmap v3971: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:32:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:18 smithi078 ceph-mon[91136]: pgmap v3971: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:32:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:18 smithi078 ceph-mon[99959]: pgmap v3971: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:32:20.921 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:32:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:32:20.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:20.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:20.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:20.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:20.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:20.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:20 smithi146 ceph-mon[103278]: pgmap v3972: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:20 smithi078 ceph-mon[91136]: pgmap v3972: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:20 smithi078 ceph-mon[99959]: pgmap v3972: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:22 smithi146 ceph-mon[103278]: pgmap v3973: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[91136]: pgmap v3973: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:22 smithi078 ceph-mon[99959]: pgmap v3973: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:24.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:24 smithi146 ceph-mon[103278]: pgmap v3974: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:24 smithi078 ceph-mon[91136]: pgmap v3974: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:24 smithi078 ceph-mon[99959]: pgmap v3974: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:25.823 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:32:26.357 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:32:26.357 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 2h - - 2024-09-20T22:32:26.357 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 2h - - 2024-09-20T22:32:26.357 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 9s ago 2h 30.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:32:26.357 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 4s ago 2h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:32:26.357 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 9s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 4s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 9s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 9s ago 2h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 4s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 9s ago 2h 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 9s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 4s ago 2h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 9s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 9s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 9s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 9s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:32:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 4s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 4s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 4s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 4s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 4s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 9s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 9s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:32:26.359 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 4s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:32:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:32:26.682 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:32:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:26 smithi078 ceph-mon[99959]: pgmap v3975: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:26.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:26 smithi078 ceph-mon[99959]: from='client.72962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:26.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:26 smithi078 ceph-mon[91136]: pgmap v3975: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:26.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:26.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:26 smithi078 ceph-mon[91136]: from='client.72962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:32:26.963 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:32:26.964 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:32:26.964 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:32:26.964 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:32:26.964 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:32:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:26 smithi146 ceph-mon[103278]: pgmap v3975: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:26 smithi146 ceph-mon[103278]: from='client.72962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.292 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:32:27.292 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:27.292 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:27.292 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:27.293 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[91136]: from='client.72968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[91136]: from='client.72974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1233869599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[91136]: from='client.53067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3668613686' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[99959]: from='client.72968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[99959]: from='client.72974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1233869599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:27.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[99959]: from='client.53067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:27.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3668613686' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:27 smithi146 ceph-mon[103278]: from='client.72968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:27 smithi146 ceph-mon[103278]: from='client.72974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1233869599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:27 smithi146 ceph-mon[103278]: from='client.53067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3668613686' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:28 smithi146 ceph-mon[103278]: pgmap v3976: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:28 smithi078 ceph-mon[91136]: pgmap v3976: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:28 smithi078 ceph-mon[99959]: pgmap v3976: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:30 smithi078 ceph-mon[91136]: pgmap v3977: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:30 smithi078 ceph-mon[99959]: pgmap v3977: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:32:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:32:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:30.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:30.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:30.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:30 smithi146 ceph-mon[103278]: pgmap v3977: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:32 smithi146 ceph-mon[103278]: pgmap v3978: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:32 smithi078 ceph-mon[91136]: pgmap v3978: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:32 smithi078 ceph-mon[99959]: pgmap v3978: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:34 smithi146 ceph-mon[103278]: pgmap v3979: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:34 smithi078 ceph-mon[91136]: pgmap v3979: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:34 smithi078 ceph-mon[99959]: pgmap v3979: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:36 smithi146 ceph-mon[103278]: pgmap v3980: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:36 smithi078 ceph-mon[91136]: pgmap v3980: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:36 smithi078 ceph-mon[99959]: pgmap v3980: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:39 smithi146 ceph-mon[103278]: pgmap v3981: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:39 smithi078 ceph-mon[91136]: pgmap v3981: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:39 smithi078 ceph-mon[99959]: pgmap v3981: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:32:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:32:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:40.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:40.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:40.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:41 smithi146 ceph-mon[103278]: pgmap v3982: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:41 smithi078 ceph-mon[91136]: pgmap v3982: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:41 smithi078 ceph-mon[99959]: pgmap v3982: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:42 smithi146 ceph-mon[103278]: pgmap v3983: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[99959]: pgmap v3983: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[91136]: pgmap v3983: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:43.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:44.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:44 smithi146 ceph-mon[103278]: pgmap v3984: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:44.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:44 smithi078 ceph-mon[99959]: pgmap v3984: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:44.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:44 smithi078 ceph-mon[91136]: pgmap v3984: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:46 smithi078 ceph-mon[91136]: pgmap v3985: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:46 smithi078 ceph-mon[99959]: pgmap v3985: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:46.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:46 smithi146 ceph-mon[103278]: pgmap v3985: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:46.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:48 smithi146 ceph-mon[103278]: pgmap v3986: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:48 smithi078 ceph-mon[91136]: pgmap v3986: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:48 smithi078 ceph-mon[99959]: pgmap v3986: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:50 smithi146 ceph-mon[103278]: pgmap v3987: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:50 smithi078 ceph-mon[91136]: pgmap v3987: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:50 smithi078 ceph-mon[99959]: pgmap v3987: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:32:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:32:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:50.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:50.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:32:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:32:50.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:32:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:52 smithi146 ceph-mon[103278]: pgmap v3988: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:52 smithi078 ceph-mon[99959]: pgmap v3988: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:52 smithi078 ceph-mon[91136]: pgmap v3988: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:54 smithi146 ceph-mon[103278]: pgmap v3989: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:55.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:54 smithi078 ceph-mon[91136]: pgmap v3989: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:55.092 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:54 smithi078 ceph-mon[99959]: pgmap v3989: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:32:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:56 smithi146 ceph-mon[103278]: pgmap v3990: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:56 smithi078 ceph-mon[91136]: pgmap v3990: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:56 smithi078 ceph-mon[99959]: pgmap v3990: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:32:57.577 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:32:58.107 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:32:58.107 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 2h - - 2024-09-20T22:32:58.107 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 2h - - 2024-09-20T22:32:58.107 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 0s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:32:58.107 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 16s ago 2h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 0s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 16s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 0s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 0s ago 2h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 16s ago 2h 166M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 0s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 0s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 16s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 0s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 0s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 0s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 0s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 16s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 16s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:32:58.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 16s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:32:58.109 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 16s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:32:58.109 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 16s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:32:58.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 0s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:32:58.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 16s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:32:58.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 0s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:32:58.109 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 16s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:58.423 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:32:58.424 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:32:58.424 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.424 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.424 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[91136]: pgmap v3991: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[91136]: from='client.53079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[91136]: from='client.53085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[99959]: pgmap v3991: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[99959]: from='client.53079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:58 smithi078 ceph-mon[99959]: from='client.53085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:32:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:58 smithi146 ceph-mon[103278]: pgmap v3991: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:32:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:58 smithi146 ceph-mon[103278]: from='client.53079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:58 smithi146 ceph-mon[103278]: from='client.53085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:58.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:32:58.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:32:59.003 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:32:59.003 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:32:59.004 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:32:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[91136]: from='client.53091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1570354910' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[91136]: from='client.53103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1712823368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[99959]: from='client.53091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1570354910' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[99959]: from='client.53103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:32:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1712823368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:32:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:59 smithi146 ceph-mon[103278]: from='client.53091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:32:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1570354910' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:32:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:59 smithi146 ceph-mon[103278]: from='client.53103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:32:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1712823368' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:00 smithi146 ceph-mon[103278]: pgmap v3992: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:00 smithi078 ceph-mon[99959]: pgmap v3992: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:00 smithi078 ceph-mon[91136]: pgmap v3992: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:33:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:33:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:00.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:00.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:00.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:02 smithi146 ceph-mon[103278]: pgmap v3993: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[91136]: pgmap v3993: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[99959]: pgmap v3993: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:04 smithi146 ceph-mon[103278]: pgmap v3994: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:04 smithi078 ceph-mon[91136]: pgmap v3994: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:04 smithi078 ceph-mon[99959]: pgmap v3994: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:06.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:06 smithi146 ceph-mon[103278]: pgmap v3995: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:06.787 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:06 smithi078 ceph-mon[91136]: pgmap v3995: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:06 smithi078 ceph-mon[99959]: pgmap v3995: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:08.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:08 smithi078 ceph-mon[91136]: pgmap v3996: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:08.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:08 smithi078 ceph-mon[99959]: pgmap v3996: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:08 smithi146 ceph-mon[103278]: pgmap v3996: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:10 smithi146 ceph-mon[103278]: pgmap v3997: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:10 smithi078 ceph-mon[91136]: pgmap v3997: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:10 smithi078 ceph-mon[99959]: pgmap v3997: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:33:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:33:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:10.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:10.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:10.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:12 smithi146 ceph-mon[103278]: pgmap v3998: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:12 smithi078 ceph-mon[91136]: pgmap v3998: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:12 smithi078 ceph-mon[99959]: pgmap v3998: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:14 smithi146 ceph-mon[103278]: pgmap v3999: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:14 smithi078 ceph-mon[99959]: pgmap v3999: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:14 smithi078 ceph-mon[91136]: pgmap v3999: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:16 smithi146 ceph-mon[103278]: pgmap v4000: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:16 smithi078 ceph-mon[91136]: pgmap v4000: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:16 smithi078 ceph-mon[99959]: pgmap v4000: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:18 smithi146 ceph-mon[103278]: pgmap v4001: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[91136]: pgmap v4001: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:18 smithi078 ceph-mon[99959]: pgmap v4001: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:20 smithi146 ceph-mon[103278]: pgmap v4002: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:20 smithi078 ceph-mon[99959]: pgmap v4002: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:20 smithi078 ceph-mon[91136]: pgmap v4002: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:33:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:33:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:20.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:20.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:20.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:23.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:22 smithi146 ceph-mon[103278]: pgmap v4003: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:23.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[99959]: pgmap v4003: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[91136]: pgmap v4003: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:25.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:24 smithi146 ceph-mon[103278]: pgmap v4004: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:24 smithi078 ceph-mon[99959]: pgmap v4004: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:24 smithi078 ceph-mon[91136]: pgmap v4004: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:26 smithi078 ceph-mon[91136]: pgmap v4005: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:27.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:27.091 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:26 smithi078 ceph-mon[99959]: pgmap v4005: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:27.091 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:26 smithi146 ceph-mon[103278]: pgmap v4005: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:29.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:28 smithi078 ceph-mon[91136]: pgmap v4006: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:29.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:29.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:28 smithi078 ceph-mon[99959]: pgmap v4006: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:29.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:28 smithi146 ceph-mon[103278]: pgmap v4006: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:29.295 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:33:29.833 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:33:29.833 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 2h - - 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 12s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 12s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 12s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 12s ago 2h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 8s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 12s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 12s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 12s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:33:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 12s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 12s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 12s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 8s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 12s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 12s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:33:29.835 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:33:30.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:33:30.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:33:30.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:33:30.145 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:33:30.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:33:30.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:33:30.145 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:33:30.413 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:33:30.413 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:33:30.414 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:33:30.680 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:33:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:33:30.726 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:33:30.726 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:33:30.726 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:33:30.726 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:33:30.726 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:33:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[91136]: from='client.73034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[91136]: pgmap v4007: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[91136]: from='client.53121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[91136]: from='client.53127 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3214377663' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3939876042' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:30.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:30.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:30.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[99959]: from='client.73034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[99959]: pgmap v4007: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[99959]: from='client.53121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[99959]: from='client.53127 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3214377663' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3939876042' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:30 smithi146 ceph-mon[103278]: from='client.73034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:30 smithi146 ceph-mon[103278]: pgmap v4007: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:30 smithi146 ceph-mon[103278]: from='client.53121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:30 smithi146 ceph-mon[103278]: from='client.53127 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3214377663' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:33:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3939876042' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:33:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:31 smithi146 ceph-mon[103278]: from='client.53139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:31 smithi078 ceph-mon[91136]: from='client.53139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:31 smithi078 ceph-mon[99959]: from='client.53139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:32 smithi146 ceph-mon[103278]: pgmap v4008: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:32 smithi078 ceph-mon[91136]: pgmap v4008: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:32 smithi078 ceph-mon[99959]: pgmap v4008: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:34 smithi146 ceph-mon[103278]: pgmap v4009: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:34 smithi078 ceph-mon[91136]: pgmap v4009: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:34 smithi078 ceph-mon[99959]: pgmap v4009: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:36 smithi146 ceph-mon[103278]: pgmap v4010: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:36 smithi078 ceph-mon[91136]: pgmap v4010: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:36 smithi078 ceph-mon[99959]: pgmap v4010: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:38 smithi146 ceph-mon[103278]: pgmap v4011: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[99959]: pgmap v4011: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:38 smithi078 ceph-mon[91136]: pgmap v4011: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:40 smithi146 ceph-mon[103278]: pgmap v4012: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:40 smithi078 ceph-mon[91136]: pgmap v4012: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:40 smithi078 ceph-mon[99959]: pgmap v4012: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:33:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:33:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:40.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:40.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:40.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:42 smithi146 ceph-mon[103278]: pgmap v4013: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[99959]: pgmap v4013: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[91136]: pgmap v4013: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:44 smithi146 ceph-mon[103278]: pgmap v4014: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:44 smithi078 ceph-mon[91136]: pgmap v4014: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:44 smithi078 ceph-mon[99959]: pgmap v4014: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:46 smithi146 ceph-mon[103278]: pgmap v4015: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:46 smithi078 ceph-mon[91136]: pgmap v4015: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:46 smithi078 ceph-mon[99959]: pgmap v4015: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:49.078 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:48 smithi078 ceph-mon[91136]: pgmap v4016: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:49.078 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:48 smithi078 ceph-mon[99959]: pgmap v4016: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:48 smithi146 ceph-mon[103278]: pgmap v4016: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:50.967 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:33:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:33:50.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:50.680Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:50.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:50.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:33:50.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:33:50.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:33:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:50 smithi146 ceph-mon[103278]: pgmap v4017: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:50 smithi078 ceph-mon[91136]: pgmap v4017: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:50 smithi078 ceph-mon[99959]: pgmap v4017: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:52 smithi146 ceph-mon[103278]: pgmap v4018: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:52 smithi078 ceph-mon[91136]: pgmap v4018: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:52 smithi078 ceph-mon[99959]: pgmap v4018: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:55.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:54 smithi146 ceph-mon[103278]: pgmap v4019: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:54 smithi078 ceph-mon[99959]: pgmap v4019: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:54 smithi078 ceph-mon[91136]: pgmap v4019: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:33:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:56 smithi146 ceph-mon[103278]: pgmap v4020: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:56 smithi078 ceph-mon[99959]: pgmap v4020: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:56 smithi078 ceph-mon[91136]: pgmap v4020: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:33:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:58 smithi146 ceph-mon[103278]: pgmap v4021: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[91136]: pgmap v4021: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[99959]: pgmap v4021: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:33:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:00 smithi146 ceph-mon[103278]: pgmap v4022: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:01.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:00 smithi078 ceph-mon[99959]: pgmap v4022: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:01.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:00 smithi078 ceph-mon[91136]: pgmap v4022: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:01.019 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:34:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:34:01.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:00.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:01.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:00.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:01.020 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:00.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:01.020 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:34:01.546 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 2h - - 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 2h - - 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 3s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 19s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 3s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 19s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 3s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 3s ago 2h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 19s ago 2h 167M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 3s ago 2h 174M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 3s ago 2h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 19s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:34:01.547 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 3s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 3s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 3s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 3s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 19s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 19s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 19s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 19s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 19s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 3s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 3s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:34:01.548 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 19s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:34:01.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:01 smithi078 ceph-mon[91136]: from='client.53151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:01.859 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:01 smithi078 ceph-mon[99959]: from='client.53151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:01.859 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:34:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:34:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:34:01.861 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:34:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:01 smithi146 ceph-mon[103278]: from='client.53151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:02.133 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:34:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:34:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:34:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:34:02.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:34:02.444 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:34:02.445 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:02.445 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:02.445 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:02.445 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='client.73076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: pgmap v4023: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='client.73082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2278214421' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3203276747' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='client.73076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: pgmap v4023: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='client.73082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:02.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2278214421' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:02.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:02.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:02.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:02.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3203276747' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='client.73076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: pgmap v4023: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='client.73082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2278214421' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3203276747' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:03 smithi146 ceph-mon[103278]: from='client.53175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:03 smithi078 ceph-mon[91136]: from='client.53175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:03 smithi078 ceph-mon[99959]: from='client.53175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:04 smithi146 ceph-mon[103278]: pgmap v4024: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:04 smithi078 ceph-mon[91136]: pgmap v4024: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:04 smithi078 ceph-mon[99959]: pgmap v4024: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:06 smithi146 ceph-mon[103278]: pgmap v4025: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:06 smithi078 ceph-mon[99959]: pgmap v4025: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:06 smithi078 ceph-mon[91136]: pgmap v4025: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:08.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:08 smithi078 ceph-mon[91136]: pgmap v4026: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:08.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:08 smithi078 ceph-mon[99959]: pgmap v4026: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:08.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:08 smithi146 ceph-mon[103278]: pgmap v4026: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:10.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:10 smithi078 ceph-mon[91136]: pgmap v4027: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:10.993 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:34:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:34:10.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:10 smithi078 ceph-mon[99959]: pgmap v4027: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:10.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:10.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:10.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:10.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:10.994 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:10.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:10 smithi146 ceph-mon[103278]: pgmap v4027: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:12 smithi146 ceph-mon[103278]: pgmap v4028: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:12 smithi078 ceph-mon[91136]: pgmap v4028: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:12 smithi078 ceph-mon[99959]: pgmap v4028: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:34:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:14.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:14 smithi078 ceph-mon[91136]: pgmap v4029: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:14.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:14 smithi078 ceph-mon[99959]: pgmap v4029: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:14 smithi146 ceph-mon[103278]: pgmap v4029: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:16 smithi146 ceph-mon[103278]: pgmap v4030: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:16 smithi078 ceph-mon[91136]: pgmap v4030: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:16 smithi078 ceph-mon[99959]: pgmap v4030: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:18 smithi146 ceph-mon[103278]: pgmap v4031: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[91136]: pgmap v4031: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[99959]: pgmap v4031: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:20 smithi146 ceph-mon[103278]: pgmap v4032: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:20 smithi078 ceph-mon[91136]: pgmap v4032: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:20 smithi078 ceph-mon[99959]: pgmap v4032: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:34:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:34:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:20.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:20.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:20.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:22 smithi146 ceph-mon[103278]: pgmap v4033: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[91136]: pgmap v4033: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[99959]: pgmap v4033: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:24 smithi146 ceph-mon[103278]: pgmap v4034: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:24 smithi078 ceph-mon[91136]: pgmap v4034: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:24 smithi078 ceph-mon[99959]: pgmap v4034: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 22:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T22:34:25.84721544Z level=info msg="Completed cleanup jobs" duration=12.000883ms 2024-09-20T22:34:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:26 smithi146 ceph-mon[103278]: pgmap v4035: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:26 smithi078 ceph-mon[91136]: pgmap v4035: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:26 smithi078 ceph-mon[99959]: pgmap v4035: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:28.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:28 smithi146 ceph-mon[103278]: pgmap v4036: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:28.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:28 smithi078 ceph-mon[91136]: pgmap v4036: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:28 smithi078 ceph-mon[99959]: pgmap v4036: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:30.684 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:34:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:34:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:30 smithi146 ceph-mon[103278]: pgmap v4037: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:30 smithi078 ceph-mon[91136]: pgmap v4037: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:30 smithi078 ceph-mon[99959]: pgmap v4037: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:30.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:30.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:30.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:32.733 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:34:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:32 smithi146 ceph-mon[103278]: pgmap v4038: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:32 smithi078 ceph-mon[91136]: pgmap v4038: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:32 smithi078 ceph-mon[99959]: pgmap v4038: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:33.255 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:34:33.255 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 2h - - 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 2h - - 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 15s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 11s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 15s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 11s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 15s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 15s ago 2h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 11s ago 2h 168M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 15s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 15s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 11s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 15s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 15s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 15s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:34:33.256 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 15s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 11s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 11s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 11s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 11s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 11s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 15s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 15s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:34:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 11s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:34:33.563 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:34:33.564 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:34:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:34:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:34:33.835 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:34:33.835 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:33 smithi078 ceph-mon[99959]: from='client.53187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:33.835 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:33 smithi078 ceph-mon[99959]: from='client.73112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:33.835 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2224254566' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:33.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:33 smithi078 ceph-mon[91136]: from='client.53187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:33.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:33 smithi078 ceph-mon[91136]: from='client.73112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:33.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2224254566' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:33 smithi146 ceph-mon[103278]: from='client.53187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:33 smithi146 ceph-mon[103278]: from='client.73112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2224254566' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:34:34.141 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:34:34.141 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:34.141 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:34:34.141 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:34.141 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:34:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:34 smithi146 ceph-mon[103278]: from='client.53199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:34 smithi146 ceph-mon[103278]: pgmap v4039: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:34 smithi146 ceph-mon[103278]: from='client.73130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1340080682' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[91136]: from='client.53199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[91136]: pgmap v4039: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[91136]: from='client.73130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1340080682' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[99959]: from='client.53199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[99959]: pgmap v4039: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[99959]: from='client.73130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1340080682' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:34:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:36 smithi146 ceph-mon[103278]: pgmap v4040: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:36 smithi078 ceph-mon[91136]: pgmap v4040: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:36 smithi078 ceph-mon[99959]: pgmap v4040: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:38 smithi146 ceph-mon[103278]: pgmap v4041: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[91136]: pgmap v4041: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[99959]: pgmap v4041: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:40 smithi078 ceph-mon[91136]: pgmap v4042: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:40 smithi078 ceph-mon[99959]: pgmap v4042: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:41.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:34:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:34:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:40.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:40.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:40.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:40 smithi146 ceph-mon[103278]: pgmap v4042: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:42 smithi146 ceph-mon[103278]: pgmap v4043: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[99959]: pgmap v4043: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[91136]: pgmap v4043: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:44.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:44.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:44 smithi078 ceph-mon[91136]: pgmap v4044: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:44 smithi078 ceph-mon[99959]: pgmap v4044: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:44 smithi146 ceph-mon[103278]: pgmap v4044: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:47.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:46 smithi146 ceph-mon[103278]: pgmap v4045: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:47.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:46 smithi078 ceph-mon[91136]: pgmap v4045: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:46 smithi078 ceph-mon[99959]: pgmap v4045: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:49.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:48 smithi146 ceph-mon[103278]: pgmap v4046: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:48 smithi078 ceph-mon[91136]: pgmap v4046: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:48 smithi078 ceph-mon[99959]: pgmap v4046: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:50 smithi078 ceph-mon[91136]: pgmap v4047: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:50 smithi078 ceph-mon[99959]: pgmap v4047: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:34:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:34:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:50.684Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:50.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:34:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:34:50.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:34:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:50 smithi146 ceph-mon[103278]: pgmap v4047: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:52 smithi078 ceph-mon[91136]: pgmap v4048: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:52 smithi078 ceph-mon[99959]: pgmap v4048: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:52 smithi146 ceph-mon[103278]: pgmap v4048: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:54 smithi146 ceph-mon[103278]: pgmap v4049: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:54 smithi078 ceph-mon[91136]: pgmap v4049: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:54 smithi078 ceph-mon[99959]: pgmap v4049: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:34:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:56 smithi146 ceph-mon[103278]: pgmap v4050: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:56 smithi078 ceph-mon[99959]: pgmap v4050: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:56 smithi078 ceph-mon[91136]: pgmap v4050: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:34:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:58 smithi146 ceph-mon[103278]: pgmap v4051: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[91136]: pgmap v4051: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:34:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[99959]: pgmap v4051: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:34:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:34:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:00.941 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:35:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:35:00.941 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:00.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:00.942 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:00.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:00.942 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:00.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:00 smithi146 ceph-mon[103278]: pgmap v4052: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:00 smithi078 ceph-mon[91136]: pgmap v4052: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:00 smithi078 ceph-mon[99959]: pgmap v4052: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:02 smithi146 ceph-mon[103278]: pgmap v4053: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[91136]: pgmap v4053: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[99959]: pgmap v4053: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:04.434 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 2h - - 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 2h - - 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 2s ago 2h 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 7s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:35:04.961 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 2s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 7s ago 2h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 2s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 2s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 2s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 2s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:35:04.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 2s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:35:04.963 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 2s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:35:04.963 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 2s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:35:04.963 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:35:04.963 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:35:04.963 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:35:04.963 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 2s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:35:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:04 smithi146 ceph-mon[103278]: pgmap v4054: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:05.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:04 smithi078 ceph-mon[99959]: pgmap v4054: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:05.269 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:35:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:35:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:35:05.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:35:05.270 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:35:05.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:04 smithi078 ceph-mon[91136]: pgmap v4054: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:35:05.540 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:35:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:35:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:35:05.541 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:35:05.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:35:05.849 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:35:05.850 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:35:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:35:05.850 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:05.850 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:05 smithi146 ceph-mon[103278]: from='client.53223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:05 smithi146 ceph-mon[103278]: from='client.53229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:05 smithi146 ceph-mon[103278]: from='client.73154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/654899225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3996240911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[99959]: from='client.53223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[99959]: from='client.53229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[99959]: from='client.73154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/654899225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3996240911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[91136]: from='client.53223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[91136]: from='client.53229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[91136]: from='client.73154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/654899225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3996240911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:06 smithi146 ceph-mon[103278]: from='client.73166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:06 smithi146 ceph-mon[103278]: pgmap v4055: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:06 smithi078 ceph-mon[91136]: from='client.73166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:06 smithi078 ceph-mon[91136]: pgmap v4055: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:06 smithi078 ceph-mon[99959]: from='client.73166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:06 smithi078 ceph-mon[99959]: pgmap v4055: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:08.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:08 smithi146 ceph-mon[103278]: pgmap v4056: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:08 smithi078 ceph-mon[91136]: pgmap v4056: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:08 smithi078 ceph-mon[99959]: pgmap v4056: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:10.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:35:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:35:10.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:10.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:10.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:10.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:10.898 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:10.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:10 smithi146 ceph-mon[103278]: pgmap v4057: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:10 smithi078 ceph-mon[91136]: pgmap v4057: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:10 smithi078 ceph-mon[99959]: pgmap v4057: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:12 smithi146 ceph-mon[103278]: pgmap v4058: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:12 smithi078 ceph-mon[91136]: pgmap v4058: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:12 smithi078 ceph-mon[99959]: pgmap v4058: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:14 smithi146 ceph-mon[103278]: pgmap v4059: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:14 smithi078 ceph-mon[91136]: pgmap v4059: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:14 smithi078 ceph-mon[99959]: pgmap v4059: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:16 smithi146 ceph-mon[103278]: pgmap v4060: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:16 smithi078 ceph-mon[91136]: pgmap v4060: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:16 smithi078 ceph-mon[99959]: pgmap v4060: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:19 smithi146 ceph-mon[103278]: pgmap v4061: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[91136]: pgmap v4061: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[99959]: pgmap v4061: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:35:20] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:35:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:20.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:20.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:20.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:21 smithi146 ceph-mon[103278]: pgmap v4062: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:21 smithi078 ceph-mon[91136]: pgmap v4062: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:21 smithi078 ceph-mon[99959]: pgmap v4062: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:23 smithi146 ceph-mon[103278]: pgmap v4063: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[91136]: pgmap v4063: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[99959]: pgmap v4063: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:23.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:25 smithi078 ceph-mon[91136]: pgmap v4064: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:25 smithi078 ceph-mon[99959]: pgmap v4064: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:25 smithi146 ceph-mon[103278]: pgmap v4064: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:27 smithi078 ceph-mon[99959]: pgmap v4065: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:27 smithi078 ceph-mon[91136]: pgmap v4065: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:27 smithi146 ceph-mon[103278]: pgmap v4065: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:28.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:28 smithi146 ceph-mon[103278]: pgmap v4066: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:28 smithi078 ceph-mon[91136]: pgmap v4066: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:28 smithi078 ceph-mon[99959]: pgmap v4066: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:30 smithi146 ceph-mon[103278]: pgmap v4067: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:30 smithi078 ceph-mon[91136]: pgmap v4067: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:30 smithi078 ceph-mon[99959]: pgmap v4067: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:35:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:35:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:30.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:30.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:30.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:32.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:32 smithi078 ceph-mon[99959]: pgmap v4068: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:32.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:32 smithi078 ceph-mon[91136]: pgmap v4068: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:32 smithi146 ceph-mon[103278]: pgmap v4068: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:34 smithi146 ceph-mon[103278]: pgmap v4069: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:34 smithi078 ceph-mon[91136]: pgmap v4069: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:34 smithi078 ceph-mon[99959]: pgmap v4069: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:36.145 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:35:36.671 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:35:36.671 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 2h - - 2024-09-20T22:35:36.671 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 2h - - 2024-09-20T22:35:36.671 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 18s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:35:36.671 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 14s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 18s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 14s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 18s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 18s ago 2h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 14s ago 2h 169M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 18s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 18s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 14s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 18s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 18s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 18s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 18s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 14s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 14s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 14s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:35:36.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 14s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:35:36.673 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 14s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:35:36.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 18s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:35:36.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:35:36.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 18s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:35:36.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 14s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:35:36.980 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:35:36.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:35:36.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:35:36.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:36 smithi078 ceph-mon[91136]: pgmap v4070: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:36.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:36.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:36 smithi078 ceph-mon[99959]: pgmap v4070: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:36.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:36 smithi146 ceph-mon[103278]: pgmap v4070: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:37.254 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:35:37.254 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:35:37.254 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:35:37.255 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:35:37.565 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:35:37.565 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:35:37.565 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:35:37.565 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:37.565 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:35:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:37 smithi146 ceph-mon[103278]: from='client.73178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:37 smithi146 ceph-mon[103278]: from='client.53265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:37 smithi146 ceph-mon[103278]: from='client.53271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4099665755' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3450488206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[91136]: from='client.73178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[91136]: from='client.53265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[91136]: from='client.53271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4099665755' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3450488206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[99959]: from='client.73178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[99959]: from='client.53265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[99959]: from='client.53271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4099665755' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:35:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3450488206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:35:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:38 smithi146 ceph-mon[103278]: from='client.53283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:38 smithi146 ceph-mon[103278]: pgmap v4071: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[91136]: from='client.53283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[91136]: pgmap v4071: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[99959]: from='client.53283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[99959]: pgmap v4071: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:40 smithi146 ceph-mon[103278]: pgmap v4072: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:40 smithi078 ceph-mon[91136]: pgmap v4072: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:40 smithi078 ceph-mon[99959]: pgmap v4072: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:35:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:35:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:40.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:40.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:40.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:42 smithi146 ceph-mon[103278]: pgmap v4073: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:42 smithi078 ceph-mon[91136]: pgmap v4073: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:42 smithi078 ceph-mon[99959]: pgmap v4073: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:44 smithi078 ceph-mon[99959]: pgmap v4074: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:44 smithi078 ceph-mon[91136]: pgmap v4074: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:44 smithi146 ceph-mon[103278]: pgmap v4074: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:46 smithi078 ceph-mon[91136]: pgmap v4075: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:46 smithi078 ceph-mon[99959]: pgmap v4075: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:46 smithi146 ceph-mon[103278]: pgmap v4075: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:48 smithi078 ceph-mon[91136]: pgmap v4076: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:48 smithi078 ceph-mon[99959]: pgmap v4076: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:49.103 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:48 smithi146 ceph-mon[103278]: pgmap v4076: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:50.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:50 smithi146 ceph-mon[103278]: pgmap v4077: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:50 smithi078 ceph-mon[91136]: pgmap v4077: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:50 smithi078 ceph-mon[99959]: pgmap v4077: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:35:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:35:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:50.688Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:50.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:35:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:35:50.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:35:53.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:52 smithi078 ceph-mon[99959]: pgmap v4078: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:53.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:52 smithi078 ceph-mon[91136]: pgmap v4078: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:52 smithi146 ceph-mon[103278]: pgmap v4078: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:54 smithi078 ceph-mon[91136]: pgmap v4079: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:54 smithi078 ceph-mon[99959]: pgmap v4079: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:54 smithi146 ceph-mon[103278]: pgmap v4079: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:35:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:56 smithi146 ceph-mon[103278]: pgmap v4080: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:56 smithi078 ceph-mon[99959]: pgmap v4080: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:56 smithi078 ceph-mon[91136]: pgmap v4080: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:35:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:58 smithi146 ceph-mon[103278]: pgmap v4081: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[91136]: pgmap v4081: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[99959]: pgmap v4081: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:35:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:00 smithi078 ceph-mon[91136]: pgmap v4082: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:00 smithi078 ceph-mon[99959]: pgmap v4082: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:36:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:36:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:00.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:00.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:00.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:00 smithi146 ceph-mon[103278]: pgmap v4082: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:02 smithi146 ceph-mon[103278]: pgmap v4083: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:02 smithi078 ceph-mon[91136]: pgmap v4083: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:02 smithi078 ceph-mon[99959]: pgmap v4083: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:04 smithi146 ceph-mon[103278]: pgmap v4084: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:04 smithi078 ceph-mon[91136]: pgmap v4084: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:04 smithi078 ceph-mon[99959]: pgmap v4084: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:06 smithi146 ceph-mon[103278]: pgmap v4085: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:06 smithi078 ceph-mon[91136]: pgmap v4085: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:06 smithi078 ceph-mon[99959]: pgmap v4085: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:07.852 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:36:08.382 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:36:08.382 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 2h - - 2024-09-20T22:36:08.382 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 2h - - 2024-09-20T22:36:08.382 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 10s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:36:08.382 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 5s ago 2h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:36:08.382 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 10s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 5s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 10s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 10s ago 2h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 5s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 10s ago 2h 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 10s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 5s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 10s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 10s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 10s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:36:08.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 10s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 5s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 5s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 5s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 5s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 5s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 10s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 10s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:36:08.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 5s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:36:08.692 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:36:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:36:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:36:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:36:08.692 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:36:08.693 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:36:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:36:08.960 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:36:08.960 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:36:08.960 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:36:08.961 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:36:08.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:08 smithi078 ceph-mon[99959]: pgmap v4086: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:08.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:08 smithi078 ceph-mon[99959]: from='client.73214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:08.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3139965491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:08.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:08 smithi078 ceph-mon[91136]: pgmap v4086: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:08.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:08 smithi078 ceph-mon[91136]: from='client.73214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:08.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3139965491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:08 smithi146 ceph-mon[103278]: pgmap v4086: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:08 smithi146 ceph-mon[103278]: from='client.73214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3139965491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:09.270 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:36:09.270 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:09.270 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:09.270 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:09.270 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:09 smithi146 ceph-mon[103278]: from='client.53301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:09 smithi146 ceph-mon[103278]: from='client.53307 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:09 smithi146 ceph-mon[103278]: from='client.53319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1425117512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[91136]: from='client.53301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[91136]: from='client.53307 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[91136]: from='client.53319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1425117512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[99959]: from='client.53301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[99959]: from='client.53307 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[99959]: from='client.53319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1425117512' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:10.910 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:36:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:36:10.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:10.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:10.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:10.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:10.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:10.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:10 smithi146 ceph-mon[103278]: pgmap v4087: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:10 smithi078 ceph-mon[91136]: pgmap v4087: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:10 smithi078 ceph-mon[99959]: pgmap v4087: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:13.073 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:12 smithi078 ceph-mon[91136]: pgmap v4088: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:13.074 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:12 smithi078 ceph-mon[99959]: pgmap v4088: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:12 smithi146 ceph-mon[103278]: pgmap v4088: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:14.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:14.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:14 smithi078 ceph-mon[91136]: pgmap v4089: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:14 smithi078 ceph-mon[99959]: pgmap v4089: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:14 smithi146 ceph-mon[103278]: pgmap v4089: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:16 smithi146 ceph-mon[103278]: pgmap v4090: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:16 smithi078 ceph-mon[91136]: pgmap v4090: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:16 smithi078 ceph-mon[99959]: pgmap v4090: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:18 smithi146 ceph-mon[103278]: pgmap v4091: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[91136]: pgmap v4091: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[99959]: pgmap v4091: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:20.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:36:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:36:20.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:20.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:20.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:20.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:20.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:20.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:20 smithi146 ceph-mon[103278]: pgmap v4092: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:20 smithi078 ceph-mon[91136]: pgmap v4092: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:20 smithi078 ceph-mon[99959]: pgmap v4092: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:22 smithi146 ceph-mon[103278]: pgmap v4093: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:22 smithi078 ceph-mon[91136]: pgmap v4093: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:22 smithi078 ceph-mon[99959]: pgmap v4093: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:25 smithi078 ceph-mon[91136]: pgmap v4094: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:25 smithi078 ceph-mon[99959]: pgmap v4094: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:25 smithi146 ceph-mon[103278]: pgmap v4094: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:27 smithi078 ceph-mon[91136]: pgmap v4095: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:27 smithi078 ceph-mon[99959]: pgmap v4095: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:27 smithi146 ceph-mon[103278]: pgmap v4095: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:29 smithi078 ceph-mon[91136]: pgmap v4096: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:29 smithi078 ceph-mon[99959]: pgmap v4096: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:29 smithi146 ceph-mon[103278]: pgmap v4096: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:31.030 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:36:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:36:31.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:30.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:31.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:30.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:31.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:30.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:31 smithi078 ceph-mon[91136]: pgmap v4097: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:31 smithi078 ceph-mon[99959]: pgmap v4097: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:31.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:31 smithi146 ceph-mon[103278]: pgmap v4097: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:33 smithi078 ceph-mon[91136]: pgmap v4098: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:33 smithi078 ceph-mon[99959]: pgmap v4098: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:33 smithi146 ceph-mon[103278]: pgmap v4098: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:35 smithi078 ceph-mon[91136]: pgmap v4099: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:35 smithi078 ceph-mon[99959]: pgmap v4099: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:35 smithi146 ceph-mon[103278]: pgmap v4099: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:37 smithi146 ceph-mon[103278]: pgmap v4100: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:37 smithi078 ceph-mon[99959]: pgmap v4100: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:37 smithi078 ceph-mon[91136]: pgmap v4100: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:39 smithi146 ceph-mon[103278]: pgmap v4101: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[99959]: pgmap v4101: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[91136]: pgmap v4101: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:39.566 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:36:40.097 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 2h - - 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 2h - - 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 1s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 17s ago 2h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 1s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 17s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 1s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 1s ago 2h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 17s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 1s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 1s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 17s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 1s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:36:40.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 1s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 1s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 1s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 17s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 17s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 17s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 17s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 17s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 1s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 1s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:36:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 17s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:36:40.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:36:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:36:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:36:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:36:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:36:40.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:40.408 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:36:40.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:36:40.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:40.408 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:36:40.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:36:40.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:36:40.409 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:36:40.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:36:40.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:36:40.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:36:40.673 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:36:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:36:40.674 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:36:40.674 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:36:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:36:40.979 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:36:40.979 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:40.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:36:40.979 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:40.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:36:40.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:40.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:40.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:40.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:40.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:40.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[91136]: from='client.53331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[91136]: pgmap v4102: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[91136]: from='client.73256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[91136]: from='client.73262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1281656377' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2392696959' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[99959]: from='client.53331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[99959]: pgmap v4102: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[99959]: from='client.73256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[99959]: from='client.73262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1281656377' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2392696959' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:41 smithi146 ceph-mon[103278]: from='client.53331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:41 smithi146 ceph-mon[103278]: pgmap v4102: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:41 smithi146 ceph-mon[103278]: from='client.73256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:41 smithi146 ceph-mon[103278]: from='client.73262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1281656377' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:36:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2392696959' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:36:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:42 smithi146 ceph-mon[103278]: from='client.53355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:42 smithi078 ceph-mon[91136]: from='client.53355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:42 smithi078 ceph-mon[99959]: from='client.53355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:43 smithi146 ceph-mon[103278]: pgmap v4103: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:43 smithi078 ceph-mon[91136]: pgmap v4103: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:43 smithi078 ceph-mon[99959]: pgmap v4103: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[99959]: pgmap v4104: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:44.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:44.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:44 smithi078 ceph-mon[91136]: pgmap v4104: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:36:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:44 smithi146 ceph-mon[103278]: pgmap v4104: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:46 smithi146 ceph-mon[103278]: pgmap v4105: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:46 smithi078 ceph-mon[91136]: pgmap v4105: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:46 smithi078 ceph-mon[99959]: pgmap v4105: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:49.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:48 smithi146 ceph-mon[103278]: pgmap v4106: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:48 smithi078 ceph-mon[91136]: pgmap v4106: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:48 smithi078 ceph-mon[99959]: pgmap v4106: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:50.850 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:50 smithi146 ceph-mon[103278]: pgmap v4107: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:50 smithi078 ceph-mon[91136]: pgmap v4107: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:50 smithi078 ceph-mon[99959]: pgmap v4107: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:36:50] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:36:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:50.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:50.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:36:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:36:50.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:36:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:52 smithi146 ceph-mon[103278]: pgmap v4108: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:52 smithi078 ceph-mon[91136]: pgmap v4108: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:52 smithi078 ceph-mon[99959]: pgmap v4108: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:54 smithi078 ceph-mon[91136]: pgmap v4109: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:54 smithi078 ceph-mon[99959]: pgmap v4109: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:54 smithi146 ceph-mon[103278]: pgmap v4109: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:56 smithi078 ceph-mon[91136]: pgmap v4110: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:56 smithi078 ceph-mon[99959]: pgmap v4110: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:56 smithi146 ceph-mon[103278]: pgmap v4110: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:58 smithi146 ceph-mon[103278]: pgmap v4111: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:58 smithi078 ceph-mon[91136]: pgmap v4111: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:36:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:58 smithi078 ceph-mon[99959]: pgmap v4111: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:00 smithi146 ceph-mon[103278]: pgmap v4112: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:00 smithi078 ceph-mon[91136]: pgmap v4112: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:00 smithi078 ceph-mon[99959]: pgmap v4112: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:37:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:37:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:00.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:00.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:00.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:02 smithi146 ceph-mon[103278]: pgmap v4113: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:02 smithi078 ceph-mon[91136]: pgmap v4113: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:02 smithi078 ceph-mon[99959]: pgmap v4113: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[91136]: pgmap v4114: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:04 smithi078 ceph-mon[99959]: pgmap v4114: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:04 smithi146 ceph-mon[103278]: pgmap v4114: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:06 smithi146 ceph-mon[103278]: pgmap v4115: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:06 smithi078 ceph-mon[99959]: pgmap v4115: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:06 smithi078 ceph-mon[91136]: pgmap v4115: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:08 smithi146 ceph-mon[103278]: pgmap v4116: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:08 smithi078 ceph-mon[99959]: pgmap v4116: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:08 smithi078 ceph-mon[91136]: pgmap v4116: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:10 smithi146 ceph-mon[103278]: pgmap v4117: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:10 smithi078 ceph-mon[91136]: pgmap v4117: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:10 smithi078 ceph-mon[99959]: pgmap v4117: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:37:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:37:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:10.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:10.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:10.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:11.275 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:37:11.802 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 2h - - 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 13s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 13s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 13s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 13s ago 2h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 8s ago 2h 172M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 13s ago 2h 179M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 13s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:37:11.803 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 13s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 13s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 13s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 13s ago 2h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 2h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 8s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 13s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 13s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:37:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:12.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:37:12.113 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:37:12.377 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:37:12.378 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:37:12.379 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:37:12.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[91136]: from='client.53367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:12.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[91136]: from='client.53373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:12.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[91136]: pgmap v4118: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:12.690 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:37:12.690 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:12.690 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:12.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:12 smithi146 ceph-mon[103278]: from='client.53367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:12.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:12 smithi146 ceph-mon[103278]: from='client.53373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:12.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:12 smithi146 ceph-mon[103278]: pgmap v4118: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:12.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:12 smithi146 ceph-mon[103278]: from='client.73298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:12.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1138518336' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[91136]: from='client.73298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1138518336' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[99959]: from='client.53367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[99959]: from='client.53373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[99959]: pgmap v4118: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[99959]: from='client.73298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1138518336' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:13 smithi146 ceph-mon[103278]: from='client.53391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3145702482' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:13 smithi078 ceph-mon[91136]: from='client.53391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3145702482' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:13 smithi078 ceph-mon[99959]: from='client.53391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3145702482' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:14.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:14 smithi078 ceph-mon[99959]: pgmap v4119: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:14.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:14 smithi078 ceph-mon[91136]: pgmap v4119: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:14 smithi146 ceph-mon[103278]: pgmap v4119: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:16 smithi146 ceph-mon[103278]: pgmap v4120: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:16 smithi078 ceph-mon[91136]: pgmap v4120: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:16 smithi078 ceph-mon[99959]: pgmap v4120: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:18 smithi146 ceph-mon[103278]: pgmap v4121: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:18 smithi078 ceph-mon[91136]: pgmap v4121: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:18 smithi078 ceph-mon[99959]: pgmap v4121: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:20 smithi078 ceph-mon[99959]: pgmap v4122: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:20 smithi078 ceph-mon[91136]: pgmap v4122: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:37:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:37:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:20.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:20.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:20.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:20 smithi146 ceph-mon[103278]: pgmap v4122: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:22 smithi146 ceph-mon[103278]: pgmap v4123: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:22 smithi078 ceph-mon[91136]: pgmap v4123: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:22 smithi078 ceph-mon[99959]: pgmap v4123: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:24 smithi146 ceph-mon[103278]: pgmap v4124: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[91136]: pgmap v4124: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:24 smithi078 ceph-mon[99959]: pgmap v4124: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:26 smithi146 ceph-mon[103278]: pgmap v4125: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:26 smithi078 ceph-mon[99959]: pgmap v4125: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:26 smithi078 ceph-mon[91136]: pgmap v4125: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:28 smithi078 ceph-mon[99959]: pgmap v4126: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:28 smithi078 ceph-mon[91136]: pgmap v4126: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:28 smithi146 ceph-mon[103278]: pgmap v4126: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:30 smithi146 ceph-mon[103278]: pgmap v4127: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:30 smithi078 ceph-mon[91136]: pgmap v4127: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:30 smithi078 ceph-mon[99959]: pgmap v4127: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:37:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:37:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:30.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:30.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:30.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:32.928 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:32 smithi146 ceph-mon[103278]: pgmap v4128: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:32 smithi078 ceph-mon[91136]: pgmap v4128: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:32 smithi078 ceph-mon[99959]: pgmap v4128: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:34 smithi146 ceph-mon[103278]: pgmap v4129: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:34 smithi078 ceph-mon[91136]: pgmap v4129: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:34 smithi078 ceph-mon[99959]: pgmap v4129: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:36.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:36 smithi078 ceph-mon[99959]: pgmap v4130: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:36.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:36.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:36 smithi078 ceph-mon[91136]: pgmap v4130: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:36.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:36 smithi146 ceph-mon[103278]: pgmap v4130: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:38.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:38 smithi078 ceph-mon[91136]: pgmap v4131: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:38.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:38 smithi078 ceph-mon[99959]: pgmap v4131: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:38 smithi146 ceph-mon[103278]: pgmap v4131: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:40.975 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:37:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:37:40.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:40.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:40.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:40.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:40.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:40.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:40 smithi146 ceph-mon[103278]: pgmap v4132: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:40 smithi078 ceph-mon[91136]: pgmap v4132: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:40 smithi078 ceph-mon[99959]: pgmap v4132: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:42.990 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:37:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:42 smithi146 ceph-mon[103278]: pgmap v4133: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:42 smithi078 ceph-mon[99959]: pgmap v4133: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:42 smithi078 ceph-mon[91136]: pgmap v4133: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:43.518 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 2h - - 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 2h - - 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 4s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 20s ago 2h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 4s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 20s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 4s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 4s ago 2h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 20s ago 2h 170M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 4s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 4s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:37:43.519 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 20s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 4s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 4s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 4s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 4s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 20s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 20s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 20s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 20s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 20s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 4s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 20s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 4s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:37:43.520 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 20s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:37:43.835 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:37:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:37:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:37:43.837 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:37:44.105 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:37:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:37:44.106 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[91136]: from='client.53403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2870765534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[99959]: from='client.53403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2870765534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:43 smithi146 ceph-mon[103278]: from='client.53403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:37:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2870765534' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:37:44.416 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:37:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:44.417 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:37:44.417 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:44.417 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:37:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:44 smithi146 ceph-mon[103278]: from='client.73328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:44 smithi146 ceph-mon[103278]: from='client.53415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:44 smithi146 ceph-mon[103278]: pgmap v4134: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4217439383' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[99959]: from='client.73328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[99959]: from='client.53415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[99959]: pgmap v4134: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4217439383' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[91136]: from='client.73328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[91136]: from='client.53415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[91136]: pgmap v4134: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4217439383' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:37:46.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:45 smithi078 ceph-mon[91136]: from='client.73346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:46.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:45 smithi078 ceph-mon[99959]: from='client.73346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:45 smithi146 ceph-mon[103278]: from='client.73346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:46 smithi146 ceph-mon[103278]: pgmap v4135: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:46 smithi078 ceph-mon[99959]: pgmap v4135: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:46 smithi078 ceph-mon[91136]: pgmap v4135: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:48 smithi146 ceph-mon[103278]: pgmap v4136: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:48 smithi078 ceph-mon[91136]: pgmap v4136: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:48 smithi078 ceph-mon[99959]: pgmap v4136: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:50.994 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:37:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:37:50.994 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:50.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:50.994 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:50.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:37:50.995 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:37:50.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:37:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:50 smithi146 ceph-mon[103278]: pgmap v4137: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:50 smithi078 ceph-mon[91136]: pgmap v4137: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:50 smithi078 ceph-mon[99959]: pgmap v4137: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:52 smithi146 ceph-mon[103278]: pgmap v4138: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:52 smithi078 ceph-mon[91136]: pgmap v4138: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:52 smithi078 ceph-mon[99959]: pgmap v4138: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:54 smithi146 ceph-mon[103278]: pgmap v4139: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:54 smithi078 ceph-mon[91136]: pgmap v4139: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:54 smithi078 ceph-mon[99959]: pgmap v4139: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:57 smithi146 ceph-mon[103278]: pgmap v4140: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:57 smithi078 ceph-mon[91136]: pgmap v4140: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:57 smithi078 ceph-mon[99959]: pgmap v4140: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:59 smithi146 ceph-mon[103278]: pgmap v4141: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:37:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:59 smithi078 ceph-mon[91136]: pgmap v4141: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:37:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:37:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:59 smithi078 ceph-mon[99959]: pgmap v4141: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:37:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:38:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:38:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:00.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:00.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:00.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:01 smithi146 ceph-mon[103278]: pgmap v4142: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:01 smithi078 ceph-mon[91136]: pgmap v4142: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:01 smithi078 ceph-mon[99959]: pgmap v4142: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:02 smithi146 ceph-mon[103278]: pgmap v4143: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:02 smithi078 ceph-mon[91136]: pgmap v4143: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:02 smithi078 ceph-mon[99959]: pgmap v4143: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:04 smithi146 ceph-mon[103278]: pgmap v4144: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[91136]: pgmap v4144: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[99959]: pgmap v4144: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:06 smithi146 ceph-mon[103278]: pgmap v4145: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:06 smithi078 ceph-mon[91136]: pgmap v4145: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:06 smithi078 ceph-mon[99959]: pgmap v4145: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:08.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:08 smithi078 ceph-mon[99959]: pgmap v4146: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:08.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:08 smithi078 ceph-mon[91136]: pgmap v4146: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:08 smithi146 ceph-mon[103278]: pgmap v4146: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:10 smithi146 ceph-mon[103278]: pgmap v4147: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:10 smithi078 ceph-mon[91136]: pgmap v4147: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:10 smithi078 ceph-mon[99959]: pgmap v4147: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:38:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:38:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:10.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:10.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:10.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:12.980 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:12 smithi146 ceph-mon[103278]: pgmap v4148: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:12 smithi078 ceph-mon[91136]: pgmap v4148: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:12 smithi078 ceph-mon[99959]: pgmap v4148: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:38:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:14 smithi146 ceph-mon[103278]: pgmap v4149: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:14 smithi078 ceph-mon[99959]: pgmap v4149: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:14 smithi078 ceph-mon[91136]: pgmap v4149: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:15.240 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:38:15.240 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 2h - - 2024-09-20T22:38:15.240 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 2h - - 2024-09-20T22:38:15.240 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 16s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 11s ago 2h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 16s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 11s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 16s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 16s ago 2h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 11s ago 2h 171M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 16s ago 2h 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 16s ago 2h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 11s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 16s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 16s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 16s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 16s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 11s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:38:15.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 11s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 11s ago 2h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 11s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 11s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 16s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 16s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:38:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 11s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:38:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:38:15.552 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:38:15.820 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:38:15.821 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:38:15.821 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:38:15.821 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:38:15.821 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:38:15.821 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:15 smithi078 ceph-mon[99959]: from='client.73358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.821 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:15 smithi078 ceph-mon[99959]: from='client.73364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.821 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3802918122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:15.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:15 smithi078 ceph-mon[91136]: from='client.73358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:15 smithi078 ceph-mon[91136]: from='client.73364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3802918122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:15 smithi146 ceph-mon[103278]: from='client.73358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:15 smithi146 ceph-mon[103278]: from='client.73364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3802918122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:16.135 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:38:16.135 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:16.135 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:16.135 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:16.135 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:16.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:16 smithi146 ceph-mon[103278]: from='client.53451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:16.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:16 smithi146 ceph-mon[103278]: pgmap v4150: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:16.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:16 smithi146 ceph-mon[103278]: from='client.53463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:16.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:16.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4095752017' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[91136]: from='client.53451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[91136]: pgmap v4150: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[91136]: from='client.53463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4095752017' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[99959]: from='client.53451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[99959]: pgmap v4150: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:17.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[99959]: from='client.53463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:17.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4095752017' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:18.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:18 smithi078 ceph-mon[99959]: pgmap v4151: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:18.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:18 smithi078 ceph-mon[91136]: pgmap v4151: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:18 smithi146 ceph-mon[103278]: pgmap v4151: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:20 smithi146 ceph-mon[103278]: pgmap v4152: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[91136]: pgmap v4152: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-mon[99959]: pgmap v4152: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:38:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:38:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:20.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:20.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:20.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:22 smithi146 ceph-mon[103278]: pgmap v4153: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:22 smithi078 ceph-mon[91136]: pgmap v4153: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:22 smithi078 ceph-mon[99959]: pgmap v4153: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:24 smithi146 ceph-mon[103278]: pgmap v4154: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[91136]: pgmap v4154: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[99959]: pgmap v4154: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:26 smithi078 ceph-mon[91136]: pgmap v4155: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:26 smithi078 ceph-mon[99959]: pgmap v4155: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:26 smithi146 ceph-mon[103278]: pgmap v4155: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:29.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:28 smithi078 ceph-mon[91136]: pgmap v4156: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:29.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:29.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:28 smithi078 ceph-mon[99959]: pgmap v4156: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:29.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:28 smithi146 ceph-mon[103278]: pgmap v4156: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:30 smithi078 ceph-mon[91136]: pgmap v4157: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:30 smithi078 ceph-mon[99959]: pgmap v4157: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:38:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:38:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:30.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:30.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:30.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:30 smithi146 ceph-mon[103278]: pgmap v4157: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:32 smithi146 ceph-mon[103278]: pgmap v4158: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:32 smithi078 ceph-mon[91136]: pgmap v4158: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:32 smithi078 ceph-mon[99959]: pgmap v4158: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:34.861 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:34 smithi146 ceph-mon[103278]: pgmap v4159: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:34 smithi078 ceph-mon[91136]: pgmap v4159: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:34 smithi078 ceph-mon[99959]: pgmap v4159: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:36 smithi146 ceph-mon[103278]: pgmap v4160: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:36 smithi078 ceph-mon[91136]: pgmap v4160: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:36 smithi078 ceph-mon[99959]: pgmap v4160: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:38 smithi078 ceph-mon[91136]: pgmap v4161: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:38 smithi078 ceph-mon[99959]: pgmap v4161: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:38 smithi146 ceph-mon[103278]: pgmap v4161: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[91136]: pgmap v4162: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-mon[99959]: pgmap v4162: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:40 smithi146 ceph-mon[103278]: pgmap v4162: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:41.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:38:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:38:41.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:40.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:41.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:40.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:41.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:40.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:42 smithi146 ceph-mon[103278]: pgmap v4163: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:42 smithi078 ceph-mon[91136]: pgmap v4163: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:42 smithi078 ceph-mon[99959]: pgmap v4163: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:43.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:44 smithi146 ceph-mon[103278]: pgmap v4164: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[99959]: pgmap v4164: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[91136]: pgmap v4164: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:38:46.435 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:38:46.962 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:38:46.962 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 2h - - 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 2h - - 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 3s ago 2h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 7s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 3s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 7s ago 2h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 3s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 2h 179M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 3s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:38:46.963 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 3s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 3s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 3s ago 2h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 3s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 3s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:38:46.964 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 3s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:38:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:46 smithi146 ceph-mon[103278]: pgmap v4165: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:38:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:38:47.273 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:38:47.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:46 smithi078 ceph-mon[99959]: pgmap v4165: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:47.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:47.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:46 smithi078 ceph-mon[91136]: pgmap v4165: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:47.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:47.540 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:38:47.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:38:47.854 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:38:47.855 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:47.855 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:38:47.855 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:47.855 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:38:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:47 smithi146 ceph-mon[103278]: from='client.53475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:47 smithi146 ceph-mon[103278]: from='client.73400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:47 smithi146 ceph-mon[103278]: from='client.73406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1649181494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/445403999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[91136]: from='client.53475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[91136]: from='client.73400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[91136]: from='client.73406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1649181494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/445403999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[99959]: from='client.53475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[99959]: from='client.73400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[99959]: from='client.73406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1649181494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:38:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/445403999' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:38:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:48 smithi146 ceph-mon[103278]: from='client.53499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:48 smithi146 ceph-mon[103278]: pgmap v4166: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:48 smithi078 ceph-mon[91136]: from='client.53499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:48 smithi078 ceph-mon[91136]: pgmap v4166: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:48 smithi078 ceph-mon[99959]: from='client.53499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:48 smithi078 ceph-mon[99959]: pgmap v4166: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:50.952 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:38:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:38:50.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:50.701Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:50.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:50.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:38:50.953 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:38:50.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:38:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:50 smithi146 ceph-mon[103278]: pgmap v4167: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:50 smithi078 ceph-mon[91136]: pgmap v4167: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:50 smithi078 ceph-mon[99959]: pgmap v4167: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:52 smithi146 ceph-mon[103278]: pgmap v4168: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:52 smithi078 ceph-mon[99959]: pgmap v4168: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:52 smithi078 ceph-mon[91136]: pgmap v4168: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:54 smithi146 ceph-mon[103278]: pgmap v4169: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:54 smithi078 ceph-mon[99959]: pgmap v4169: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:54 smithi078 ceph-mon[91136]: pgmap v4169: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:56 smithi146 ceph-mon[103278]: pgmap v4170: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:56 smithi078 ceph-mon[99959]: pgmap v4170: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:56 smithi078 ceph-mon[91136]: pgmap v4170: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:59.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:58 smithi078 ceph-mon[91136]: pgmap v4171: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:59.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:59.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:58 smithi078 ceph-mon[99959]: pgmap v4171: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:59.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:38:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:58 smithi146 ceph-mon[103278]: pgmap v4171: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:00.703 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:39:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[91136]: pgmap v4172: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-mon[99959]: pgmap v4172: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:00 smithi146 ceph-mon[103278]: pgmap v4172: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:00.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:00.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:00.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:02 smithi146 ceph-mon[103278]: pgmap v4173: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:02 smithi078 ceph-mon[91136]: pgmap v4173: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:02 smithi078 ceph-mon[99959]: pgmap v4173: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:05 smithi146 ceph-mon[103278]: pgmap v4174: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[91136]: pgmap v4174: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[99959]: pgmap v4174: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:07 smithi146 ceph-mon[103278]: pgmap v4175: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:07 smithi078 ceph-mon[99959]: pgmap v4175: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:07 smithi078 ceph-mon[91136]: pgmap v4175: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:09 smithi146 ceph-mon[103278]: pgmap v4176: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:09 smithi078 ceph-mon[99959]: pgmap v4176: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:09 smithi078 ceph-mon[91136]: pgmap v4176: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:39:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:39:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:10.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:10.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:10.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:11 smithi146 ceph-mon[103278]: pgmap v4177: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:11 smithi078 ceph-mon[99959]: pgmap v4177: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:11 smithi078 ceph-mon[91136]: pgmap v4177: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:13 smithi146 ceph-mon[103278]: pgmap v4178: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:13 smithi078 ceph-mon[99959]: pgmap v4178: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:13 smithi078 ceph-mon[91136]: pgmap v4178: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:15.192 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:15 smithi146 ceph-mon[103278]: pgmap v4179: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:15.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:15 smithi078 ceph-mon[99959]: pgmap v4179: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:15.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:15 smithi078 ceph-mon[91136]: pgmap v4179: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:17 smithi146 ceph-mon[103278]: pgmap v4180: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:17 smithi078 ceph-mon[91136]: pgmap v4180: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:17 smithi078 ceph-mon[99959]: pgmap v4180: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:18.140 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:39:18.673 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:39:18.673 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 2h - - 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 2h - - 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 19s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 14s ago 2h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 19s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 14s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 19s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 19s ago 2h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 14s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 19s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 19s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 14s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 19s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:39:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 19s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 19s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 19s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 14s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 14s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 14s ago 2h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 14s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 14s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 19s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 19s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:39:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 14s ago 2h 88.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:39:18.992 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:39:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:39:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:39:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:39:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:39:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:39:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:39:18.994 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:39:19.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:19 smithi078 ceph-mon[91136]: pgmap v4181: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:19.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3407476625' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:19.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:19 smithi078 ceph-mon[99959]: pgmap v4181: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:19.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3407476625' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:19.262 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:39:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:39:19.262 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:39:19.263 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:39:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:19 smithi146 ceph-mon[103278]: pgmap v4181: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3407476625' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:19.585 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:39:19.585 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:19.585 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:19.585 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:19.585 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:20.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='client.53511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='client.53517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='client.73442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1310247755' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='client.53511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='client.53517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='client.73442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1310247755' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:20.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.105 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='client.53511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='client.53517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='client.73442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1310247755' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:39:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:39:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:20.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:20.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:20.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:20.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:21.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:21 smithi078 ceph-mon[99959]: from='client.73454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:21 smithi078 ceph-mon[99959]: pgmap v4182: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:21.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:21 smithi078 ceph-mon[91136]: from='client.73454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:21 smithi078 ceph-mon[91136]: pgmap v4182: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:21 smithi146 ceph-mon[103278]: from='client.73454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:21 smithi146 ceph-mon[103278]: pgmap v4182: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:22 smithi146 ceph-mon[103278]: pgmap v4183: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:22 smithi078 ceph-mon[91136]: pgmap v4183: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:22 smithi078 ceph-mon[99959]: pgmap v4183: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:24 smithi146 ceph-mon[103278]: pgmap v4184: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[91136]: pgmap v4184: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[99959]: pgmap v4184: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:26 smithi146 ceph-mon[103278]: pgmap v4185: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:26 smithi078 ceph-mon[91136]: pgmap v4185: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:26 smithi078 ceph-mon[99959]: pgmap v4185: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:28 smithi146 ceph-mon[103278]: pgmap v4186: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:28 smithi078 ceph-mon[91136]: pgmap v4186: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:28 smithi078 ceph-mon[99959]: pgmap v4186: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:30 smithi078 ceph-mon[91136]: pgmap v4187: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:39:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:39:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:30 smithi078 ceph-mon[99959]: pgmap v4187: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:30.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:30.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:30.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:30.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:30 smithi146 ceph-mon[103278]: pgmap v4187: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:32 smithi146 ceph-mon[103278]: pgmap v4188: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:32 smithi078 ceph-mon[91136]: pgmap v4188: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:32 smithi078 ceph-mon[99959]: pgmap v4188: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:34 smithi146 ceph-mon[103278]: pgmap v4189: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:34 smithi078 ceph-mon[99959]: pgmap v4189: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:34 smithi078 ceph-mon[91136]: pgmap v4189: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:36 smithi146 ceph-mon[103278]: pgmap v4190: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:36 smithi078 ceph-mon[91136]: pgmap v4190: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:36 smithi078 ceph-mon[99959]: pgmap v4190: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:38 smithi146 ceph-mon[103278]: pgmap v4191: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:38 smithi078 ceph-mon[99959]: pgmap v4191: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:38 smithi078 ceph-mon[91136]: pgmap v4191: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[91136]: pgmap v4192: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:39:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:39:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[99959]: pgmap v4192: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:40.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:40.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:40.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:40.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:40.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:40 smithi146 ceph-mon[103278]: pgmap v4192: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:42 smithi146 ceph-mon[103278]: pgmap v4193: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:42 smithi078 ceph-mon[91136]: pgmap v4193: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:42 smithi078 ceph-mon[99959]: pgmap v4193: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:44 smithi146 ceph-mon[103278]: pgmap v4194: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[91136]: pgmap v4194: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[99959]: pgmap v4194: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:39:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:46 smithi146 ceph-mon[103278]: pgmap v4195: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:46 smithi078 ceph-mon[91136]: pgmap v4195: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:46 smithi078 ceph-mon[99959]: pgmap v4195: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:48 smithi146 ceph-mon[103278]: pgmap v4196: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:48 smithi078 ceph-mon[91136]: pgmap v4196: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:48 smithi078 ceph-mon[99959]: pgmap v4196: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:49.885 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:39:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:39:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 2h - - 2024-09-20T22:39:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 2h - - 2024-09-20T22:39:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 10s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:39:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 6s ago 2h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 10s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 6s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 10s ago 2h 569M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 10s ago 2h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 6s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 10s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 10s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 6s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 10s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 10s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 10s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 10s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:39:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 6s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 6s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 6s ago 2h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 6s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 6s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 10s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 10s ago 2h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:39:50.439 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:39:50.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:39:50.751 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:39:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:50 smithi146 ceph-mon[103278]: pgmap v4197: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:50 smithi146 ceph-mon[103278]: from='client.73466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:39:51.019 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:39:51.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:39:51.020 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:39:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:39:51.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-mon[99959]: pgmap v4197: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:51.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-mon[99959]: from='client.73466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:51.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-mon[91136]: pgmap v4197: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:51.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-mon[91136]: from='client.73466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:51.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:50.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:51.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:50.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:39:51.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:39:50.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:39:51.329 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:39:51.329 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:39:51.329 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:39:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:51 smithi146 ceph-mon[103278]: from='client.53553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:51 smithi146 ceph-mon[103278]: from='client.53559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3425950218' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:51 smithi146 ceph-mon[103278]: from='client.53571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/50829451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[91136]: from='client.53553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[91136]: from='client.53559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3425950218' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[91136]: from='client.53571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/50829451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[99959]: from='client.53553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[99959]: from='client.53559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3425950218' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:39:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[99959]: from='client.53571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/50829451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:39:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:52 smithi146 ceph-mon[103278]: pgmap v4198: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:52 smithi078 ceph-mon[91136]: pgmap v4198: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:52 smithi078 ceph-mon[99959]: pgmap v4198: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:54 smithi146 ceph-mon[103278]: pgmap v4199: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:54 smithi078 ceph-mon[91136]: pgmap v4199: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:54 smithi078 ceph-mon[99959]: pgmap v4199: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:56.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:56 smithi146 ceph-mon[103278]: pgmap v4200: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:56.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:56 smithi078 ceph-mon[91136]: pgmap v4200: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:56 smithi078 ceph-mon[99959]: pgmap v4200: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:58 smithi146 ceph-mon[103278]: pgmap v4201: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:58 smithi078 ceph-mon[91136]: pgmap v4201: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:39:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:58 smithi078 ceph-mon[99959]: pgmap v4201: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:00.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T22:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:00.707 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:40:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:40:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[99959]: pgmap v4202: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[91136]: pgmap v4202: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:00.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:00.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:00.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:00 smithi146 ceph-mon[103278]: pgmap v4202: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:02.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:02 smithi078 ceph-mon[91136]: pgmap v4203: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:02.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:02 smithi078 ceph-mon[99959]: pgmap v4203: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:02 smithi146 ceph-mon[103278]: pgmap v4203: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:04 smithi146 ceph-mon[103278]: pgmap v4204: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[91136]: pgmap v4204: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[99959]: pgmap v4204: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:06 smithi146 ceph-mon[103278]: pgmap v4205: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:06 smithi078 ceph-mon[99959]: pgmap v4205: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:06 smithi078 ceph-mon[91136]: pgmap v4205: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:08 smithi146 ceph-mon[103278]: pgmap v4206: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:08 smithi078 ceph-mon[91136]: pgmap v4206: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:08 smithi078 ceph-mon[99959]: pgmap v4206: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:10.922 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:40:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:40:10.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:10.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:10.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:10.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:10.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:10.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:10 smithi146 ceph-mon[103278]: pgmap v4207: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:10 smithi078 ceph-mon[91136]: pgmap v4207: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:10 smithi078 ceph-mon[99959]: pgmap v4207: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:12 smithi146 ceph-mon[103278]: pgmap v4208: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:12 smithi078 ceph-mon[91136]: pgmap v4208: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:12 smithi078 ceph-mon[99959]: pgmap v4208: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:14.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:14.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:14 smithi146 ceph-mon[103278]: pgmap v4209: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:14 smithi078 ceph-mon[91136]: pgmap v4209: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:14 smithi078 ceph-mon[99959]: pgmap v4209: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:16 smithi146 ceph-mon[103278]: pgmap v4210: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:16 smithi078 ceph-mon[91136]: pgmap v4210: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:16 smithi078 ceph-mon[99959]: pgmap v4210: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:19.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:18 smithi146 ceph-mon[103278]: pgmap v4211: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:40:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:18 smithi078 ceph-mon[91136]: pgmap v4211: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:40:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:18 smithi078 ceph-mon[99959]: pgmap v4211: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:40:20.880 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:40:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:40:20.880 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:20.707Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:20.880 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:20.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:20.880 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:20.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:21.148 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[99959]: pgmap v4212: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:21.148 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.149 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[91136]: pgmap v4212: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:21.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:20 smithi146 ceph-mon[103278]: pgmap v4212: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:21.632 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:40:22.183 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:40:22.183 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 2h - - 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 2h - - 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 2s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 17s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 2s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 17s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 2s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 2s ago 2h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 17s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 2s ago 2h 179M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 2s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 17s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 2s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 2s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:40:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 2s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 2s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 17s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 17s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 17s ago 2h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 17s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 17s ago 2h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 2s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 2s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:40:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:22.508 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:40:22.509 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:40:22.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:40:22.509 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:40:22.790 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:40:22.791 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:40:23.109 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:23.109 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:23.109 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:23.109 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:22 smithi146 ceph-mon[103278]: from='client.73502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:22 smithi146 ceph-mon[103278]: pgmap v4213: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:22 smithi146 ceph-mon[103278]: from='client.73508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/900039088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[91136]: from='client.73502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[91136]: pgmap v4213: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[91136]: from='client.73508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/900039088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[99959]: from='client.73502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[99959]: pgmap v4213: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[99959]: from='client.73508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/900039088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:23 smithi146 ceph-mon[103278]: from='client.73514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:23 smithi146 ceph-mon[103278]: from='client.73526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2797927480' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:23 smithi078 ceph-mon[99959]: from='client.73514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:23 smithi078 ceph-mon[99959]: from='client.73526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2797927480' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:23 smithi078 ceph-mon[91136]: from='client.73514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:23 smithi078 ceph-mon[91136]: from='client.73526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2797927480' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:24 smithi146 ceph-mon[103278]: pgmap v4214: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[99959]: pgmap v4214: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[91136]: pgmap v4214: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:25.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:26 smithi146 ceph-mon[103278]: pgmap v4215: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:26 smithi078 ceph-mon[99959]: pgmap v4215: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:26 smithi078 ceph-mon[91136]: pgmap v4215: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:28 smithi146 ceph-mon[103278]: pgmap v4216: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:28 smithi078 ceph-mon[99959]: pgmap v4216: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:28 smithi078 ceph-mon[91136]: pgmap v4216: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:31.003 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:40:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:40:31.003 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:30.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:31.003 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:30.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:31.003 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:30.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:31 smithi146 ceph-mon[103278]: pgmap v4217: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:31 smithi078 ceph-mon[91136]: pgmap v4217: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:31 smithi078 ceph-mon[99959]: pgmap v4217: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:33 smithi146 ceph-mon[103278]: pgmap v4218: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:33 smithi078 ceph-mon[99959]: pgmap v4218: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:33 smithi078 ceph-mon[91136]: pgmap v4218: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:35 smithi146 ceph-mon[103278]: pgmap v4219: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:35 smithi078 ceph-mon[99959]: pgmap v4219: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:35 smithi078 ceph-mon[91136]: pgmap v4219: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:37 smithi146 ceph-mon[103278]: pgmap v4220: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:37 smithi078 ceph-mon[91136]: pgmap v4220: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:37 smithi078 ceph-mon[99959]: pgmap v4220: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:39.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:39 smithi146 ceph-mon[103278]: pgmap v4221: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:39 smithi078 ceph-mon[91136]: pgmap v4221: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:39 smithi078 ceph-mon[99959]: pgmap v4221: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:40:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:40:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:40.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:40.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:40.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:41.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[91136]: pgmap v4222: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[99959]: pgmap v4222: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:41 smithi146 ceph-mon[103278]: pgmap v4222: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:43 smithi146 ceph-mon[103278]: pgmap v4223: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:43 smithi078 ceph-mon[91136]: pgmap v4223: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:43 smithi078 ceph-mon[99959]: pgmap v4223: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:44.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:44.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:45 smithi146 ceph-mon[103278]: pgmap v4224: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[99959]: pgmap v4224: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[91136]: pgmap v4224: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:40:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:47 smithi146 ceph-mon[103278]: pgmap v4225: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:47 smithi078 ceph-mon[91136]: pgmap v4225: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:47 smithi078 ceph-mon[99959]: pgmap v4225: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:49 smithi146 ceph-mon[103278]: pgmap v4226: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:49 smithi078 ceph-mon[91136]: pgmap v4226: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:49 smithi078 ceph-mon[99959]: pgmap v4226: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:50 smithi146 ceph-mon[103278]: pgmap v4227: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:50 smithi078 ceph-mon[91136]: pgmap v4227: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:50 smithi078 ceph-mon[99959]: pgmap v4227: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:40:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:40:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:50.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:50.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:40:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:40:50.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:40:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:52 smithi146 ceph-mon[103278]: pgmap v4228: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:52 smithi078 ceph-mon[91136]: pgmap v4228: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:52 smithi078 ceph-mon[99959]: pgmap v4228: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:53.406 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 2h - - 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 2h - - 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 13s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 9s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 13s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 9s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 13s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 13s ago 2h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:40:53.937 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 9s ago 2h 173M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 13s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 13s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 9s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 13s ago 2h 74.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 13s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 13s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 13s ago 2h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 9s ago 2h 83.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 9s ago 2h 68.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 9s ago 2h 72.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 9s ago 2h 68.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 9s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 13s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:40:53.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 13s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:40:53.939 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:40:54.246 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:40:54.513 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:40:54.513 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:40:54.513 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:40:54.514 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:40:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[99959]: from='client.53619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[99959]: from='client.73544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[99959]: pgmap v4229: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[99959]: from='client.53631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1946444952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[91136]: from='client.53619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[91136]: from='client.73544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[91136]: pgmap v4229: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[91136]: from='client.53631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1946444952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:54.825 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:40:54.826 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:54.826 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:40:54.826 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:54.826 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:40:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:54 smithi146 ceph-mon[103278]: from='client.53619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:54 smithi146 ceph-mon[103278]: from='client.73544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:54 smithi146 ceph-mon[103278]: pgmap v4229: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:40:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:54 smithi146 ceph-mon[103278]: from='client.53631 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1946444952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:40:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:55 smithi146 ceph-mon[103278]: from='client.53643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2264365005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:55 smithi078 ceph-mon[91136]: from='client.53643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2264365005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:55 smithi078 ceph-mon[99959]: from='client.53643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2264365005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:40:56.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:56 smithi146 ceph-mon[103278]: pgmap v4230: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:40:56.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:56 smithi078 ceph-mon[91136]: pgmap v4230: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:40:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:56 smithi078 ceph-mon[99959]: pgmap v4230: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:40:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:58 smithi146 ceph-mon[103278]: pgmap v4231: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:40:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:58 smithi078 ceph-mon[91136]: pgmap v4231: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:40:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:40:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:58 smithi078 ceph-mon[99959]: pgmap v4231: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:40:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:00 smithi146 ceph-mon[103278]: pgmap v4232: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:41:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:41:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:41:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[91136]: pgmap v4232: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:41:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[99959]: pgmap v4232: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:41:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:00.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:00.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:00.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:02 smithi146 ceph-mon[103278]: pgmap v4233: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:41:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:02 smithi078 ceph-mon[91136]: pgmap v4233: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:41:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:02 smithi078 ceph-mon[99959]: pgmap v4233: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:41:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:04 smithi078 ceph-mon[99959]: pgmap v4234: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:41:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:04 smithi078 ceph-mon[91136]: pgmap v4234: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:41:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:04 smithi146 ceph-mon[103278]: pgmap v4234: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:41:06.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:06 smithi146 ceph-mon[103278]: pgmap v4235: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:41:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:06 smithi078 ceph-mon[99959]: pgmap v4235: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:41:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:06 smithi078 ceph-mon[91136]: pgmap v4235: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:41:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:08 smithi146 ceph-mon[103278]: pgmap v4236: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:41:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:08 smithi078 ceph-mon[99959]: pgmap v4236: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:41:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:08 smithi078 ceph-mon[91136]: pgmap v4236: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:41:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:10 smithi078 ceph-mon[91136]: pgmap v4237: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:41:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:10 smithi078 ceph-mon[99959]: pgmap v4237: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:41:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:41:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:41:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:10.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:10.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:10.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:10 smithi146 ceph-mon[103278]: pgmap v4237: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:41:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:12 smithi146 ceph-mon[103278]: pgmap v4238: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T22:41:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:12 smithi078 ceph-mon[91136]: pgmap v4238: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T22:41:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:12 smithi078 ceph-mon[99959]: pgmap v4238: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T22:41:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:14 smithi146 ceph-mon[103278]: pgmap v4239: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:41:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:14 smithi078 ceph-mon[91136]: pgmap v4239: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:41:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:14 smithi078 ceph-mon[99959]: pgmap v4239: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:41:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:16 smithi146 ceph-mon[103278]: pgmap v4240: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T22:41:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:16 smithi078 ceph-mon[91136]: pgmap v4240: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T22:41:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:16 smithi078 ceph-mon[99959]: pgmap v4240: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T22:41:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:19.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:18 smithi146 ceph-mon[103278]: pgmap v4241: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:41:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:18 smithi078 ceph-mon[91136]: pgmap v4241: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:41:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:18 smithi078 ceph-mon[99959]: pgmap v4241: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:41:20.712 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:41:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[91136]: pgmap v4242: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[99959]: pgmap v4242: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:20.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:20.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:20.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:20 smithi146 ceph-mon[103278]: pgmap v4242: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:41:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:23.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:22 smithi078 ceph-mon[99959]: pgmap v4243: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T22:41:23.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:22 smithi078 ceph-mon[91136]: pgmap v4243: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T22:41:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:22 smithi146 ceph-mon[103278]: pgmap v4243: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T22:41:25.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:24 smithi078 ceph-mon[99959]: pgmap v4244: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:41:25.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:24 smithi078 ceph-mon[91136]: pgmap v4244: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:41:25.137 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:41:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:24 smithi146 ceph-mon[103278]: pgmap v4244: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:41:25.672 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:41:25.672 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:41:25.672 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 2h - - 2024-09-20T22:41:25.672 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 5s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 0s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 5s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 0s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 5s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 5s ago 2h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 0s ago 2h 175M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 5s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 5s ago 2h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 0s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 5s ago 2h 74.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 5s ago 2h 78.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 5s ago 2h 68.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 5s ago 2h 77.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 0s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:41:25.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 0s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 0s ago 2h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 0s ago 2h 69.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 0s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 5s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 5s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:41:25.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:41:25.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:41:25.982 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:41:26.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:25 smithi146 ceph-mon[103278]: from='client.73574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:26.249 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:41:26.249 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:41:26.249 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:41:26.249 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:41:26.249 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:41:26.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:41:26.250 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:41:26.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[99959]: from='client.73574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:26.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:26.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:25 smithi078 ceph-mon[91136]: from='client.73574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:26.559 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:41:26.559 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:26.559 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:26 smithi146 ceph-mon[103278]: from='client.73580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:26 smithi146 ceph-mon[103278]: from='client.53667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:26 smithi146 ceph-mon[103278]: pgmap v4245: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:41:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2793612223' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4232790354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[91136]: from='client.73580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[91136]: from='client.53667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[91136]: pgmap v4245: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2793612223' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4232790354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[99959]: from='client.73580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[99959]: from='client.53667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[99959]: pgmap v4245: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2793612223' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4232790354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:27 smithi146 ceph-mon[103278]: from='client.53679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:27 smithi078 ceph-mon[91136]: from='client.53679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:27 smithi078 ceph-mon[99959]: from='client.53679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:28 smithi146 ceph-mon[103278]: pgmap v4246: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:28 smithi078 ceph-mon[91136]: pgmap v4246: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:28 smithi078 ceph-mon[99959]: pgmap v4246: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:30.976 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:41:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:41:30.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:30.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:30.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:30.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:30.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:30.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:30 smithi146 ceph-mon[103278]: pgmap v4247: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:30 smithi078 ceph-mon[91136]: pgmap v4247: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:30 smithi078 ceph-mon[99959]: pgmap v4247: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:32 smithi146 ceph-mon[103278]: pgmap v4248: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:32 smithi078 ceph-mon[91136]: pgmap v4248: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:32 smithi078 ceph-mon[99959]: pgmap v4248: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:34 smithi146 ceph-mon[103278]: pgmap v4249: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:34 smithi078 ceph-mon[91136]: pgmap v4249: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:34 smithi078 ceph-mon[99959]: pgmap v4249: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:36 smithi146 ceph-mon[103278]: pgmap v4250: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:37.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:36 smithi078 ceph-mon[91136]: pgmap v4250: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:36 smithi078 ceph-mon[99959]: pgmap v4250: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:39.239 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:38 smithi146 ceph-mon[103278]: pgmap v4251: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:38 smithi078 ceph-mon[91136]: pgmap v4251: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:38 smithi078 ceph-mon[99959]: pgmap v4251: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:41:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:41:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:40.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:40.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:40.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:40.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:41.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:40 smithi146 ceph-mon[103278]: pgmap v4252: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:41.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[91136]: pgmap v4252: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[99959]: pgmap v4252: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:41.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:42 smithi146 ceph-mon[103278]: pgmap v4253: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:42 smithi078 ceph-mon[91136]: pgmap v4253: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:42 smithi078 ceph-mon[99959]: pgmap v4253: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:45 smithi146 ceph-mon[103278]: pgmap v4254: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:45 smithi078 ceph-mon[91136]: pgmap v4254: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:45 smithi078 ceph-mon[99959]: pgmap v4254: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:41:47.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:47 smithi078 ceph-mon[99959]: pgmap v4255: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:47.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:47.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:47 smithi078 ceph-mon[91136]: pgmap v4255: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:47.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:47 smithi146 ceph-mon[103278]: pgmap v4255: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:49 smithi146 ceph-mon[103278]: pgmap v4256: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:49.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:49 smithi078 ceph-mon[91136]: pgmap v4256: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:49.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:49 smithi078 ceph-mon[99959]: pgmap v4256: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:41:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:41:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:50.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:50.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:41:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:41:50.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:41:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:51 smithi146 ceph-mon[103278]: pgmap v4257: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:51 smithi078 ceph-mon[91136]: pgmap v4257: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:51 smithi078 ceph-mon[99959]: pgmap v4257: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:53 smithi146 ceph-mon[103278]: pgmap v4258: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:53 smithi078 ceph-mon[99959]: pgmap v4258: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:53 smithi078 ceph-mon[91136]: pgmap v4258: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:55 smithi146 ceph-mon[103278]: pgmap v4259: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:55 smithi078 ceph-mon[99959]: pgmap v4259: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:55 smithi078 ceph-mon[91136]: pgmap v4259: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:56.853 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:41:57.379 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 2h - - 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 2h - - 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 16s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 12s ago 2h 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 16s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 12s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 16s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 16s ago 2h 277M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 12s ago 2h 176M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 16s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 16s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:41:57.380 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 12s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 16s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 16s ago 2h 77.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 16s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 16s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 12s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 12s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 12s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 12s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 12s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 16s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 16s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:41:57.381 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:41:57.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:57 smithi078 ceph-mon[91136]: pgmap v4260: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:57.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:57.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:57 smithi078 ceph-mon[99959]: pgmap v4260: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:57.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:57 smithi146 ceph-mon[103278]: pgmap v4260: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:57.692 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:41:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:41:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:41:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:41:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:41:57.694 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:41:57.964 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:41:57.964 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:41:57.965 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:41:57.966 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:41:57.966 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:41:58.275 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:41:58.275 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:41:58.276 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:41:58.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:58 smithi078 ceph-mon[99959]: from='client.53691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:58.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:58 smithi078 ceph-mon[99959]: from='client.73616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:58.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1139006826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:58.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:58 smithi078 ceph-mon[91136]: from='client.53691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:58.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:58 smithi078 ceph-mon[91136]: from='client.73616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:58.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1139006826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:58 smithi146 ceph-mon[103278]: from='client.53691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:58 smithi146 ceph-mon[103278]: from='client.73616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1139006826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:41:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:59 smithi146 ceph-mon[103278]: from='client.73622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:59 smithi146 ceph-mon[103278]: pgmap v4261: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:59 smithi146 ceph-mon[103278]: from='client.53715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:41:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4181263667' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[99959]: from='client.73622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[99959]: pgmap v4261: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[99959]: from='client.53715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4181263667' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[91136]: from='client.73622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[91136]: pgmap v4261: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[91136]: from='client.53715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:41:59.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:41:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4181263667' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:42:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:42:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:00.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:00.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:00.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:01 smithi146 ceph-mon[103278]: pgmap v4262: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[91136]: pgmap v4262: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[99959]: pgmap v4262: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:03 smithi146 ceph-mon[103278]: pgmap v4263: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:03 smithi078 ceph-mon[99959]: pgmap v4263: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:03 smithi078 ceph-mon[91136]: pgmap v4263: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:05 smithi146 ceph-mon[103278]: pgmap v4264: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:05 smithi078 ceph-mon[91136]: pgmap v4264: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:05 smithi078 ceph-mon[99959]: pgmap v4264: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:06 smithi146 ceph-mon[103278]: pgmap v4265: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[91136]: pgmap v4265: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:06.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:06 smithi078 ceph-mon[99959]: pgmap v4265: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:07.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:08 smithi146 ceph-mon[103278]: pgmap v4266: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:08 smithi078 ceph-mon[91136]: pgmap v4266: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:08 smithi078 ceph-mon[99959]: pgmap v4266: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:10 smithi146 ceph-mon[103278]: pgmap v4267: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:10 smithi078 ceph-mon[91136]: pgmap v4267: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:10 smithi078 ceph-mon[99959]: pgmap v4267: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:42:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:42:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:10.715Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:10.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:10.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:12 smithi146 ceph-mon[103278]: pgmap v4268: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:12 smithi078 ceph-mon[91136]: pgmap v4268: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:12 smithi078 ceph-mon[99959]: pgmap v4268: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:14 smithi146 ceph-mon[103278]: pgmap v4269: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:14 smithi078 ceph-mon[91136]: pgmap v4269: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:14 smithi078 ceph-mon[99959]: pgmap v4269: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:16 smithi078 ceph-mon[91136]: pgmap v4270: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:16 smithi078 ceph-mon[99959]: pgmap v4270: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:16 smithi146 ceph-mon[103278]: pgmap v4270: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:18 smithi078 ceph-mon[91136]: pgmap v4271: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:18 smithi078 ceph-mon[99959]: pgmap v4271: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:18 smithi146 ceph-mon[103278]: pgmap v4271: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:20.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:20 smithi146 ceph-mon[103278]: pgmap v4272: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:20 smithi078 ceph-mon[91136]: pgmap v4272: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:20 smithi078 ceph-mon[99959]: pgmap v4272: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:42:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:42:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:20.716Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:20.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:20.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:22.138 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:22 smithi146 ceph-mon[103278]: pgmap v4273: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:22 smithi078 ceph-mon[99959]: pgmap v4273: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:22 smithi078 ceph-mon[91136]: pgmap v4273: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:25.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:24 smithi078 ceph-mon[99959]: pgmap v4274: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:25.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:24 smithi078 ceph-mon[91136]: pgmap v4274: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:24 smithi146 ceph-mon[103278]: pgmap v4274: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[99959]: pgmap v4275: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:26 smithi078 ceph-mon[91136]: pgmap v4275: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:26 smithi146 ceph-mon[103278]: pgmap v4275: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:27.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:27.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:42:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:28 smithi146 ceph-mon[103278]: pgmap v4276: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:28 smithi078 ceph-mon[91136]: pgmap v4276: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:28 smithi078 ceph-mon[99959]: pgmap v4276: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 2h - - 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 2h - - 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 8s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 3s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 8s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 3s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 8s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 8s ago 2h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 3s ago 2h 177M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 8s ago 2h 183M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 8s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 3s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:42:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 8s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 8s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 8s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 8s ago 2h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 3s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 3s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 3s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 3s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 3s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 8s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 8s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:42:29.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:42:29.412 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:42:29.413 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:42:29.414 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:42:29.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:42:29.414 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:42:29.683 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:42:29.684 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:42:29.684 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:42:29.684 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:42:29.684 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:42:29.684 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:42:29.684 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:42:29.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:29 smithi078 ceph-mon[99959]: from='client.73646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:29 smithi078 ceph-mon[99959]: from='client.53733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:29 smithi078 ceph-mon[99959]: from='client.53739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:29 smithi078 ceph-mon[91136]: from='client.73646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:29 smithi078 ceph-mon[91136]: from='client.53733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:29 smithi078 ceph-mon[91136]: from='client.53739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:29 smithi146 ceph-mon[103278]: from='client.73646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:29 smithi146 ceph-mon[103278]: from='client.53733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:29 smithi146 ceph-mon[103278]: from='client.53739 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:29.992 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:42:29.992 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:42:29.992 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:42:29.992 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:42:29.992 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:42:30.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4137168108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[99959]: from='client.53751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[99959]: pgmap v4277: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3233888672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4137168108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[91136]: from='client.53751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[91136]: pgmap v4277: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:30.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3233888672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:30.719 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:42:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:42:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4137168108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:42:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:30 smithi146 ceph-mon[103278]: from='client.53751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:30 smithi146 ceph-mon[103278]: pgmap v4277: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3233888672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:42:31.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:30.716Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:30.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:30.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:32 smithi146 ceph-mon[103278]: pgmap v4278: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:32 smithi078 ceph-mon[91136]: pgmap v4278: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:32 smithi078 ceph-mon[99959]: pgmap v4278: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:34 smithi146 ceph-mon[103278]: pgmap v4279: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:34 smithi078 ceph-mon[91136]: pgmap v4279: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:34 smithi078 ceph-mon[99959]: pgmap v4279: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:36 smithi146 ceph-mon[103278]: pgmap v4280: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:36 smithi078 ceph-mon[91136]: pgmap v4280: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:36 smithi078 ceph-mon[99959]: pgmap v4280: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:38 smithi078 ceph-mon[91136]: pgmap v4281: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:38 smithi078 ceph-mon[99959]: pgmap v4281: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:38 smithi146 ceph-mon[103278]: pgmap v4281: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:40.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:40 smithi078 ceph-mon[99959]: pgmap v4282: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:40.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:40 smithi078 ceph-mon[91136]: pgmap v4282: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:40.918 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:42:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:42:40.918 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:40.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:40.918 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:40.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:40.918 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:40.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:40 smithi146 ceph-mon[103278]: pgmap v4282: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:42.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:42 smithi146 ceph-mon[103278]: pgmap v4283: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:42 smithi078 ceph-mon[99959]: pgmap v4283: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:42 smithi078 ceph-mon[91136]: pgmap v4283: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:44 smithi146 ceph-mon[103278]: pgmap v4284: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:44 smithi078 ceph-mon[91136]: pgmap v4284: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:44 smithi078 ceph-mon[99959]: pgmap v4284: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:46 smithi146 ceph-mon[103278]: pgmap v4285: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[99959]: pgmap v4285: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:42:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:46 smithi078 ceph-mon[91136]: pgmap v4285: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:48.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:48 smithi078 ceph-mon[99959]: pgmap v4286: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:48.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:48 smithi078 ceph-mon[91136]: pgmap v4286: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:48 smithi146 ceph-mon[103278]: pgmap v4286: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:50 smithi146 ceph-mon[103278]: pgmap v4287: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:50 smithi078 ceph-mon[91136]: pgmap v4287: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:42:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:42:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:50 smithi078 ceph-mon[99959]: pgmap v4287: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:50.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:50.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:42:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:42:50.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:42:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:52 smithi078 ceph-mon[91136]: pgmap v4288: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:52 smithi078 ceph-mon[99959]: pgmap v4288: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:52 smithi146 ceph-mon[103278]: pgmap v4288: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:54 smithi078 ceph-mon[91136]: pgmap v4289: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:54 smithi078 ceph-mon[99959]: pgmap v4289: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:55.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:54 smithi146 ceph-mon[103278]: pgmap v4289: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:56 smithi078 ceph-mon[91136]: pgmap v4290: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:56 smithi078 ceph-mon[99959]: pgmap v4290: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:56 smithi146 ceph-mon[103278]: pgmap v4290: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:58 smithi078 ceph-mon[91136]: pgmap v4291: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:58 smithi078 ceph-mon[99959]: pgmap v4291: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:42:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:58 smithi146 ceph-mon[103278]: pgmap v4291: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:00.281 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:43:00.809 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:43:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 2h - - 2024-09-20T22:43:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 2h - - 2024-09-20T22:43:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 19s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:43:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 15s ago 2h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:43:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 19s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:43:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 15s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:43:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 19s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:43:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 19s ago 2h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:43:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 15s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:43:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 19s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:43:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 19s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:43:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 15s ago 2h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:43:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 19s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:43:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 19s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:43:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 19s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:43:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 19s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:43:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 15s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:43:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 15s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:43:00.813 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 15s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:43:00.813 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 15s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:43:00.813 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 15s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:43:00.813 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 19s ago 2h 88.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:43:00.813 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:43:00.813 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 19s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:43:00.814 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:43:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:00 smithi078 ceph-mon[99959]: pgmap v4292: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:43:00] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:43:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:00 smithi078 ceph-mon[91136]: pgmap v4292: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:00.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:00.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:00.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:43:01.133 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:43:01.145 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:00 smithi146 ceph-mon[103278]: pgmap v4292: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:43:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:43:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:43:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:43:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:43:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:43:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:43:01.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:43:01.712 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:43:01.712 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:01.712 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:01.712 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:01.712 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='client.53763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='client.53769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='client.73694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.123 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2588040008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:02.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1833922940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='client.53763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='client.53769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='client.73694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2588040008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:02.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1833922940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='client.53763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='client.53769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='client.73694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2588040008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:02.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1833922940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:03 smithi146 ceph-mon[103278]: from='client.73706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:03 smithi146 ceph-mon[103278]: pgmap v4293: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:03 smithi078 ceph-mon[99959]: from='client.73706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:03 smithi078 ceph-mon[99959]: pgmap v4293: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:03 smithi078 ceph-mon[91136]: from='client.73706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:03 smithi078 ceph-mon[91136]: pgmap v4293: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:05 smithi146 ceph-mon[103278]: pgmap v4294: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:05 smithi078 ceph-mon[91136]: pgmap v4294: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:05 smithi078 ceph-mon[99959]: pgmap v4294: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:06 smithi146 ceph-mon[103278]: pgmap v4295: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[99959]: pgmap v4295: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:06 smithi078 ceph-mon[91136]: pgmap v4295: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:07.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:08 smithi146 ceph-mon[103278]: pgmap v4296: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:08 smithi078 ceph-mon[91136]: pgmap v4296: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:08 smithi078 ceph-mon[99959]: pgmap v4296: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:10 smithi078 ceph-mon[91136]: pgmap v4297: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:10 smithi078 ceph-mon[99959]: pgmap v4297: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:43:10] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:43:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:10.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:10.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:11.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:10.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:10 smithi146 ceph-mon[103278]: pgmap v4297: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:12 smithi078 ceph-mon[91136]: pgmap v4298: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:12 smithi078 ceph-mon[99959]: pgmap v4298: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:12 smithi146 ceph-mon[103278]: pgmap v4298: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:14 smithi078 ceph-mon[91136]: pgmap v4299: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:14 smithi078 ceph-mon[99959]: pgmap v4299: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:14 smithi146 ceph-mon[103278]: pgmap v4299: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:16 smithi078 ceph-mon[91136]: pgmap v4300: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:16 smithi078 ceph-mon[99959]: pgmap v4300: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:16 smithi146 ceph-mon[103278]: pgmap v4300: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:17.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:17.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:18 smithi078 ceph-mon[91136]: pgmap v4301: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:18 smithi078 ceph-mon[99959]: pgmap v4301: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:18 smithi146 ceph-mon[103278]: pgmap v4301: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:20 smithi078 ceph-mon[91136]: pgmap v4302: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:20 smithi078 ceph-mon[99959]: pgmap v4302: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:43:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:43:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:20.720Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:20.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:20.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:20 smithi146 ceph-mon[103278]: pgmap v4302: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:22.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:22 smithi146 ceph-mon[103278]: pgmap v4303: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[91136]: pgmap v4303: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:22 smithi078 ceph-mon[99959]: pgmap v4303: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:24 smithi078 ceph-mon[91136]: pgmap v4304: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:24 smithi078 ceph-mon[99959]: pgmap v4304: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:24 smithi146 ceph-mon[103278]: pgmap v4304: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:26 smithi146 ceph-mon[103278]: pgmap v4305: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[91136]: pgmap v4305: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[99959]: pgmap v4305: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:27.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:27.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:28 smithi078 ceph-mon[91136]: pgmap v4306: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:28 smithi078 ceph-mon[99959]: pgmap v4306: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:28 smithi146 ceph-mon[103278]: pgmap v4306: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:30 smithi146 ceph-mon[103278]: pgmap v4307: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:30 smithi078 ceph-mon[91136]: pgmap v4307: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:43:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:43:31.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:30 smithi078 ceph-mon[99959]: pgmap v4307: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:30.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:30.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:30.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:32.011 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:43:32.537 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 2h - - 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 2h - - 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 11s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 6s ago 2h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 11s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 6s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 11s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 11s ago 2h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 6s ago 2h 179M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 11s ago 2h 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:43:32.538 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 11s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 6s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 11s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 11s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 11s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 11s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 6s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 6s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 6s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 6s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 6s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 11s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 11s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:43:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:43:32.851 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:43:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:43:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:43:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:43:32.852 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:43:32.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:32 smithi078 ceph-mon[99959]: pgmap v4308: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:32.853 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:32 smithi078 ceph-mon[99959]: from='client.53799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:32.853 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:32 smithi078 ceph-mon[91136]: pgmap v4308: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:32.853 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:32 smithi078 ceph-mon[91136]: from='client.53799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:33.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:32 smithi146 ceph-mon[103278]: pgmap v4308: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:33.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:32 smithi146 ceph-mon[103278]: from='client.53799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:43:33.145 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:43:33.146 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:43:33.458 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:43:33.459 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:33.459 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:43:33.459 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:33.459 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:43:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:33 smithi146 ceph-mon[103278]: from='client.53805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:33 smithi146 ceph-mon[103278]: from='client.53811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1265601998' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:33 smithi146 ceph-mon[103278]: from='client.53823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/174942097' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[91136]: from='client.53805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[91136]: from='client.53811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1265601998' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[91136]: from='client.53823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/174942097' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:34.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[99959]: from='client.53805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[99959]: from='client.53811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1265601998' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:43:34.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[99959]: from='client.53823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:34.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/174942097' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:43:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:34 smithi146 ceph-mon[103278]: pgmap v4309: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:34 smithi078 ceph-mon[99959]: pgmap v4309: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:34 smithi078 ceph-mon[91136]: pgmap v4309: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:36 smithi146 ceph-mon[103278]: pgmap v4310: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:36 smithi078 ceph-mon[91136]: pgmap v4310: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:36 smithi078 ceph-mon[99959]: pgmap v4310: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:38 smithi146 ceph-mon[103278]: pgmap v4311: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:38 smithi078 ceph-mon[91136]: pgmap v4311: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:38 smithi078 ceph-mon[99959]: pgmap v4311: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:40 smithi078 ceph-mon[91136]: pgmap v4312: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:40 smithi078 ceph-mon[99959]: pgmap v4312: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:43:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:43:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:40.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:40.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:40.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:40 smithi146 ceph-mon[103278]: pgmap v4312: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[91136]: pgmap v4313: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:42 smithi078 ceph-mon[99959]: pgmap v4313: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:42.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:42.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:42 smithi146 ceph-mon[103278]: pgmap v4313: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:43.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:44.549 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:44 smithi146 ceph-mon[103278]: pgmap v4314: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:44 smithi078 ceph-mon[91136]: pgmap v4314: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:44 smithi078 ceph-mon[99959]: pgmap v4314: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:46 smithi146 ceph-mon[103278]: pgmap v4315: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[91136]: pgmap v4315: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[99959]: pgmap v4315: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:43:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:49.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:48 smithi078 ceph-mon[99959]: pgmap v4316: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:49.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:48 smithi078 ceph-mon[91136]: pgmap v4316: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:48 smithi146 ceph-mon[103278]: pgmap v4316: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:50.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:43:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:43:50.862 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:50.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:50.862 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:50.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:43:50.862 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:43:50.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:43:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:50 smithi146 ceph-mon[103278]: pgmap v4317: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:50 smithi078 ceph-mon[91136]: pgmap v4317: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:50 smithi078 ceph-mon[99959]: pgmap v4317: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:52 smithi146 ceph-mon[103278]: pgmap v4318: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:52 smithi078 ceph-mon[91136]: pgmap v4318: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:52 smithi078 ceph-mon[99959]: pgmap v4318: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:54 smithi146 ceph-mon[103278]: pgmap v4319: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:54 smithi078 ceph-mon[99959]: pgmap v4319: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:54 smithi078 ceph-mon[91136]: pgmap v4319: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:56 smithi146 ceph-mon[103278]: pgmap v4320: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:56 smithi078 ceph-mon[91136]: pgmap v4320: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:56 smithi078 ceph-mon[99959]: pgmap v4320: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:58.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:58.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:58 smithi146 ceph-mon[103278]: pgmap v4321: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:58 smithi078 ceph-mon[99959]: pgmap v4321: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:43:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:58 smithi078 ceph-mon[91136]: pgmap v4321: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:00.967 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:44:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:44:00.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:00.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:00.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:00.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:00.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:00.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:00 smithi146 ceph-mon[103278]: pgmap v4322: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:00 smithi078 ceph-mon[91136]: pgmap v4322: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:00 smithi078 ceph-mon[99959]: pgmap v4322: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:02.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:02 smithi146 ceph-mon[103278]: pgmap v4323: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[99959]: pgmap v4323: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:02 smithi078 ceph-mon[91136]: pgmap v4323: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:03.756 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:44:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:44:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 2h - - 2024-09-20T22:44:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 2h - - 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 2s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 18s ago 2h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 2s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 18s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 2s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 2s ago 2h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 18s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 2s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 2s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 18s ago 2h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 2s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 2s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:44:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 2s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 2s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 18s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 18s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 18s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 18s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 18s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 2s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 2s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:44:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:44:04.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:44:04.595 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:44:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:44:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:44:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:44:04.862 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:44:04.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[91136]: pgmap v4324: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:04.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[91136]: from='client.73754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:04.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[91136]: from='client.53841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:04.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/25922107' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:04.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[99959]: pgmap v4324: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:04.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[99959]: from='client.73754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:04.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[99959]: from='client.53841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:04.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/25922107' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:04.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:04 smithi146 ceph-mon[103278]: pgmap v4324: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:04.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:04 smithi146 ceph-mon[103278]: from='client.73754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:04.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:04 smithi146 ceph-mon[103278]: from='client.53841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:04.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/25922107' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:05.170 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:44:05.170 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:44:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:44:05.170 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:44:05.170 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:44:06.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:05 smithi078 ceph-mon[91136]: from='client.73766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:06.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:05 smithi078 ceph-mon[91136]: from='client.73778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:06.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3832897868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:06.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:05 smithi078 ceph-mon[99959]: from='client.73766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:06.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:05 smithi078 ceph-mon[99959]: from='client.73778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:06.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3832897868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:05 smithi146 ceph-mon[103278]: from='client.73766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:05 smithi146 ceph-mon[103278]: from='client.73778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3832897868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:06 smithi146 ceph-mon[103278]: pgmap v4325: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[91136]: pgmap v4325: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[99959]: pgmap v4325: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:08 smithi078 ceph-mon[91136]: pgmap v4326: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:08 smithi078 ceph-mon[99959]: pgmap v4326: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:08 smithi146 ceph-mon[103278]: pgmap v4326: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:10.963 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:44:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:44:10.964 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:10.724Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:10.964 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:10.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:10.964 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:10.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:10 smithi146 ceph-mon[103278]: pgmap v4327: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:10 smithi078 ceph-mon[99959]: pgmap v4327: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:10 smithi078 ceph-mon[91136]: pgmap v4327: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:12 smithi146 ceph-mon[103278]: pgmap v4328: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:12 smithi078 ceph-mon[91136]: pgmap v4328: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:12 smithi078 ceph-mon[99959]: pgmap v4328: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:15 smithi146 ceph-mon[103278]: pgmap v4329: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:15 smithi078 ceph-mon[91136]: pgmap v4329: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:15 smithi078 ceph-mon[99959]: pgmap v4329: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:17 smithi146 ceph-mon[103278]: pgmap v4330: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:17 smithi078 ceph-mon[99959]: pgmap v4330: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:17 smithi078 ceph-mon[91136]: pgmap v4330: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:19 smithi078 ceph-mon[91136]: pgmap v4331: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:19 smithi078 ceph-mon[99959]: pgmap v4331: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:19 smithi146 ceph-mon[103278]: pgmap v4331: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:20.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:44:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:44:20.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:20.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:20.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:20.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:20.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:20.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:21 smithi078 ceph-mon[91136]: pgmap v4332: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:21 smithi078 ceph-mon[99959]: pgmap v4332: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:21 smithi146 ceph-mon[103278]: pgmap v4332: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:22 smithi146 ceph-mon[103278]: pgmap v4333: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[91136]: pgmap v4333: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:22 smithi078 ceph-mon[99959]: pgmap v4333: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:24.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:24 smithi146 ceph-mon[103278]: pgmap v4334: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:24 smithi078 ceph-mon[91136]: pgmap v4334: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:24 smithi078 ceph-mon[99959]: pgmap v4334: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:26.119 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 22:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T22:44:25.847206942Z level=info msg="Completed cleanup jobs" duration=11.979659ms 2024-09-20T22:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:27 smithi146 ceph-mon[103278]: pgmap v4335: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[91136]: pgmap v4335: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[99959]: pgmap v4335: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:29.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:29 smithi078 ceph-mon[99959]: pgmap v4336: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:29.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:29.424 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:29 smithi078 ceph-mon[91136]: pgmap v4336: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:29.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:29 smithi146 ceph-mon[103278]: pgmap v4336: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:44:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:44:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:30.726Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:30.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:30.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:31.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:31 smithi078 ceph-mon[99959]: pgmap v4337: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:31.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:31 smithi078 ceph-mon[91136]: pgmap v4337: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:31 smithi146 ceph-mon[103278]: pgmap v4337: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:33 smithi146 ceph-mon[103278]: pgmap v4338: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:33 smithi078 ceph-mon[91136]: pgmap v4338: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:33 smithi078 ceph-mon[99959]: pgmap v4338: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:35.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:35 smithi078 ceph-mon[99959]: pgmap v4339: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:35.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:35 smithi078 ceph-mon[91136]: pgmap v4339: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:35.467 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:44:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:35 smithi146 ceph-mon[103278]: pgmap v4339: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 2h - - 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 2h - - 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 14s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 9s ago 2h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 9s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 2h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 9s ago 2h 178M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 2h 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 14s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 9s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 14s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 9s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 9s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 9s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 9s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 9s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:44:35.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 14s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:44:35.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:44:36.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:44:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:44:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:44:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:44:36.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:44:36.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:44:36.576 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:44:36.576 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:44:36.576 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:44:36.576 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:44:36.577 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:44:36.884 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:44:36.884 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:44:36.884 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:44:36.884 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:44:36.884 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:44:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[91136]: from='client.73790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[91136]: from='client.53877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[91136]: pgmap v4340: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[91136]: from='client.73802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1639544363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3400546409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:37.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[99959]: from='client.73790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[99959]: from='client.53877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[99959]: pgmap v4340: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:37.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[99959]: from='client.73802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1639544363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:37.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3400546409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:37 smithi146 ceph-mon[103278]: from='client.73790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:37 smithi146 ceph-mon[103278]: from='client.53877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:37 smithi146 ceph-mon[103278]: pgmap v4340: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:37 smithi146 ceph-mon[103278]: from='client.73802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1639544363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:44:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3400546409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:44:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:38 smithi146 ceph-mon[103278]: from='client.53895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:38 smithi078 ceph-mon[91136]: from='client.53895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:38 smithi078 ceph-mon[99959]: from='client.53895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:39 smithi146 ceph-mon[103278]: pgmap v4341: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:39 smithi078 ceph-mon[91136]: pgmap v4341: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:39 smithi078 ceph-mon[99959]: pgmap v4341: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:44:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:44:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:40.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:40.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:40.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:41 smithi146 ceph-mon[103278]: pgmap v4342: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:41 smithi078 ceph-mon[99959]: pgmap v4342: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:41 smithi078 ceph-mon[91136]: pgmap v4342: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:42 smithi146 ceph-mon[103278]: pgmap v4343: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[99959]: pgmap v4343: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:42 smithi078 ceph-mon[91136]: pgmap v4343: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:44.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:44.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:44 smithi146 ceph-mon[103278]: pgmap v4344: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:44 smithi078 ceph-mon[99959]: pgmap v4344: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:44 smithi078 ceph-mon[91136]: pgmap v4344: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:47.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[91136]: pgmap v4345: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:47.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[99959]: pgmap v4345: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:47.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:46 smithi146 ceph-mon[103278]: pgmap v4345: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:44:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:48 smithi146 ceph-mon[103278]: pgmap v4346: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:48 smithi078 ceph-mon[91136]: pgmap v4346: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:48 smithi078 ceph-mon[99959]: pgmap v4346: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:44:50.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:50 smithi078 ceph-mon[99959]: pgmap v4347: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:50.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:50 smithi078 ceph-mon[91136]: pgmap v4347: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:50.895 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:44:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:44:50.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:50.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:50.896 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:50.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:44:50.896 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:44:50.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:44:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:50 smithi146 ceph-mon[103278]: pgmap v4347: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:52 smithi146 ceph-mon[103278]: pgmap v4348: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:52 smithi078 ceph-mon[91136]: pgmap v4348: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:52 smithi078 ceph-mon[99959]: pgmap v4348: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:54 smithi146 ceph-mon[103278]: pgmap v4349: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:54 smithi078 ceph-mon[91136]: pgmap v4349: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:54 smithi078 ceph-mon[99959]: pgmap v4349: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:57.086 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:56 smithi146 ceph-mon[103278]: pgmap v4350: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:56 smithi078 ceph-mon[99959]: pgmap v4350: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:56 smithi078 ceph-mon[91136]: pgmap v4350: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:58.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:58 smithi146 ceph-mon[103278]: pgmap v4351: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:58 smithi078 ceph-mon[91136]: pgmap v4351: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:44:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:58 smithi078 ceph-mon[99959]: pgmap v4351: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:00 smithi078 ceph-mon[91136]: pgmap v4352: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:00 smithi078 ceph-mon[99959]: pgmap v4352: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:45:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:45:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:00.728Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:00.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:00.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:00 smithi146 ceph-mon[103278]: pgmap v4352: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:02 smithi146 ceph-mon[103278]: pgmap v4353: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[91136]: pgmap v4353: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[99959]: pgmap v4353: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:03.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:05.010 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:04 smithi146 ceph-mon[103278]: pgmap v4354: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:04 smithi078 ceph-mon[91136]: pgmap v4354: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:04 smithi078 ceph-mon[99959]: pgmap v4354: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:06 smithi146 ceph-mon[103278]: pgmap v4355: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[99959]: pgmap v4355: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[91136]: pgmap v4355: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:07.181 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:45:07.700 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 2h - - 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 5s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 1s ago 2h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 5s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 1s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 5s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 5s ago 2h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 1s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 5s ago 2h 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 5s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 1s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 5s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:45:07.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 5s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 5s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 5s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 1s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 1s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 1s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 1s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 1s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 5s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 5s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:45:07.702 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:45:08.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:08.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:08.285 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:45:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:45:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:45:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:45:08.596 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:45:08.596 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:08.596 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[99959]: from='client.53907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[99959]: from='client.53913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[99959]: from='client.53919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[99959]: pgmap v4356: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1760351797' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1295971202' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[91136]: from='client.53907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[91136]: from='client.53913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[91136]: from='client.53919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[91136]: pgmap v4356: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:09.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1760351797' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:09.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1295971202' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:08 smithi146 ceph-mon[103278]: from='client.53907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:08 smithi146 ceph-mon[103278]: from='client.53913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:08 smithi146 ceph-mon[103278]: from='client.53919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:08 smithi146 ceph-mon[103278]: pgmap v4356: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1760351797' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1295971202' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:09 smithi146 ceph-mon[103278]: from='client.73850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:09 smithi078 ceph-mon[91136]: from='client.73850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:09 smithi078 ceph-mon[99959]: from='client.73850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:10.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:10 smithi078 ceph-mon[99959]: pgmap v4357: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:10.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:10 smithi078 ceph-mon[91136]: pgmap v4357: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:10.940 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:45:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:45:10.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:10.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:10.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:10.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:10.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:10.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:10 smithi146 ceph-mon[103278]: pgmap v4357: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:12 smithi146 ceph-mon[103278]: pgmap v4358: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:12 smithi078 ceph-mon[91136]: pgmap v4358: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:12 smithi078 ceph-mon[99959]: pgmap v4358: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:14 smithi146 ceph-mon[103278]: pgmap v4359: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:14 smithi078 ceph-mon[91136]: pgmap v4359: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:14 smithi078 ceph-mon[99959]: pgmap v4359: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:17.044 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:16 smithi078 ceph-mon[99959]: pgmap v4360: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:17.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:16 smithi078 ceph-mon[91136]: pgmap v4360: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:16 smithi146 ceph-mon[103278]: pgmap v4360: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:18 smithi146 ceph-mon[103278]: pgmap v4361: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:18 smithi078 ceph-mon[99959]: pgmap v4361: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:18 smithi078 ceph-mon[91136]: pgmap v4361: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:20 smithi078 ceph-mon[91136]: pgmap v4362: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:20 smithi078 ceph-mon[99959]: pgmap v4362: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:45:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:45:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:20.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:20.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:20.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:20 smithi146 ceph-mon[103278]: pgmap v4362: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:22 smithi146 ceph-mon[103278]: pgmap v4363: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[91136]: pgmap v4363: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[99959]: pgmap v4363: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:25.169 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:24 smithi146 ceph-mon[103278]: pgmap v4364: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:24 smithi078 ceph-mon[91136]: pgmap v4364: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:24 smithi078 ceph-mon[99959]: pgmap v4364: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:26 smithi146 ceph-mon[103278]: pgmap v4365: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[91136]: pgmap v4365: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[99959]: pgmap v4365: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:28 smithi146 ceph-mon[103278]: pgmap v4366: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:28 smithi078 ceph-mon[91136]: pgmap v4366: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:28 smithi078 ceph-mon[99959]: pgmap v4366: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:30.946 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:45:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:45:30.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:30.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:30.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:30.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:30.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:30.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:30 smithi146 ceph-mon[103278]: pgmap v4367: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:30 smithi078 ceph-mon[91136]: pgmap v4367: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:30 smithi078 ceph-mon[99959]: pgmap v4367: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:32 smithi146 ceph-mon[103278]: pgmap v4368: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:32 smithi078 ceph-mon[91136]: pgmap v4368: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:32 smithi078 ceph-mon[99959]: pgmap v4368: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:34 smithi146 ceph-mon[103278]: pgmap v4369: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:34 smithi078 ceph-mon[91136]: pgmap v4369: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:34 smithi078 ceph-mon[99959]: pgmap v4369: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:36 smithi146 ceph-mon[103278]: pgmap v4370: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:36 smithi078 ceph-mon[91136]: pgmap v4370: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:36 smithi078 ceph-mon[99959]: pgmap v4370: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:38.891 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:45:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:38 smithi146 ceph-mon[103278]: pgmap v4371: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:38 smithi078 ceph-mon[91136]: pgmap v4371: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:38 smithi078 ceph-mon[99959]: pgmap v4371: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:39.417 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 2h - - 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 2h - - 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 17s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 12s ago 2h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 12s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 2h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 12s ago 2h 180M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 17s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 12s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:45:39.418 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 12s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 12s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 12s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 12s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 12s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:45:39.419 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:45:39.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:45:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:45:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:45:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:45:39.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:45:40.008 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:45:40.009 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:45:40.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:39 smithi078 ceph-mon[99959]: from='client.73862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:40.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:39 smithi078 ceph-mon[99959]: from='client.53949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:40.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1934737349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:40.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:39 smithi078 ceph-mon[91136]: from='client.73862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:40.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:39 smithi078 ceph-mon[91136]: from='client.53949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:40.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1934737349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:39 smithi146 ceph-mon[103278]: from='client.73862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:39 smithi146 ceph-mon[103278]: from='client.53949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1934737349' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:45:40.321 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:45:40.321 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:40.321 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:45:40.321 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:40.322 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:45:40.968 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:45:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:45:40.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:40.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:40.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:40.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:40.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:40.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:40 smithi146 ceph-mon[103278]: from='client.53955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:40 smithi146 ceph-mon[103278]: pgmap v4372: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:40 smithi146 ceph-mon[103278]: from='client.73886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/833639465' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[91136]: from='client.53955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[91136]: pgmap v4372: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[91136]: from='client.73886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/833639465' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[99959]: from='client.53955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[99959]: pgmap v4372: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[99959]: from='client.73886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/833639465' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:45:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:43 smithi146 ceph-mon[103278]: pgmap v4373: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[99959]: pgmap v4373: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[91136]: pgmap v4373: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:45.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:45 smithi146 ceph-mon[103278]: pgmap v4374: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:45 smithi078 ceph-mon[91136]: pgmap v4374: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:45 smithi078 ceph-mon[99959]: pgmap v4374: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:47.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:47 smithi146 ceph-mon[103278]: pgmap v4375: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:47.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[99959]: pgmap v4375: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[91136]: pgmap v4375: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:45:48.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:49 smithi146 ceph-mon[103278]: pgmap v4376: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:49 smithi078 ceph-mon[91136]: pgmap v4376: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:49 smithi078 ceph-mon[99959]: pgmap v4376: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:45:50] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:45:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:50.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:50.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:45:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:45:50.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:45:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:51 smithi146 ceph-mon[103278]: pgmap v4377: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:51 smithi078 ceph-mon[91136]: pgmap v4377: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:51 smithi078 ceph-mon[99959]: pgmap v4377: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:53.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:53 smithi078 ceph-mon[91136]: pgmap v4378: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:53.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:53 smithi078 ceph-mon[99959]: pgmap v4378: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:53 smithi146 ceph-mon[103278]: pgmap v4378: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:55 smithi146 ceph-mon[103278]: pgmap v4379: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:55 smithi078 ceph-mon[91136]: pgmap v4379: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:55 smithi078 ceph-mon[99959]: pgmap v4379: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:57 smithi146 ceph-mon[103278]: pgmap v4380: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:57 smithi078 ceph-mon[91136]: pgmap v4380: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:57 smithi078 ceph-mon[99959]: pgmap v4380: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:59 smithi146 ceph-mon[103278]: pgmap v4381: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:59 smithi078 ceph-mon[91136]: pgmap v4381: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:45:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:59 smithi078 ceph-mon[99959]: pgmap v4381: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:46:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:46:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:00.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:00.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:00.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:01 smithi146 ceph-mon[103278]: pgmap v4382: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:01 smithi078 ceph-mon[91136]: pgmap v4382: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:01 smithi078 ceph-mon[99959]: pgmap v4382: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:03 smithi146 ceph-mon[103278]: pgmap v4383: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[91136]: pgmap v4383: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[99959]: pgmap v4383: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:05 smithi146 ceph-mon[103278]: pgmap v4384: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:05 smithi078 ceph-mon[91136]: pgmap v4384: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:05 smithi078 ceph-mon[99959]: pgmap v4384: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:07.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:07 smithi146 ceph-mon[103278]: pgmap v4385: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:07.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.404 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[91136]: pgmap v4385: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[99959]: pgmap v4385: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:09 smithi146 ceph-mon[103278]: pgmap v4386: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:09 smithi078 ceph-mon[99959]: pgmap v4386: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:09 smithi078 ceph-mon[91136]: pgmap v4386: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:10.621 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:46:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:46:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:46:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:10.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:10.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:10.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 2h - - 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 2h - - 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 8s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 4s ago 2h 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 8s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:46:11.151 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 4s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 8s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 8s ago 2h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 4s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 8s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 8s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 4s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 8s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 8s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 8s ago 2h 67.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 8s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:46:11.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 4s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 4s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 4s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 4s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 4s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 8s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 8s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:46:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:46:11.460 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:46:11.460 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:46:11.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:46:11.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:46:11.461 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:46:11.462 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:11 smithi078 ceph-mon[99959]: pgmap v4387: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:11.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:11 smithi078 ceph-mon[91136]: pgmap v4387: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:11 smithi146 ceph-mon[103278]: pgmap v4387: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:46:11.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:46:11.731 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:46:11.731 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:46:11.731 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:46:11.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:46:12.042 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:46:12.042 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:12.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:12.042 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:12.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:12 smithi146 ceph-mon[103278]: from='client.73898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:12 smithi146 ceph-mon[103278]: from='client.53985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:12 smithi146 ceph-mon[103278]: from='client.53991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2240599519' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/482473440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[91136]: from='client.73898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[91136]: from='client.53985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[91136]: from='client.53991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2240599519' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/482473440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[99959]: from='client.73898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[99959]: from='client.53985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[99959]: from='client.53991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2240599519' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/482473440' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:13 smithi146 ceph-mon[103278]: from='client.54003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:13 smithi146 ceph-mon[103278]: pgmap v4388: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:13 smithi078 ceph-mon[91136]: from='client.54003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:13 smithi078 ceph-mon[91136]: pgmap v4388: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:13 smithi078 ceph-mon[99959]: from='client.54003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:13 smithi078 ceph-mon[99959]: pgmap v4388: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:14.461 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:14.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:15.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:15 smithi078 ceph-mon[91136]: pgmap v4389: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:15.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:15 smithi078 ceph-mon[99959]: pgmap v4389: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:15 smithi146 ceph-mon[103278]: pgmap v4389: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:17 smithi078 ceph-mon[91136]: pgmap v4390: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:17 smithi078 ceph-mon[99959]: pgmap v4390: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:17 smithi146 ceph-mon[103278]: pgmap v4390: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:19 smithi146 ceph-mon[103278]: pgmap v4391: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:19 smithi078 ceph-mon[91136]: pgmap v4391: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:19 smithi078 ceph-mon[99959]: pgmap v4391: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:46:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T22:46:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:20.736Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:20.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:20.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:21 smithi146 ceph-mon[103278]: pgmap v4392: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:21 smithi078 ceph-mon[91136]: pgmap v4392: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:21 smithi078 ceph-mon[99959]: pgmap v4392: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:23 smithi146 ceph-mon[103278]: pgmap v4393: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[99959]: pgmap v4393: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[91136]: pgmap v4393: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:25 smithi146 ceph-mon[103278]: pgmap v4394: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:25 smithi078 ceph-mon[99959]: pgmap v4394: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:25 smithi078 ceph-mon[91136]: pgmap v4394: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[91136]: pgmap v4395: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[99959]: pgmap v4395: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:27 smithi146 ceph-mon[103278]: pgmap v4395: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:29.360 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:29 smithi146 ceph-mon[103278]: pgmap v4396: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:29.360 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:29 smithi078 ceph-mon[91136]: pgmap v4396: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:29 smithi078 ceph-mon[99959]: pgmap v4396: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:46:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:46:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:30.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:30.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:30.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:31 smithi146 ceph-mon[103278]: pgmap v4397: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:31 smithi078 ceph-mon[91136]: pgmap v4397: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:31 smithi078 ceph-mon[99959]: pgmap v4397: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:32 smithi146 ceph-mon[103278]: pgmap v4398: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:32 smithi078 ceph-mon[91136]: pgmap v4398: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:32 smithi078 ceph-mon[99959]: pgmap v4398: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:34.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:34 smithi078 ceph-mon[99959]: pgmap v4399: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:34.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:34 smithi078 ceph-mon[91136]: pgmap v4399: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:34 smithi146 ceph-mon[103278]: pgmap v4399: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:36 smithi146 ceph-mon[103278]: pgmap v4400: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:36 smithi078 ceph-mon[91136]: pgmap v4400: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:36 smithi078 ceph-mon[99959]: pgmap v4400: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:38 smithi146 ceph-mon[103278]: pgmap v4401: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:38 smithi078 ceph-mon[99959]: pgmap v4401: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:38 smithi078 ceph-mon[91136]: pgmap v4401: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:40 smithi078 ceph-mon[91136]: pgmap v4402: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:40 smithi078 ceph-mon[99959]: pgmap v4402: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:46:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:46:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:40.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:40.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:40.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:40 smithi146 ceph-mon[103278]: pgmap v4402: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:42.333 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 2h - - 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 2h - - 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 0s ago 2h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 15s ago 2h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 0s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:46:42.875 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 15s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 0s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 0s ago 2h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 15s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 0s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 0s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 15s ago 2h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 0s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 0s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 0s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 0s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 15s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 15s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 15s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 15s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:46:42.876 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 15s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:46:42.877 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 0s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:46:42.877 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:46:42.877 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 0s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:46:42.877 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:46:43.185 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:46:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:46:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:46:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:46:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:46:43.187 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:46:43.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[91136]: pgmap v4403: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:43.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[99959]: pgmap v4403: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:43.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:42 smithi146 ceph-mon[103278]: pgmap v4403: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:46:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:46:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:46:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:46:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:46:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:46:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:46:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:46:43.769 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:46:43.769 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:46:43.769 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[91136]: from='client.73934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[91136]: from='client.54021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[91136]: from='client.54027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/188482892' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1202912240' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[99959]: from='client.73934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[99959]: from='client.54021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[99959]: from='client.54027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/188482892' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1202912240' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:43 smithi146 ceph-mon[103278]: from='client.73934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:43 smithi146 ceph-mon[103278]: from='client.54021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:43 smithi146 ceph-mon[103278]: from='client.54027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/188482892' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1202912240' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:46:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:44 smithi146 ceph-mon[103278]: from='client.73958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:44 smithi146 ceph-mon[103278]: pgmap v4404: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:44 smithi078 ceph-mon[91136]: from='client.73958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:44 smithi078 ceph-mon[91136]: pgmap v4404: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:44 smithi078 ceph-mon[99959]: from='client.73958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:44 smithi078 ceph-mon[99959]: pgmap v4404: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:46 smithi078 ceph-mon[99959]: pgmap v4405: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:46 smithi078 ceph-mon[91136]: pgmap v4405: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:46 smithi146 ceph-mon[103278]: pgmap v4405: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:48.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:48.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:46:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:49 smithi078 ceph-mon[91136]: pgmap v4406: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:49 smithi078 ceph-mon[99959]: pgmap v4406: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:49.288 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:49 smithi146 ceph-mon[103278]: pgmap v4406: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:46:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:46:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:50.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:50.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:46:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:46:50.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:46:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:51 smithi146 ceph-mon[103278]: pgmap v4407: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:51 smithi078 ceph-mon[91136]: pgmap v4407: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:51 smithi078 ceph-mon[99959]: pgmap v4407: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:53 smithi146 ceph-mon[103278]: pgmap v4408: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:53 smithi078 ceph-mon[91136]: pgmap v4408: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:53 smithi078 ceph-mon[99959]: pgmap v4408: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:55.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:55 smithi078 ceph-mon[99959]: pgmap v4409: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:55.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:55 smithi078 ceph-mon[91136]: pgmap v4409: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:55 smithi146 ceph-mon[103278]: pgmap v4409: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:57.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:57 smithi146 ceph-mon[103278]: pgmap v4410: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:57 smithi078 ceph-mon[91136]: pgmap v4410: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:57 smithi078 ceph-mon[99959]: pgmap v4410: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:58.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:59 smithi146 ceph-mon[103278]: pgmap v4411: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:59 smithi078 ceph-mon[91136]: pgmap v4411: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:46:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:59 smithi078 ceph-mon[99959]: pgmap v4411: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:47:00] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:47:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:00.739Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:00.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:00.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:01 smithi146 ceph-mon[103278]: pgmap v4412: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:01 smithi078 ceph-mon[91136]: pgmap v4412: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:01 smithi078 ceph-mon[99959]: pgmap v4412: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:03 smithi146 ceph-mon[103278]: pgmap v4413: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[91136]: pgmap v4413: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[99959]: pgmap v4413: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:05 smithi146 ceph-mon[103278]: pgmap v4414: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:05 smithi078 ceph-mon[91136]: pgmap v4414: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:05 smithi078 ceph-mon[99959]: pgmap v4414: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:07 smithi146 ceph-mon[103278]: pgmap v4415: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:07 smithi078 ceph-mon[99959]: pgmap v4415: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:07 smithi078 ceph-mon[91136]: pgmap v4415: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:08.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:08.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:09 smithi146 ceph-mon[103278]: pgmap v4416: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:09 smithi078 ceph-mon[91136]: pgmap v4416: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:09 smithi078 ceph-mon[99959]: pgmap v4416: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:47:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:47:10] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:47:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:10.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:10.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:10.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:11 smithi146 ceph-mon[103278]: pgmap v4417: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:11 smithi078 ceph-mon[91136]: pgmap v4417: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:11 smithi078 ceph-mon[99959]: pgmap v4417: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:13 smithi146 ceph-mon[103278]: pgmap v4418: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:13 smithi078 ceph-mon[91136]: pgmap v4418: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:13 smithi078 ceph-mon[99959]: pgmap v4418: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:14.059 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:47:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:14.593 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:47:14.593 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 2h - - 2024-09-20T22:47:14.593 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 2h - - 2024-09-20T22:47:14.593 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 12s ago 2h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:47:14.593 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 7s ago 2h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 12s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 7s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 12s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 12s ago 2h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 7s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 12s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 12s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 7s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 12s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 12s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 12s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 12s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 7s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:47:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 7s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 7s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 7s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 7s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 12s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 7s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 12s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:47:14.595 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 7s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:14.914 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:47:14.915 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:47:15.189 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:47:15.190 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:47:15.513 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:47:15.514 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:15.514 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:15.514 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:15.514 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:15.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:15 smithi078 ceph-mon[99959]: pgmap v4419: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:15.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:15 smithi078 ceph-mon[99959]: from='client.73970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:15.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1226088913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:15.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:15 smithi078 ceph-mon[91136]: pgmap v4419: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:15.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:15 smithi078 ceph-mon[91136]: from='client.73970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:15.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1226088913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:15 smithi146 ceph-mon[103278]: pgmap v4419: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:15 smithi146 ceph-mon[103278]: from='client.73970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1226088913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:16.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:16 smithi078 ceph-mon[91136]: from='client.73976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:16.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:16 smithi078 ceph-mon[91136]: from='client.73982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:16.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1138720270' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:16.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:16 smithi078 ceph-mon[99959]: from='client.73976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:16.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:16 smithi078 ceph-mon[99959]: from='client.73982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:16.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1138720270' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:16 smithi146 ceph-mon[103278]: from='client.73976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:16 smithi146 ceph-mon[103278]: from='client.73982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1138720270' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:17 smithi146 ceph-mon[103278]: from='client.54075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:17 smithi146 ceph-mon[103278]: pgmap v4420: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:17 smithi078 ceph-mon[91136]: from='client.54075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:17 smithi078 ceph-mon[91136]: pgmap v4420: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:17 smithi078 ceph-mon[99959]: from='client.54075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:17 smithi078 ceph-mon[99959]: pgmap v4420: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:18 smithi146 ceph-mon[103278]: pgmap v4421: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:18 smithi078 ceph-mon[91136]: pgmap v4421: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:18 smithi078 ceph-mon[99959]: pgmap v4421: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:20 smithi078 ceph-mon[91136]: pgmap v4422: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:20 smithi078 ceph-mon[99959]: pgmap v4422: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:47:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:47:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:20.741Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:20.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:21.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:20.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:20 smithi146 ceph-mon[103278]: pgmap v4422: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[91136]: pgmap v4423: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[99959]: pgmap v4423: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:22 smithi146 ceph-mon[103278]: pgmap v4423: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:24 smithi146 ceph-mon[103278]: pgmap v4424: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:24 smithi078 ceph-mon[91136]: pgmap v4424: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:24 smithi078 ceph-mon[99959]: pgmap v4424: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:27.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:26 smithi078 ceph-mon[99959]: pgmap v4425: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:27.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:26 smithi078 ceph-mon[91136]: pgmap v4425: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:26 smithi146 ceph-mon[103278]: pgmap v4425: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:28 smithi146 ceph-mon[103278]: pgmap v4426: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:28 smithi078 ceph-mon[91136]: pgmap v4426: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:28 smithi078 ceph-mon[99959]: pgmap v4426: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:30.869 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:47:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:47:30.870 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:30.742Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:30.870 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:30.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:30.870 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:30.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:30 smithi146 ceph-mon[103278]: pgmap v4427: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:30 smithi078 ceph-mon[91136]: pgmap v4427: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:30 smithi078 ceph-mon[99959]: pgmap v4427: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:32 smithi146 ceph-mon[103278]: pgmap v4428: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:32 smithi078 ceph-mon[91136]: pgmap v4428: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:32 smithi078 ceph-mon[99959]: pgmap v4428: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:35.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:34 smithi078 ceph-mon[91136]: pgmap v4429: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:35.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:34 smithi078 ceph-mon[99959]: pgmap v4429: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:34 smithi146 ceph-mon[103278]: pgmap v4429: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:37.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:36 smithi078 ceph-mon[91136]: pgmap v4430: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:37.138 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:36 smithi078 ceph-mon[99959]: pgmap v4430: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:36 smithi146 ceph-mon[103278]: pgmap v4430: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:38 smithi146 ceph-mon[103278]: pgmap v4431: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:38 smithi078 ceph-mon[91136]: pgmap v4431: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:38 smithi078 ceph-mon[99959]: pgmap v4431: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:40.892 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:47:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:47:40.892 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:40.743Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:40.892 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:40.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:40.892 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:40.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:40 smithi146 ceph-mon[103278]: pgmap v4432: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:40 smithi078 ceph-mon[91136]: pgmap v4432: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:40 smithi078 ceph-mon[99959]: pgmap v4432: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:42 smithi146 ceph-mon[103278]: pgmap v4433: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[91136]: pgmap v4433: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[99959]: pgmap v4433: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:44 smithi146 ceph-mon[103278]: pgmap v4434: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:44 smithi078 ceph-mon[91136]: pgmap v4434: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:44 smithi078 ceph-mon[99959]: pgmap v4434: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:45.809 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:47:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:47:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 2h - - 2024-09-20T22:47:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 2h - - 2024-09-20T22:47:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 3s ago 2h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:47:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 19s ago 2h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:47:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 3s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 19s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 3s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 3s ago 2h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 19s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 3s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 3s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 19s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 3s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 3s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 3s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 3s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 19s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 19s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:47:46.335 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 19s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:47:46.336 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 19s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:47:46.336 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 19s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:47:46.336 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 3s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:47:46.336 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:47:46.336 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 3s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:47:46.336 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:47:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:47:46.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:47:46.916 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:47:46.917 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:47:47.224 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:47:47.224 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:47:47.224 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:47:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[91136]: from='client.74006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[91136]: pgmap v4435: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:47.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[91136]: from='client.54093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:47.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/999947240' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:47.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[99959]: from='client.74006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:47.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[99959]: pgmap v4435: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:47.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[99959]: from='client.54093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:47.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/999947240' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:46 smithi146 ceph-mon[103278]: from='client.74006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:46 smithi146 ceph-mon[103278]: pgmap v4435: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:46 smithi146 ceph-mon[103278]: from='client.54093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/999947240' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:47:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='client.54099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='client.54111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1696602723' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='client.54099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='client.54111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1696602723' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='client.54099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:48.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:48.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='client.54111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:48.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1696602723' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:47:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:47:49.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:48 smithi146 ceph-mon[103278]: pgmap v4436: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:48 smithi078 ceph-mon[91136]: pgmap v4436: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:48 smithi078 ceph-mon[99959]: pgmap v4436: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:50.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:47:50] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:47:50.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:50.744Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:50.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:50.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:47:50.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:47:50.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:47:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:50 smithi146 ceph-mon[103278]: pgmap v4437: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:50 smithi078 ceph-mon[91136]: pgmap v4437: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:50 smithi078 ceph-mon[99959]: pgmap v4437: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:52 smithi146 ceph-mon[103278]: pgmap v4438: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:52 smithi078 ceph-mon[91136]: pgmap v4438: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:52 smithi078 ceph-mon[99959]: pgmap v4438: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:54 smithi146 ceph-mon[103278]: pgmap v4439: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:54 smithi078 ceph-mon[99959]: pgmap v4439: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:54 smithi078 ceph-mon[91136]: pgmap v4439: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:57.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:56 smithi078 ceph-mon[91136]: pgmap v4440: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:57.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:56 smithi078 ceph-mon[99959]: pgmap v4440: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:56 smithi146 ceph-mon[103278]: pgmap v4440: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:58 smithi146 ceph-mon[103278]: pgmap v4441: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:47:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:58 smithi078 ceph-mon[91136]: pgmap v4441: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:47:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:47:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:58 smithi078 ceph-mon[99959]: pgmap v4441: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:47:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:00.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:48:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:48:00.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:00.745Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:00.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:00.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:00.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:00.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:00 smithi146 ceph-mon[103278]: pgmap v4442: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:00 smithi078 ceph-mon[91136]: pgmap v4442: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:00 smithi078 ceph-mon[99959]: pgmap v4442: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:02 smithi146 ceph-mon[103278]: pgmap v4443: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:02 smithi078 ceph-mon[91136]: pgmap v4443: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:02 smithi078 ceph-mon[99959]: pgmap v4443: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:04 smithi146 ceph-mon[103278]: pgmap v4444: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:04 smithi078 ceph-mon[91136]: pgmap v4444: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:04 smithi078 ceph-mon[99959]: pgmap v4444: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:06 smithi146 ceph-mon[103278]: pgmap v4445: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:06 smithi078 ceph-mon[91136]: pgmap v4445: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:06 smithi078 ceph-mon[99959]: pgmap v4445: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:08 smithi146 ceph-mon[103278]: pgmap v4446: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:08 smithi078 ceph-mon[91136]: pgmap v4446: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:08 smithi078 ceph-mon[99959]: pgmap v4446: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:10.990 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:48:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:48:10.990 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:10.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:10.990 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:10.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:10.990 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:10.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:11.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:10 smithi146 ceph-mon[103278]: pgmap v4447: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:10 smithi078 ceph-mon[91136]: pgmap v4447: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:10 smithi078 ceph-mon[99959]: pgmap v4447: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:12 smithi146 ceph-mon[103278]: pgmap v4448: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:12 smithi078 ceph-mon[99959]: pgmap v4448: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:12 smithi078 ceph-mon[91136]: pgmap v4448: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:15 smithi146 ceph-mon[103278]: pgmap v4449: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:14 smithi078 ceph-mon[91136]: pgmap v4449: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:14 smithi078 ceph-mon[99959]: pgmap v4449: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:17.145 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:17 smithi078 ceph-mon[99959]: pgmap v4450: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:17.145 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:17 smithi078 ceph-mon[91136]: pgmap v4450: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:17 smithi146 ceph-mon[103278]: pgmap v4450: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:17.520 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:48:18.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:18.069 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:18.069 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 2h - - 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 2h - - 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 15s ago 2h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 10s ago 2h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 15s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 10s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 15s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 15s ago 2h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 10s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 15s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 15s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 10s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 15s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:48:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 15s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 15s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 15s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 10s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 10s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 10s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 10s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 10s ago 2h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 15s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 10s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 15s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:48:18.071 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 10s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:48:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:18.395 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:48:18.395 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:18.396 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:48:18.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:48:18.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:48:18.397 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:48:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:48:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:48:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:48:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:48:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:48:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:48:18.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:48:19.014 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:48:19.014 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:19.014 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:19.014 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:19.014 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:19 smithi146 ceph-mon[103278]: from='client.74042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:19 smithi146 ceph-mon[103278]: from='client.54129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:19 smithi146 ceph-mon[103278]: pgmap v4451: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:19 smithi146 ceph-mon[103278]: from='client.54135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4062003659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[91136]: from='client.74042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[91136]: from='client.54129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[91136]: pgmap v4451: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[91136]: from='client.54135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4062003659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[99959]: from='client.74042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[99959]: from='client.54129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[99959]: pgmap v4451: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[99959]: from='client.54135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:19.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4062003659' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-mon[91136]: from='client.54147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/734489256' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-mon[99959]: from='client.54147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/734489256' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:20 smithi146 ceph-mon[103278]: from='client.54147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/734489256' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:21.019 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:48:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:48:21.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:20.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:21.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:20.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:21.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:20.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:21 smithi078 ceph-mon[91136]: pgmap v4452: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:21 smithi078 ceph-mon[99959]: pgmap v4452: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:21 smithi146 ceph-mon[103278]: pgmap v4452: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:23 smithi078 ceph-mon[91136]: pgmap v4453: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:23 smithi078 ceph-mon[99959]: pgmap v4453: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:23 smithi146 ceph-mon[103278]: pgmap v4453: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:25 smithi146 ceph-mon[103278]: pgmap v4454: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:25 smithi078 ceph-mon[99959]: pgmap v4454: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:25 smithi078 ceph-mon[91136]: pgmap v4454: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:27 smithi146 ceph-mon[103278]: pgmap v4455: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:27 smithi078 ceph-mon[91136]: pgmap v4455: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:27 smithi078 ceph-mon[99959]: pgmap v4455: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:28.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:28.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:29 smithi146 ceph-mon[103278]: pgmap v4456: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:29 smithi078 ceph-mon[91136]: pgmap v4456: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:29 smithi078 ceph-mon[99959]: pgmap v4456: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:48:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:48:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:30.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:30.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:30.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:31.325 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:31 smithi146 ceph-mon[103278]: pgmap v4457: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:31 smithi078 ceph-mon[91136]: pgmap v4457: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:31 smithi078 ceph-mon[99959]: pgmap v4457: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:33 smithi146 ceph-mon[103278]: pgmap v4458: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:33 smithi078 ceph-mon[91136]: pgmap v4458: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:33 smithi078 ceph-mon[99959]: pgmap v4458: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:35 smithi146 ceph-mon[103278]: pgmap v4459: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:35 smithi078 ceph-mon[99959]: pgmap v4459: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:35.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:35 smithi078 ceph-mon[91136]: pgmap v4459: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:37.424 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:37 smithi078 ceph-mon[91136]: pgmap v4460: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:37.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:37 smithi078 ceph-mon[99959]: pgmap v4460: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:37 smithi146 ceph-mon[103278]: pgmap v4460: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:38.461 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:38.461 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:39 smithi078 ceph-mon[91136]: pgmap v4461: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:39 smithi078 ceph-mon[99959]: pgmap v4461: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:39 smithi146 ceph-mon[103278]: pgmap v4461: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:48:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:48:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:40.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:40.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:40.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:41 smithi146 ceph-mon[103278]: pgmap v4462: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:41 smithi078 ceph-mon[91136]: pgmap v4462: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:41 smithi078 ceph-mon[99959]: pgmap v4462: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:43 smithi146 ceph-mon[103278]: pgmap v4463: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:43 smithi078 ceph-mon[91136]: pgmap v4463: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:43 smithi078 ceph-mon[99959]: pgmap v4463: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:44 smithi146 ceph-mon[103278]: pgmap v4464: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[99959]: pgmap v4464: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:44 smithi078 ceph-mon[91136]: pgmap v4464: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:47.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:46 smithi078 ceph-mon[99959]: pgmap v4465: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:47.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:46 smithi078 ceph-mon[91136]: pgmap v4465: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:46 smithi146 ceph-mon[103278]: pgmap v4465: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:48:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:48 smithi146 ceph-mon[103278]: pgmap v4466: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:48 smithi078 ceph-mon[99959]: pgmap v4466: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:48 smithi078 ceph-mon[91136]: pgmap v4466: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:49.314 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:48:49.839 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:48:49.839 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 2h - - 2024-09-20T22:48:49.839 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 2h - - 2024-09-20T22:48:49.839 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 6s ago 2h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:48:49.839 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 2s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:48:49.839 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 6s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 2s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 6s ago 2h 570M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 6s ago 2h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 2s ago 2h 181M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 6s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 6s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 2s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 6s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 6s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 6s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 6s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:48:49.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 2s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 2s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 2s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 2s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 2s ago 2h 81.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 6s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 6s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:48:49.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:48:50.148 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:48:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:48:50.150 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:48:50.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:48:50.418 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:48:50.419 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:48:50.727 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:48:50.727 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:50.728 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:48:50.728 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:50.728 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:48:50.728 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:48:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:48:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:50 smithi146 ceph-mon[103278]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:50 smithi146 ceph-mon[103278]: from='client.74084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:50 smithi146 ceph-mon[103278]: from='client.54171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:50 smithi146 ceph-mon[103278]: pgmap v4467: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3766317386' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3387114944' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[91136]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[91136]: from='client.74084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[91136]: from='client.54171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[91136]: pgmap v4467: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3766317386' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3387114944' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:50.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:50.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:48:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:48:50.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:48:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[99959]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[99959]: from='client.74084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[99959]: from='client.54171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[99959]: pgmap v4467: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3766317386' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:48:51.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3387114944' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:48:51.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:51 smithi146 ceph-mon[103278]: from='client.74102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:51 smithi078 ceph-mon[91136]: from='client.74102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:51 smithi078 ceph-mon[99959]: from='client.74102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:52 smithi146 ceph-mon[103278]: pgmap v4468: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:52 smithi078 ceph-mon[91136]: pgmap v4468: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:52 smithi078 ceph-mon[99959]: pgmap v4468: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:54 smithi146 ceph-mon[103278]: pgmap v4469: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:54 smithi078 ceph-mon[91136]: pgmap v4469: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:54 smithi078 ceph-mon[99959]: pgmap v4469: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:56 smithi146 ceph-mon[103278]: pgmap v4470: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:56 smithi078 ceph-mon[91136]: pgmap v4470: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:56 smithi078 ceph-mon[99959]: pgmap v4470: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:58.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:59.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:58 smithi078 ceph-mon[99959]: pgmap v4471: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:59.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:59.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:58 smithi078 ceph-mon[91136]: pgmap v4471: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:59.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:48:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:58 smithi146 ceph-mon[103278]: pgmap v4471: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:00.744 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:49:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:49:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:00 smithi078 ceph-mon[91136]: pgmap v4472: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:00 smithi078 ceph-mon[99959]: pgmap v4472: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:00.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:00.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:00.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:00 smithi146 ceph-mon[103278]: pgmap v4472: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:02 smithi146 ceph-mon[103278]: pgmap v4473: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:02 smithi078 ceph-mon[99959]: pgmap v4473: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:03.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:02 smithi078 ceph-mon[91136]: pgmap v4473: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:04 smithi146 ceph-mon[103278]: pgmap v4474: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[99959]: pgmap v4474: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:04 smithi078 ceph-mon[91136]: pgmap v4474: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:06 smithi146 ceph-mon[103278]: pgmap v4475: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:06 smithi078 ceph-mon[91136]: pgmap v4475: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:06 smithi078 ceph-mon[99959]: pgmap v4475: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:08 smithi146 ceph-mon[103278]: pgmap v4476: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[91136]: pgmap v4476: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[99959]: pgmap v4476: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:10.925 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:49:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:49:10.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:10.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:10.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:10.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:10.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:10.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:11.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:10 smithi146 ceph-mon[103278]: pgmap v4477: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:10 smithi078 ceph-mon[91136]: pgmap v4477: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:10 smithi078 ceph-mon[99959]: pgmap v4477: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:12 smithi146 ceph-mon[103278]: pgmap v4478: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:12 smithi078 ceph-mon[91136]: pgmap v4478: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:12 smithi078 ceph-mon[99959]: pgmap v4478: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:14 smithi146 ceph-mon[103278]: pgmap v4479: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:14 smithi078 ceph-mon[91136]: pgmap v4479: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:14 smithi078 ceph-mon[99959]: pgmap v4479: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:16 smithi146 ceph-mon[103278]: pgmap v4480: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:16 smithi078 ceph-mon[91136]: pgmap v4480: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:16 smithi078 ceph-mon[99959]: pgmap v4480: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:18.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:18.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:19.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:18 smithi078 ceph-mon[99959]: pgmap v4481: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:19.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:18 smithi078 ceph-mon[91136]: pgmap v4481: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:18 smithi146 ceph-mon[103278]: pgmap v4481: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:20.694 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:49:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:49:20.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:20.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:20.946 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:20.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:20.947 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:20.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:21.016 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:49:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:20 smithi146 ceph-mon[103278]: pgmap v4482: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:20 smithi078 ceph-mon[91136]: pgmap v4482: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:20 smithi078 ceph-mon[99959]: pgmap v4482: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 2h - - 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 2h - - 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 18s ago 2h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 13s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 18s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 13s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 18s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 18s ago 2h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 13s ago 2h 182M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:49:21.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 18s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 18s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 13s ago 2h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 18s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 18s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 18s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 18s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 13s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 13s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 13s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 13s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 13s ago 2h 81.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:49:21.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 18s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:49:21.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:49:21.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 18s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:49:21.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:49:21.885 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:49:21.885 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:49:21.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:49:21.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:21.885 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:49:21.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:49:21.886 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:49:22.154 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:49:22.154 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:49:22.154 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:49:22.154 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:49:22.154 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:49:22.155 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:49:22.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:21 smithi078 ceph-mon[99959]: from='client.54195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:22.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1067506941' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:22.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:21 smithi078 ceph-mon[91136]: from='client.54195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:22.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1067506941' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:21 smithi146 ceph-mon[103278]: from='client.54195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1067506941' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:22.461 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:49:22.461 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:22.461 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:22.461 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:22.461 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:22 smithi146 ceph-mon[103278]: from='client.74120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:22 smithi146 ceph-mon[103278]: from='client.54207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:22 smithi146 ceph-mon[103278]: pgmap v4483: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:22 smithi146 ceph-mon[103278]: from='client.74138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/32929076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[91136]: from='client.74120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[91136]: from='client.54207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[91136]: pgmap v4483: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[91136]: from='client.74138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/32929076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[99959]: from='client.74120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[99959]: from='client.54207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[99959]: pgmap v4483: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[99959]: from='client.74138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/32929076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:24 smithi146 ceph-mon[103278]: pgmap v4484: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[91136]: pgmap v4484: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:24 smithi078 ceph-mon[99959]: pgmap v4484: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:26 smithi146 ceph-mon[103278]: pgmap v4485: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:26 smithi078 ceph-mon[91136]: pgmap v4485: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:26 smithi078 ceph-mon[99959]: pgmap v4485: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:29 smithi146 ceph-mon[103278]: pgmap v4486: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[91136]: pgmap v4486: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[99959]: pgmap v4486: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:49:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:49:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:30.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:30.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:30.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:31.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:31 smithi146 ceph-mon[103278]: pgmap v4487: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:31 smithi078 ceph-mon[91136]: pgmap v4487: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:31 smithi078 ceph-mon[99959]: pgmap v4487: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:33 smithi146 ceph-mon[103278]: pgmap v4488: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:33 smithi078 ceph-mon[91136]: pgmap v4488: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:33 smithi078 ceph-mon[99959]: pgmap v4488: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:35 smithi146 ceph-mon[103278]: pgmap v4489: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:35 smithi078 ceph-mon[91136]: pgmap v4489: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:35 smithi078 ceph-mon[99959]: pgmap v4489: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:37 smithi146 ceph-mon[103278]: pgmap v4490: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:37 smithi078 ceph-mon[91136]: pgmap v4490: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:37 smithi078 ceph-mon[99959]: pgmap v4490: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:39.372 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:39 smithi078 ceph-mon[99959]: pgmap v4491: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:39.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:39 smithi078 ceph-mon[91136]: pgmap v4491: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:39 smithi146 ceph-mon[103278]: pgmap v4491: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:40.752 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:49:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:49:41.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:40.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:40.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:40.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:41.341 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:41 smithi078 ceph-mon[91136]: pgmap v4492: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:41.341 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:41 smithi078 ceph-mon[99959]: pgmap v4492: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:41 smithi146 ceph-mon[103278]: pgmap v4492: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:43 smithi146 ceph-mon[103278]: pgmap v4493: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:43 smithi078 ceph-mon[91136]: pgmap v4493: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:43 smithi078 ceph-mon[99959]: pgmap v4493: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:44.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:45 smithi146 ceph-mon[103278]: pgmap v4494: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:45 smithi078 ceph-mon[91136]: pgmap v4494: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:45 smithi078 ceph-mon[99959]: pgmap v4494: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:47 smithi146 ceph-mon[103278]: pgmap v4495: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:47 smithi078 ceph-mon[91136]: pgmap v4495: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:47 smithi078 ceph-mon[99959]: pgmap v4495: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:49 smithi146 ceph-mon[103278]: pgmap v4496: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[91136]: pgmap v4496: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[99959]: pgmap v4496: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:49:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:49:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:49:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:50.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:50.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:49:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:49:50.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:49:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:51 smithi146 ceph-mon[103278]: pgmap v4497: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:51 smithi078 ceph-mon[99959]: pgmap v4497: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:51 smithi078 ceph-mon[91136]: pgmap v4497: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:52.753 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:49:53.281 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:49:53.281 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 2h - - 2024-09-20T22:49:53.281 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 2h - - 2024-09-20T22:49:53.281 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 9s ago 2h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:49:53.281 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 5s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:49:53.281 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 9s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 5s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 9s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 9s ago 2h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 5s ago 2h 183M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 9s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 9s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 5s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 9s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 9s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 9s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 9s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 5s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:49:53.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 5s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 5s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 5s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 5s ago 2h 81.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 9s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 9s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:49:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:49:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:53 smithi146 ceph-mon[103278]: pgmap v4498: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:53 smithi078 ceph-mon[91136]: pgmap v4498: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:53 smithi078 ceph-mon[99959]: pgmap v4498: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:49:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:49:53.590 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:49:53.856 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:49:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:49:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:49:53.857 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:49:54.164 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:49:54.164 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:54.165 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:49:54.165 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:54.165 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:49:54.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:54 smithi146 ceph-mon[103278]: from='client.54231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:54.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:54 smithi146 ceph-mon[103278]: from='client.74156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:54.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/637361526' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:54.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1243896370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[91136]: from='client.54231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[91136]: from='client.74156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/637361526' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1243896370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[99959]: from='client.54231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[99959]: from='client.74156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:54.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/637361526' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:49:54.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1243896370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:49:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:55 smithi146 ceph-mon[103278]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:55 smithi146 ceph-mon[103278]: from='client.74174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:55 smithi146 ceph-mon[103278]: pgmap v4499: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:55 smithi078 ceph-mon[91136]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:55 smithi078 ceph-mon[91136]: from='client.74174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:55 smithi078 ceph-mon[91136]: pgmap v4499: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:55 smithi078 ceph-mon[99959]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:55 smithi078 ceph-mon[99959]: from='client.74174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:55 smithi078 ceph-mon[99959]: pgmap v4499: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:57 smithi146 ceph-mon[103278]: pgmap v4500: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:57 smithi078 ceph-mon[91136]: pgmap v4500: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:57 smithi078 ceph-mon[99959]: pgmap v4500: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:59 smithi146 ceph-mon[103278]: pgmap v4501: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:49:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:59 smithi078 ceph-mon[99959]: pgmap v4501: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:49:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:49:59.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:59 smithi078 ceph-mon[91136]: pgmap v4501: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:59.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:49:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:00.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T22:49:59.998+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:00 smithi146 ceph-mon[103278]: pgmap v4502: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-mon[91136]: pgmap v4502: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-mon[99959]: pgmap v4502: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:00.909 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:50:00] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T22:50:00.909 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:00.752Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:00.910 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:00.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:00.910 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:00.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:02 smithi146 ceph-mon[103278]: pgmap v4503: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:02 smithi078 ceph-mon[91136]: pgmap v4503: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:02 smithi078 ceph-mon[99959]: pgmap v4503: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:04 smithi146 ceph-mon[103278]: pgmap v4504: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[99959]: pgmap v4504: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:04 smithi078 ceph-mon[91136]: pgmap v4504: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:06 smithi146 ceph-mon[103278]: pgmap v4505: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:06 smithi078 ceph-mon[91136]: pgmap v4505: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:06 smithi078 ceph-mon[99959]: pgmap v4505: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:08 smithi146 ceph-mon[103278]: pgmap v4506: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[99959]: pgmap v4506: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[91136]: pgmap v4506: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:10.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:50:10] "GET /metrics HTTP/1.1" 200 38987 "" "Prometheus/2.43.0" 2024-09-20T22:50:10.924 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:10.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:10.924 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:10.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:10.924 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:10.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:10 smithi146 ceph-mon[103278]: pgmap v4507: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:10 smithi078 ceph-mon[91136]: pgmap v4507: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:10 smithi078 ceph-mon[99959]: pgmap v4507: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:13.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:12 smithi146 ceph-mon[103278]: pgmap v4508: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:12 smithi078 ceph-mon[99959]: pgmap v4508: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:12 smithi078 ceph-mon[91136]: pgmap v4508: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:14.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:15.146 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:14 smithi146 ceph-mon[103278]: pgmap v4509: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:14 smithi078 ceph-mon[91136]: pgmap v4509: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:14 smithi078 ceph-mon[99959]: pgmap v4509: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:16 smithi146 ceph-mon[103278]: pgmap v4510: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:16 smithi078 ceph-mon[91136]: pgmap v4510: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:16 smithi078 ceph-mon[99959]: pgmap v4510: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:19 smithi078 ceph-mon[91136]: pgmap v4511: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:19 smithi078 ceph-mon[99959]: pgmap v4511: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:19 smithi146 ceph-mon[103278]: pgmap v4511: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:20.720 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:50:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:50:21.020 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:20.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:21.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:20.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:21.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:20.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:21 smithi078 ceph-mon[91136]: pgmap v4512: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:21 smithi078 ceph-mon[99959]: pgmap v4512: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:21 smithi146 ceph-mon[103278]: pgmap v4512: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:23 smithi078 ceph-mon[91136]: pgmap v4513: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:23 smithi078 ceph-mon[99959]: pgmap v4513: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:23 smithi146 ceph-mon[103278]: pgmap v4513: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:24.447 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 2h - - 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 2h - - 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 1s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 16s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 1s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 16s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 1s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:50:24.976 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 1s ago 2h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 16s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 1s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 1s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 16s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 1s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 1s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 1s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 1s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 16s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 16s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 16s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 16s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 16s ago 2h 81.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:50:24.977 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 1s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:50:24.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 16s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:50:24.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 1s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:50:24.978 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 16s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:50:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:24 smithi146 ceph-mon[103278]: pgmap v4514: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.283 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:50:25.283 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:50:25.283 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:50:25.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:50:25.284 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[91136]: pgmap v4514: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[99959]: pgmap v4514: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:25.556 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:50:25.556 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:50:25.556 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:50:25.556 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:50:25.557 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:50:25.866 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:25.866 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:25.866 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:25 smithi146 ceph-mon[103278]: from='client.74186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:25 smithi146 ceph-mon[103278]: from='client.74192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:25 smithi146 ceph-mon[103278]: from='client.74198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2074042316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/203197540' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[91136]: from='client.74186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[91136]: from='client.74192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[91136]: from='client.74198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2074042316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/203197540' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[99959]: from='client.74186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[99959]: from='client.74192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[99959]: from='client.74198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2074042316' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/203197540' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:27.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:26 smithi146 ceph-mon[103278]: from='client.54291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:27.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:26 smithi146 ceph-mon[103278]: pgmap v4515: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:26 smithi078 ceph-mon[99959]: from='client.54291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:26 smithi078 ceph-mon[99959]: pgmap v4515: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:26 smithi078 ceph-mon[91136]: from='client.54291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:26 smithi078 ceph-mon[91136]: pgmap v4515: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:28 smithi146 ceph-mon[103278]: pgmap v4516: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[91136]: pgmap v4516: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[99959]: pgmap v4516: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:30.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:50:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:50:30.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:30.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:30.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:30.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:30.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:30.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:30 smithi146 ceph-mon[103278]: pgmap v4517: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:30 smithi078 ceph-mon[91136]: pgmap v4517: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:30 smithi078 ceph-mon[99959]: pgmap v4517: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:32 smithi146 ceph-mon[103278]: pgmap v4518: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:32 smithi078 ceph-mon[91136]: pgmap v4518: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:32 smithi078 ceph-mon[99959]: pgmap v4518: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:35.158 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:34 smithi146 ceph-mon[103278]: pgmap v4519: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:34 smithi078 ceph-mon[91136]: pgmap v4519: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:34 smithi078 ceph-mon[99959]: pgmap v4519: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:36 smithi146 ceph-mon[103278]: pgmap v4520: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:36 smithi078 ceph-mon[91136]: pgmap v4520: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:36 smithi078 ceph-mon[99959]: pgmap v4520: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:38 smithi146 ceph-mon[103278]: pgmap v4521: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:38 smithi078 ceph-mon[91136]: pgmap v4521: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:38 smithi078 ceph-mon[99959]: pgmap v4521: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:40.915 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:50:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:50:40.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:40.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:40.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:40.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:40.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:40.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:41.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:40 smithi078 ceph-mon[91136]: pgmap v4522: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:41.193 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:40 smithi078 ceph-mon[99959]: pgmap v4522: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:40 smithi146 ceph-mon[103278]: pgmap v4522: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:42 smithi146 ceph-mon[103278]: pgmap v4523: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:42 smithi078 ceph-mon[99959]: pgmap v4523: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:42 smithi078 ceph-mon[91136]: pgmap v4523: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:45.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:45 smithi146 ceph-mon[103278]: pgmap v4524: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:45.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[91136]: pgmap v4524: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[99959]: pgmap v4524: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:47.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:47 smithi078 ceph-mon[99959]: pgmap v4525: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:47.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:47 smithi078 ceph-mon[91136]: pgmap v4525: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:47 smithi146 ceph-mon[103278]: pgmap v4525: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[91136]: pgmap v4526: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[99959]: pgmap v4526: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:49 smithi146 ceph-mon[103278]: pgmap v4526: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:50:51.018 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:50:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:50:51.018 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:50.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:51.018 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:50.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:50:51.019 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:50:50.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:50:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:51 smithi078 ceph-mon[91136]: pgmap v4527: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:51 smithi078 ceph-mon[99959]: pgmap v4527: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:51 smithi146 ceph-mon[103278]: pgmap v4527: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:50:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:53 smithi078 ceph-mon[91136]: pgmap v4528: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:53 smithi078 ceph-mon[99959]: pgmap v4528: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:53 smithi146 ceph-mon[103278]: pgmap v4528: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:50:55.137 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:55 smithi146 ceph-mon[103278]: pgmap v4529: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:50:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:55 smithi078 ceph-mon[91136]: pgmap v4529: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:50:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:55 smithi078 ceph-mon[99959]: pgmap v4529: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T22:50:56.169 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 2h - - 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 12s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 12s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 12s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 12s ago 2h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 8s ago 2h 184M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:50:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 12s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 12s ago 2h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 12s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 12s ago 2h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 12s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 12s ago 2h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 2h 84.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 2h 69.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 2h 73.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 2h 69.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 8s ago 2h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 12s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:50:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 12s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:50:56.701 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:50:57.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:50:57.006 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:50:57.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:50:57.275 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:50:57.275 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:50:57.275 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:50:57.275 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:50:57.275 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:50:57.276 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:50:57.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:57 smithi078 ceph-mon[91136]: pgmap v4530: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:50:57.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:57 smithi078 ceph-mon[91136]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:57.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4192841776' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:57.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:57 smithi078 ceph-mon[99959]: pgmap v4530: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:50:57.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:57 smithi078 ceph-mon[99959]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:57.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4192841776' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:57 smithi146 ceph-mon[103278]: pgmap v4530: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:50:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:57 smithi146 ceph-mon[103278]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4192841776' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:50:57.593 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:50:57.593 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:57.593 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:50:57.593 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:57.593 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:50:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:58 smithi146 ceph-mon[103278]: from='client.54309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:58 smithi146 ceph-mon[103278]: from='client.54315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/670879388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[91136]: from='client.54309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[91136]: from='client.54315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/670879388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[99959]: from='client.54309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[99959]: from='client.54315 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/670879388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:50:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:59 smithi146 ceph-mon[103278]: from='client.54327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:59 smithi146 ceph-mon[103278]: pgmap v4531: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:50:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:50:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:59 smithi078 ceph-mon[91136]: from='client.54327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:59 smithi078 ceph-mon[91136]: pgmap v4531: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:50:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:50:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:50:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:59 smithi078 ceph-mon[99959]: from='client.54327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:50:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:59 smithi078 ceph-mon[99959]: pgmap v4531: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T22:50:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:50:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:51:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:51:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:00.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:00.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:00.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:01 smithi146 ceph-mon[103278]: pgmap v4532: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:51:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:01 smithi078 ceph-mon[91136]: pgmap v4532: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:51:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:01 smithi078 ceph-mon[99959]: pgmap v4532: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T22:51:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:03 smithi146 ceph-mon[103278]: pgmap v4533: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:51:03.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:03 smithi078 ceph-mon[99959]: pgmap v4533: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:51:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:03 smithi078 ceph-mon[91136]: pgmap v4533: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T22:51:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:05 smithi146 ceph-mon[103278]: pgmap v4534: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:51:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[91136]: pgmap v4534: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[99959]: pgmap v4534: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:07 smithi146 ceph-mon[103278]: pgmap v4535: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:51:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:07 smithi078 ceph-mon[91136]: pgmap v4535: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:51:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:07 smithi078 ceph-mon[99959]: pgmap v4535: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T22:51:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:09 smithi146 ceph-mon[103278]: pgmap v4536: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:51:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[99959]: pgmap v4536: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[91136]: pgmap v4536: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:51:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:51:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:10.757Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:10.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:10.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:11 smithi146 ceph-mon[103278]: pgmap v4537: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:51:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:11 smithi078 ceph-mon[91136]: pgmap v4537: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:51:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:11 smithi078 ceph-mon[99959]: pgmap v4537: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:51:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:13 smithi146 ceph-mon[103278]: pgmap v4538: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T22:51:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:13 smithi078 ceph-mon[91136]: pgmap v4538: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T22:51:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:13 smithi078 ceph-mon[99959]: pgmap v4538: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T22:51:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:15 smithi146 ceph-mon[103278]: pgmap v4539: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:51:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:15 smithi078 ceph-mon[91136]: pgmap v4539: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:51:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:15 smithi078 ceph-mon[99959]: pgmap v4539: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T22:51:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:17 smithi146 ceph-mon[103278]: pgmap v4540: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T22:51:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:17 smithi078 ceph-mon[91136]: pgmap v4540: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T22:51:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:17 smithi078 ceph-mon[99959]: pgmap v4540: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 153 op/s 2024-09-20T22:51:18.492 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:18.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:19 smithi146 ceph-mon[103278]: pgmap v4541: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:51:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:19 smithi078 ceph-mon[91136]: pgmap v4541: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:51:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:19 smithi078 ceph-mon[99959]: pgmap v4541: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:51:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:51:20] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T22:51:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:20.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:20.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:20.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:21 smithi146 ceph-mon[103278]: pgmap v4542: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:51:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:21 smithi078 ceph-mon[99959]: pgmap v4542: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:51:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:21 smithi078 ceph-mon[91136]: pgmap v4542: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T22:51:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:23 smithi146 ceph-mon[103278]: pgmap v4543: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T22:51:23.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:23 smithi078 ceph-mon[99959]: pgmap v4543: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T22:51:23.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:23 smithi078 ceph-mon[91136]: pgmap v4543: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T22:51:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:25 smithi146 ceph-mon[103278]: pgmap v4544: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:51:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[91136]: pgmap v4544: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[99959]: pgmap v4544: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:27 smithi146 ceph-mon[103278]: pgmap v4545: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:51:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:27 smithi078 ceph-mon[91136]: pgmap v4545: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:51:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:27 smithi078 ceph-mon[99959]: pgmap v4545: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-20T22:51:27.887 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:51:28.415 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 2h - - 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 2h - - 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 4s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 19s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 4s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 19s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 4s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 4s ago 2h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 19s ago 2h 185M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 4s ago 2h 192M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 4s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:51:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 19s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 4s ago 2h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 4s ago 2h 78.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 4s ago 2h 68.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 4s ago 2h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 19s ago 2h 84.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 19s ago 2h 69.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 19s ago 2h 73.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 19s ago 2h 69.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 19s ago 2h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 4s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 4s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:51:28.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:51:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:28.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:51:28.729 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:51:28.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:51:28.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:51:29.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:51:29.003 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='client.54339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: pgmap v4546: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='client.54345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2023894101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='client.54339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: pgmap v4546: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='client.54345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2023894101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.318 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:51:29.318 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:51:29.318 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:51:29.318 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:51:29.318 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:51:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='client.54339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: pgmap v4546: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='client.54345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2023894101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:51:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:30 smithi146 ceph-mon[103278]: from='client.54351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:30 smithi146 ceph-mon[103278]: from='client.54363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/949901910' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-mon[99959]: from='client.54351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-mon[99959]: from='client.54363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/949901910' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-mon[91136]: from='client.54351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-mon[91136]: from='client.54363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:51:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/949901910' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:51:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:51:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:51:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:30.759Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:30.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:30.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:31 smithi146 ceph-mon[103278]: pgmap v4547: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:31 smithi078 ceph-mon[91136]: pgmap v4547: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:31 smithi078 ceph-mon[99959]: pgmap v4547: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:33 smithi146 ceph-mon[103278]: pgmap v4548: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:33 smithi078 ceph-mon[91136]: pgmap v4548: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:33 smithi078 ceph-mon[99959]: pgmap v4548: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:35.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:35 smithi146 ceph-mon[103278]: pgmap v4549: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:35 smithi078 ceph-mon[91136]: pgmap v4549: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:35 smithi078 ceph-mon[99959]: pgmap v4549: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:37.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:37 smithi146 ceph-mon[103278]: pgmap v4550: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:37 smithi078 ceph-mon[91136]: pgmap v4550: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:37 smithi078 ceph-mon[99959]: pgmap v4550: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:39 smithi146 ceph-mon[103278]: pgmap v4551: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:39 smithi078 ceph-mon[91136]: pgmap v4551: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:39 smithi078 ceph-mon[99959]: pgmap v4551: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:51:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:51:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:40.759Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:40.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:40.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:41 smithi146 ceph-mon[103278]: pgmap v4552: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:41 smithi078 ceph-mon[91136]: pgmap v4552: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:41 smithi078 ceph-mon[99959]: pgmap v4552: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:43.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:43 smithi078 ceph-mon[99959]: pgmap v4553: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:43.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:43 smithi078 ceph-mon[91136]: pgmap v4553: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:43 smithi146 ceph-mon[103278]: pgmap v4553: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:45.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[91136]: pgmap v4554: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:45.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[99959]: pgmap v4554: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:45.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:45 smithi146 ceph-mon[103278]: pgmap v4554: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:46 smithi146 ceph-mon[103278]: pgmap v4555: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:46 smithi078 ceph-mon[91136]: pgmap v4555: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:46 smithi078 ceph-mon[99959]: pgmap v4555: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:48 smithi146 ceph-mon[103278]: pgmap v4556: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:48 smithi078 ceph-mon[91136]: pgmap v4556: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:48 smithi078 ceph-mon[99959]: pgmap v4556: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:51:50.974 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:51:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:51:50.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:50.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:50.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:50.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:51:50.974 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:51:50.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:51:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:50 smithi146 ceph-mon[103278]: pgmap v4557: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:50 smithi078 ceph-mon[91136]: pgmap v4557: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:50 smithi078 ceph-mon[99959]: pgmap v4557: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:52 smithi146 ceph-mon[103278]: pgmap v4558: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:52 smithi078 ceph-mon[99959]: pgmap v4558: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:52 smithi078 ceph-mon[91136]: pgmap v4558: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:54 smithi146 ceph-mon[103278]: pgmap v4559: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:54 smithi078 ceph-mon[91136]: pgmap v4559: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:54 smithi078 ceph-mon[99959]: pgmap v4559: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:51:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:56 smithi146 ceph-mon[103278]: pgmap v4560: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:56 smithi078 ceph-mon[99959]: pgmap v4560: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:56 smithi078 ceph-mon[91136]: pgmap v4560: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:58.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:58 smithi146 ceph-mon[103278]: pgmap v4561: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:58 smithi078 ceph-mon[91136]: pgmap v4561: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:58 smithi078 ceph-mon[99959]: pgmap v4561: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:51:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:51:59.611 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 2h - - 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 2h - - 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 15s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 11s ago 2h 87.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 15s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 11s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 15s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 15s ago 2h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 11s ago 2h 186M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:52:00.138 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 15s ago 2h 193M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 15s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 11s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 15s ago 2h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 15s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 15s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 15s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 11s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 11s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 11s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 11s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 11s ago 2h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 15s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 15s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:52:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:52:00.446 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:52:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:52:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:52:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:52:00.446 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:52:00.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:52:00.715 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:52:00.715 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:52:00.715 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:52:00.715 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:52:00.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:52:00.716 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:52:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:52:00.994 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:00.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:00.994 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:00.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:00.994 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:00.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:01.027 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:52:01.027 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:01.027 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:01.027 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:01.027 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:00 smithi146 ceph-mon[103278]: from='client.74294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:00 smithi146 ceph-mon[103278]: from='client.54381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:00 smithi146 ceph-mon[103278]: pgmap v4562: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:00 smithi146 ceph-mon[103278]: from='client.74306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1255896602' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[91136]: from='client.74294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[91136]: from='client.54381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[91136]: pgmap v4562: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[91136]: from='client.74306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1255896602' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[99959]: from='client.74294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[99959]: from='client.54381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[99959]: pgmap v4562: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[99959]: from='client.74306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1255896602' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:01 smithi146 ceph-mon[103278]: from='client.74318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/467732034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:01 smithi078 ceph-mon[99959]: from='client.74318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/467732034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:01 smithi078 ceph-mon[91136]: from='client.74318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/467732034' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:03 smithi146 ceph-mon[103278]: pgmap v4563: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:02 smithi078 ceph-mon[91136]: pgmap v4563: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:02 smithi078 ceph-mon[99959]: pgmap v4563: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:05.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[91136]: pgmap v4564: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:05.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.098 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[99959]: pgmap v4564: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:05.098 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.098 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.098 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:05 smithi146 ceph-mon[103278]: pgmap v4564: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:07 smithi146 ceph-mon[103278]: pgmap v4565: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:07 smithi078 ceph-mon[91136]: pgmap v4565: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:07 smithi078 ceph-mon[99959]: pgmap v4565: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:09 smithi146 ceph-mon[103278]: pgmap v4566: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:09 smithi078 ceph-mon[99959]: pgmap v4566: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:09 smithi078 ceph-mon[91136]: pgmap v4566: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:52:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:52:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:10.761Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:10.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:10.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:11 smithi146 ceph-mon[103278]: pgmap v4567: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:11 smithi078 ceph-mon[99959]: pgmap v4567: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:11 smithi078 ceph-mon[91136]: pgmap v4567: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:13 smithi146 ceph-mon[103278]: pgmap v4568: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:13 smithi078 ceph-mon[99959]: pgmap v4568: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:13 smithi078 ceph-mon[91136]: pgmap v4568: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:15 smithi146 ceph-mon[103278]: pgmap v4569: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:15 smithi078 ceph-mon[91136]: pgmap v4569: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:15 smithi078 ceph-mon[99959]: pgmap v4569: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:17 smithi146 ceph-mon[103278]: pgmap v4570: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:17.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:17 smithi078 ceph-mon[99959]: pgmap v4570: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:17.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:17 smithi078 ceph-mon[91136]: pgmap v4570: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:19 smithi146 ceph-mon[103278]: pgmap v4571: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:19 smithi078 ceph-mon[91136]: pgmap v4571: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:19 smithi078 ceph-mon[99959]: pgmap v4571: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:52:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:52:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:20.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:20.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:20.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:21 smithi146 ceph-mon[103278]: pgmap v4572: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:21 smithi078 ceph-mon[91136]: pgmap v4572: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:21 smithi078 ceph-mon[99959]: pgmap v4572: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:23 smithi146 ceph-mon[103278]: pgmap v4573: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:23 smithi078 ceph-mon[91136]: pgmap v4573: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:23 smithi078 ceph-mon[99959]: pgmap v4573: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[99959]: pgmap v4574: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[91136]: pgmap v4574: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:25 smithi146 ceph-mon[103278]: pgmap v4574: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:27 smithi146 ceph-mon[103278]: pgmap v4575: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:27 smithi078 ceph-mon[91136]: pgmap v4575: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:27 smithi078 ceph-mon[99959]: pgmap v4575: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:28.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:29 smithi146 ceph-mon[103278]: pgmap v4576: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:29 smithi078 ceph-mon[91136]: pgmap v4576: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:29 smithi078 ceph-mon[99959]: pgmap v4576: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:52:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:52:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:30.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:30.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:30.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:31.328 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:52:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:31 smithi146 ceph-mon[103278]: pgmap v4577: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:31 smithi078 ceph-mon[99959]: pgmap v4577: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:31 smithi078 ceph-mon[91136]: pgmap v4577: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:31.851 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:52:31.851 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 2h - - 2024-09-20T22:52:31.851 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 2h - - 2024-09-20T22:52:31.851 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:52:31.851 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 2s ago 2h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 7s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 2s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 7s ago 2h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 2s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 2h 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 2s ago 2h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:52:31.852 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 2s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 2s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 2s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 2s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 2s ago 2h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:52:31.853 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:52:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:52:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:52:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:52:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:52:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:52:32.173 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:52:32.441 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:52:32.442 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:52:32.442 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:52:32.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3939002053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:32.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3939002053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3939002053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:52:32.751 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:52:32.751 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:52:32.751 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:32.751 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:52:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:33 smithi146 ceph-mon[103278]: from='client.74330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:33 smithi146 ceph-mon[103278]: from='client.54417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:33 smithi146 ceph-mon[103278]: from='client.54423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:33 smithi146 ceph-mon[103278]: pgmap v4578: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1739054541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[91136]: from='client.74330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[91136]: from='client.54417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[91136]: from='client.54423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[91136]: pgmap v4578: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1739054541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[99959]: from='client.74330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[99959]: from='client.54417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[99959]: from='client.54423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[99959]: pgmap v4578: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1739054541' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:52:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:34 smithi146 ceph-mon[103278]: from='client.54435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:34 smithi078 ceph-mon[91136]: from='client.54435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:34 smithi078 ceph-mon[99959]: from='client.54435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:52:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:35 smithi146 ceph-mon[103278]: pgmap v4579: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:35 smithi078 ceph-mon[91136]: pgmap v4579: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:35 smithi078 ceph-mon[99959]: pgmap v4579: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:36 smithi146 ceph-mon[103278]: pgmap v4580: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:36 smithi078 ceph-mon[91136]: pgmap v4580: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:36 smithi078 ceph-mon[99959]: pgmap v4580: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:37.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:38.483 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:38 smithi146 ceph-mon[103278]: pgmap v4581: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:38 smithi078 ceph-mon[99959]: pgmap v4581: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:38 smithi078 ceph-mon[91136]: pgmap v4581: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:40.967 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:52:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T22:52:40.967 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:40.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:40.967 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:40.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:40.967 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:40.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:40 smithi146 ceph-mon[103278]: pgmap v4582: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:40 smithi078 ceph-mon[91136]: pgmap v4582: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:40 smithi078 ceph-mon[99959]: pgmap v4582: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:42 smithi146 ceph-mon[103278]: pgmap v4583: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:42 smithi078 ceph-mon[91136]: pgmap v4583: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:42 smithi078 ceph-mon[99959]: pgmap v4583: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:44 smithi146 ceph-mon[103278]: pgmap v4584: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[91136]: pgmap v4584: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[99959]: pgmap v4584: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:47.108 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:46 smithi078 ceph-mon[91136]: pgmap v4585: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:47.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:46 smithi078 ceph-mon[99959]: pgmap v4585: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:46 smithi146 ceph-mon[103278]: pgmap v4585: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:48 smithi146 ceph-mon[103278]: pgmap v4586: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:49.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:48 smithi078 ceph-mon[91136]: pgmap v4586: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:49.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:48 smithi078 ceph-mon[99959]: pgmap v4586: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[91136]: pgmap v4587: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-mon[99959]: pgmap v4587: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:50.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:52:50.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:50 smithi146 ceph-mon[103278]: pgmap v4587: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:52:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:52:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:50.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:50.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:52:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:52:50.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:52:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:52 smithi146 ceph-mon[103278]: pgmap v4588: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:52 smithi078 ceph-mon[91136]: pgmap v4588: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:52 smithi078 ceph-mon[99959]: pgmap v4588: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:54 smithi146 ceph-mon[103278]: pgmap v4589: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:54 smithi078 ceph-mon[99959]: pgmap v4589: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:54 smithi078 ceph-mon[91136]: pgmap v4589: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:52:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:56 smithi146 ceph-mon[103278]: pgmap v4590: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:56 smithi078 ceph-mon[91136]: pgmap v4590: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:56 smithi078 ceph-mon[99959]: pgmap v4590: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:58.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:58 smithi146 ceph-mon[103278]: pgmap v4591: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:58 smithi078 ceph-mon[91136]: pgmap v4591: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:52:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:58 smithi078 ceph-mon[99959]: pgmap v4591: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:52:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:00.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:53:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:53:00.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:00.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:00.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:00.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:00.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:00.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:00 smithi146 ceph-mon[103278]: pgmap v4592: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:00 smithi078 ceph-mon[91136]: pgmap v4592: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:00 smithi078 ceph-mon[99959]: pgmap v4592: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:03.054 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:53:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:02 smithi146 ceph-mon[103278]: pgmap v4593: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:02 smithi078 ceph-mon[91136]: pgmap v4593: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:02 smithi078 ceph-mon[99959]: pgmap v4593: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 2h - - 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 2h - - 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 18s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 14s ago 2h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 18s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 14s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 18s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:53:03.581 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 18s ago 2h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 14s ago 2h 187M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 18s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 18s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 14s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 18s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 18s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 18s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 18s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 14s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 14s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 14s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 14s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 14s ago 2h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 18s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:53:03.582 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:53:03.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 18s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:53:03.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:53:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:53:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:53:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:53:03.889 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:53:04.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:03 smithi078 ceph-mon[99959]: from='client.54447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:04.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3417082254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:04.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:03 smithi078 ceph-mon[91136]: from='client.54447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:04.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3417082254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:03 smithi146 ceph-mon[103278]: from='client.54447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3417082254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:04.469 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:53:04.469 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:53:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:53:04.469 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:53:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:53:05.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='client.74372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='client.74378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: pgmap v4594: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:05.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='client.54471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1286178010' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='client.74372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='client.74378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: pgmap v4594: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='client.54471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1286178010' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='client.74372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='client.74378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: pgmap v4594: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='client.54471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1286178010' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:07 smithi146 ceph-mon[103278]: pgmap v4595: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:07 smithi078 ceph-mon[91136]: pgmap v4595: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:07 smithi078 ceph-mon[99959]: pgmap v4595: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:09 smithi146 ceph-mon[103278]: pgmap v4596: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:09 smithi078 ceph-mon[91136]: pgmap v4596: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:09 smithi078 ceph-mon[99959]: pgmap v4596: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:10 smithi146 ceph-mon[103278]: pgmap v4597: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:10.767 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:53:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T22:53:10.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[99959]: pgmap v4597: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:10.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:10.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-mon[91136]: pgmap v4597: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:10.765Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:10.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:10.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:12 smithi146 ceph-mon[103278]: pgmap v4598: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:12 smithi078 ceph-mon[91136]: pgmap v4598: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:12 smithi078 ceph-mon[99959]: pgmap v4598: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:14 smithi146 ceph-mon[103278]: pgmap v4599: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:14 smithi078 ceph-mon[91136]: pgmap v4599: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:14 smithi078 ceph-mon[99959]: pgmap v4599: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:16 smithi078 ceph-mon[91136]: pgmap v4600: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:16 smithi078 ceph-mon[99959]: pgmap v4600: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:16 smithi146 ceph-mon[103278]: pgmap v4600: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:18 smithi146 ceph-mon[103278]: pgmap v4601: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:18 smithi078 ceph-mon[91136]: pgmap v4601: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:18 smithi078 ceph-mon[99959]: pgmap v4601: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:20.998 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:53:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:53:20.999 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:20.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:20.999 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:20.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:20.999 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:20.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:20 smithi146 ceph-mon[103278]: pgmap v4602: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:20 smithi078 ceph-mon[91136]: pgmap v4602: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:20 smithi078 ceph-mon[99959]: pgmap v4602: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:23 smithi146 ceph-mon[103278]: pgmap v4603: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:23 smithi078 ceph-mon[91136]: pgmap v4603: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:23 smithi078 ceph-mon[99959]: pgmap v4603: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:25 smithi146 ceph-mon[103278]: pgmap v4604: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[91136]: pgmap v4604: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[99959]: pgmap v4604: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:27.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:27 smithi146 ceph-mon[103278]: pgmap v4605: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:27.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:27 smithi078 ceph-mon[91136]: pgmap v4605: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:27.368 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:27 smithi078 ceph-mon[99959]: pgmap v4605: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:28.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:28.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:29 smithi146 ceph-mon[103278]: pgmap v4606: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:29 smithi078 ceph-mon[91136]: pgmap v4606: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:29 smithi078 ceph-mon[99959]: pgmap v4606: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:30 smithi146 ceph-mon[103278]: pgmap v4607: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:53:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[91136]: pgmap v4607: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-mon[99959]: pgmap v4607: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:30.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:30.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:31.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:30.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:32 smithi146 ceph-mon[103278]: pgmap v4608: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:32 smithi078 ceph-mon[91136]: pgmap v4608: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:32 smithi078 ceph-mon[99959]: pgmap v4608: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:34.766 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:53:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:34 smithi078 ceph-mon[99959]: pgmap v4609: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:34 smithi078 ceph-mon[91136]: pgmap v4609: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:34 smithi146 ceph-mon[103278]: pgmap v4609: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:35.292 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:53:35.292 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 2h - - 2024-09-20T22:53:35.292 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 2h - - 2024-09-20T22:53:35.292 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 10s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:53:35.292 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 5s ago 2h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:53:35.292 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 10s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 5s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 10s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 10s ago 2h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 5s ago 2h 188M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 10s ago 2h 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 10s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 5s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 10s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 10s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 10s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 10s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 5s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 5s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:53:35.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 5s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:53:35.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 5s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:53:35.294 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 5s ago 2h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:53:35.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 10s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:53:35.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:53:35.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 10s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:53:35.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:53:35.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:53:35.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:53:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:53:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:53:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:53:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:53:35.602 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:53:35.869 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:53:35.870 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:53:36.178 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:53:36.178 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:53:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:53:36.178 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:53:36.178 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:53:36.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:35 smithi078 ceph-mon[91136]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:36.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:35 smithi078 ceph-mon[91136]: from='client.74408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:36.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2825622505' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:36.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:35 smithi078 ceph-mon[99959]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:36.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:35 smithi078 ceph-mon[99959]: from='client.74408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:36.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2825622505' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:35 smithi146 ceph-mon[103278]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:35 smithi146 ceph-mon[103278]: from='client.74408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2825622505' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[91136]: from='client.74414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[91136]: from='client.54507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[91136]: pgmap v4610: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3592534287' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[99959]: from='client.74414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[99959]: from='client.54507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[99959]: pgmap v4610: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3592534287' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:36 smithi146 ceph-mon[103278]: from='client.74414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:36 smithi146 ceph-mon[103278]: from='client.54507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:53:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:36 smithi146 ceph-mon[103278]: pgmap v4610: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3592534287' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:53:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:39 smithi146 ceph-mon[103278]: pgmap v4611: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:39 smithi078 ceph-mon[91136]: pgmap v4611: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:39 smithi078 ceph-mon[99959]: pgmap v4611: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:41.005 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:53:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:53:41.005 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:40.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:41.005 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:40.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:41.005 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:40.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:41 smithi146 ceph-mon[103278]: pgmap v4612: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:41 smithi078 ceph-mon[91136]: pgmap v4612: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:41 smithi078 ceph-mon[99959]: pgmap v4612: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:43 smithi146 ceph-mon[103278]: pgmap v4613: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:43 smithi078 ceph-mon[91136]: pgmap v4613: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:43.300 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:43 smithi078 ceph-mon[99959]: pgmap v4613: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:45 smithi078 ceph-mon[91136]: pgmap v4614: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:45 smithi078 ceph-mon[99959]: pgmap v4614: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:45 smithi146 ceph-mon[103278]: pgmap v4614: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:47 smithi146 ceph-mon[103278]: pgmap v4615: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:47 smithi078 ceph-mon[91136]: pgmap v4615: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:47 smithi078 ceph-mon[99959]: pgmap v4615: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:48.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:48.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:49 smithi146 ceph-mon[103278]: pgmap v4616: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:49 smithi078 ceph-mon[91136]: pgmap v4616: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:49 smithi078 ceph-mon[99959]: pgmap v4616: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:50 smithi146 ceph-mon[103278]: pgmap v4617: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:53:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:53:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[91136]: pgmap v4617: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:53:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-mon[99959]: pgmap v4617: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:50.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:50.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:53:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:53:50.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:53:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:52 smithi146 ceph-mon[103278]: pgmap v4618: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:52 smithi078 ceph-mon[91136]: pgmap v4618: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:52 smithi078 ceph-mon[99959]: pgmap v4618: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:54 smithi146 ceph-mon[103278]: pgmap v4619: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:54 smithi078 ceph-mon[91136]: pgmap v4619: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:54 smithi078 ceph-mon[99959]: pgmap v4619: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:56 smithi078 ceph-mon[91136]: pgmap v4620: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:56 smithi078 ceph-mon[99959]: pgmap v4620: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:56 smithi146 ceph-mon[103278]: pgmap v4620: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:53:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:59 smithi146 ceph-mon[103278]: pgmap v4621: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:53:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:59 smithi078 ceph-mon[99959]: pgmap v4621: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:53:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:53:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:59 smithi078 ceph-mon[91136]: pgmap v4621: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:53:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:53:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:01.007 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:54:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:54:01.007 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:00.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:01.008 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:00.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:01.008 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:00.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:01.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:01 smithi146 ceph-mon[103278]: pgmap v4622: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:01 smithi078 ceph-mon[91136]: pgmap v4622: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:01 smithi078 ceph-mon[99959]: pgmap v4622: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:03 smithi146 ceph-mon[103278]: pgmap v4623: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:03 smithi078 ceph-mon[91136]: pgmap v4623: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:03 smithi078 ceph-mon[99959]: pgmap v4623: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:05 smithi146 ceph-mon[103278]: pgmap v4624: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:05 smithi078 ceph-mon[91136]: pgmap v4624: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:05 smithi078 ceph-mon[99959]: pgmap v4624: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:06.479 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:54:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:06 smithi146 ceph-mon[103278]: pgmap v4625: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[91136]: pgmap v4625: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:06 smithi078 ceph-mon[99959]: pgmap v4625: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 2h - - 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 2h - - 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 1s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 17s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 1s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 17s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 1s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 1s ago 2h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:54:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 17s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 1s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 1s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 17s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 1s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 1s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 1s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 1s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 17s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 17s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 17s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 17s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 17s ago 2h 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 1s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:54:07.008 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:54:07.009 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 1s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:54:07.009 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:54:07.328 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:54:07.328 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:54:07.328 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:54:07.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:54:07.329 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:54:07.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[99959]: from='client.74438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[99959]: from='client.54525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:07.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[99959]: from='client.54531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[91136]: from='client.74438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[91136]: from='client.54525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:07.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:07 smithi078 ceph-mon[91136]: from='client.54531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:54:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:54:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:54:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:54:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:54:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:54:07.605 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:54:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:07 smithi146 ceph-mon[103278]: from='client.74438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:07 smithi146 ceph-mon[103278]: from='client.54525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:07 smithi146 ceph-mon[103278]: from='client.54531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:07.926 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:54:07.927 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:07.927 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:07.927 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:07.927 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4066927208' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[91136]: from='client.74462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/744451107' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[91136]: pgmap v4626: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4066927208' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[99959]: from='client.74462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/744451107' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:08 smithi078 ceph-mon[99959]: pgmap v4626: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4066927208' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:08 smithi146 ceph-mon[103278]: from='client.74462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/744451107' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:08 smithi146 ceph-mon[103278]: pgmap v4626: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:10.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:54:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:54:10.866 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:10.770Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:10.866 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:10.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:10.866 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:10.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:10 smithi146 ceph-mon[103278]: pgmap v4627: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[91136]: pgmap v4627: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:10 smithi078 ceph-mon[99959]: pgmap v4627: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:12 smithi146 ceph-mon[103278]: pgmap v4628: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:12 smithi078 ceph-mon[91136]: pgmap v4628: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:13 smithi078 ceph-mon[99959]: pgmap v4628: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:15 smithi146 ceph-mon[103278]: pgmap v4629: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:15 smithi078 ceph-mon[91136]: pgmap v4629: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:15 smithi078 ceph-mon[99959]: pgmap v4629: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:17.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:17 smithi078 ceph-mon[99959]: pgmap v4630: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:17.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:17 smithi078 ceph-mon[91136]: pgmap v4630: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:17 smithi146 ceph-mon[103278]: pgmap v4630: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:18.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:18.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:19 smithi078 ceph-mon[91136]: pgmap v4631: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:19 smithi078 ceph-mon[99959]: pgmap v4631: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:19 smithi146 ceph-mon[103278]: pgmap v4631: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:21.021 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:54:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:54:21.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:20.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:21.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:20.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:21.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:20.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:21 smithi078 ceph-mon[91136]: pgmap v4632: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:21 smithi078 ceph-mon[99959]: pgmap v4632: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:21 smithi146 ceph-mon[103278]: pgmap v4632: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:23 smithi078 ceph-mon[91136]: pgmap v4633: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:23 smithi078 ceph-mon[99959]: pgmap v4633: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:23 smithi146 ceph-mon[103278]: pgmap v4633: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:25 smithi078 ceph-mon[99959]: pgmap v4634: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:25 smithi078 ceph-mon[91136]: pgmap v4634: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:25 smithi146 ceph-mon[103278]: pgmap v4634: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 22:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T22:54:25.847310072Z level=info msg="Completed cleanup jobs" duration=12.035412ms 2024-09-20T22:54:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:26 smithi146 ceph-mon[103278]: pgmap v4635: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[91136]: pgmap v4635: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:26 smithi078 ceph-mon[99959]: pgmap v4635: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:27.779 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:27.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:28 smithi146 ceph-mon[103278]: pgmap v4636: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:28 smithi078 ceph-mon[91136]: pgmap v4636: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:28 smithi078 ceph-mon[99959]: pgmap v4636: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:30.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:54:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:54:30.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:30.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:30.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:30.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:30.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:30.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:30 smithi146 ceph-mon[103278]: pgmap v4637: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[99959]: pgmap v4637: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[91136]: pgmap v4637: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:31.268 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:32 smithi146 ceph-mon[103278]: pgmap v4638: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:32 smithi078 ceph-mon[91136]: pgmap v4638: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:32 smithi078 ceph-mon[99959]: pgmap v4638: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:34 smithi146 ceph-mon[103278]: pgmap v4639: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:34 smithi078 ceph-mon[91136]: pgmap v4639: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:35 smithi078 ceph-mon[99959]: pgmap v4639: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:37 smithi146 ceph-mon[103278]: pgmap v4640: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:37 smithi078 ceph-mon[91136]: pgmap v4640: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:37 smithi078 ceph-mon[99959]: pgmap v4640: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:38.227 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:54:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:38.751 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:54:38.751 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 2h - - 2024-09-20T22:54:38.751 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T22:54:38.751 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 13s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:54:38.751 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:54:38.751 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 13s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 13s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 13s ago 2h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 8s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 13s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 13s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 13s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 13s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 13s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 13s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:54:38.752 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:54:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:54:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 8s ago 2h 77.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:54:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 13s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:54:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:54:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 13s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:54:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:54:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:54:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:54:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:54:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:54:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:54:39.063 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:54:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:39 smithi146 ceph-mon[103278]: pgmap v4641: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:39 smithi146 ceph-mon[103278]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:39 smithi078 ceph-mon[91136]: pgmap v4641: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:39 smithi078 ceph-mon[91136]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:39 smithi078 ceph-mon[99959]: pgmap v4641: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:39 smithi078 ceph-mon[99959]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:39.330 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:54:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:54:39.331 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:54:39.640 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:54:39.641 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:54:39.641 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[99959]: from='client.74480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[99959]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4011804583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/942558918' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[91136]: from='client.74480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[91136]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4011804583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/942558918' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:40 smithi146 ceph-mon[103278]: from='client.74480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:40 smithi146 ceph-mon[103278]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4011804583' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/942558918' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:54:41.021 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:54:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:54:41.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:40.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:41.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:40.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:41.022 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:40.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:41 smithi078 ceph-mon[91136]: from='client.74498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:41 smithi078 ceph-mon[91136]: pgmap v4642: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:41 smithi078 ceph-mon[99959]: from='client.74498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:41 smithi078 ceph-mon[99959]: pgmap v4642: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:41.294 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:41 smithi146 ceph-mon[103278]: from='client.74498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:54:41.295 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:41 smithi146 ceph-mon[103278]: pgmap v4642: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:43 smithi078 ceph-mon[99959]: pgmap v4643: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:43 smithi078 ceph-mon[91136]: pgmap v4643: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:43 smithi146 ceph-mon[103278]: pgmap v4643: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:44.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:45 smithi146 ceph-mon[103278]: pgmap v4644: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:45 smithi078 ceph-mon[99959]: pgmap v4644: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:45 smithi078 ceph-mon[91136]: pgmap v4644: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:46.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:46.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:46.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:46.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[91136]: pgmap v4645: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:46.830 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:46.830 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:46.830 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:46.830 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:46 smithi078 ceph-mon[99959]: pgmap v4645: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:46 smithi146 ceph-mon[103278]: pgmap v4645: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:48 smithi078 ceph-mon[91136]: pgmap v4646: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:48 smithi078 ceph-mon[99959]: pgmap v4646: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:48 smithi146 ceph-mon[103278]: pgmap v4646: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:50.972 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:54:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:54:50.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:50.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:50.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:50.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:54:50.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:54:50.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[99959]: pgmap v4647: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[91136]: pgmap v4647: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:51 smithi146 ceph-mon[103278]: pgmap v4647: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:54:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:53 smithi146 ceph-mon[103278]: pgmap v4648: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:53 smithi078 ceph-mon[99959]: pgmap v4648: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:53 smithi078 ceph-mon[91136]: pgmap v4648: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:55 smithi146 ceph-mon[103278]: pgmap v4649: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:55 smithi078 ceph-mon[99959]: pgmap v4649: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:55 smithi078 ceph-mon[91136]: pgmap v4649: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:57 smithi146 ceph-mon[103278]: pgmap v4650: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:57 smithi078 ceph-mon[91136]: pgmap v4650: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:57 smithi078 ceph-mon[99959]: pgmap v4650: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:54:58.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:59 smithi146 ceph-mon[103278]: pgmap v4651: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:59 smithi078 ceph-mon[91136]: pgmap v4651: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:54:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:59 smithi078 ceph-mon[99959]: pgmap v4651: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:54:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:55:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:55:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:00.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:00.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:00.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:01 smithi146 ceph-mon[103278]: pgmap v4652: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:01 smithi078 ceph-mon[91136]: pgmap v4652: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:01 smithi078 ceph-mon[99959]: pgmap v4652: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:03 smithi146 ceph-mon[103278]: pgmap v4653: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:03 smithi078 ceph-mon[91136]: pgmap v4653: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:03 smithi078 ceph-mon[99959]: pgmap v4653: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:05 smithi146 ceph-mon[103278]: pgmap v4654: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:05 smithi078 ceph-mon[91136]: pgmap v4654: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:05 smithi078 ceph-mon[99959]: pgmap v4654: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:06 smithi146 ceph-mon[103278]: pgmap v4655: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[91136]: pgmap v4655: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:06 smithi078 ceph-mon[99959]: pgmap v4655: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:08.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:08 smithi146 ceph-mon[103278]: pgmap v4656: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:08 smithi078 ceph-mon[91136]: pgmap v4656: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:08 smithi078 ceph-mon[99959]: pgmap v4656: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:09.940 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:55:10.481 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:55:10.481 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 2h - - 2024-09-20T22:55:10.481 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 2h - - 2024-09-20T22:55:10.481 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 4s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:55:10.481 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 0s ago 2h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:55:10.481 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 4s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 0s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 4s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 4s ago 2h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 0s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 4s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 4s ago 2h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 0s ago 2h 16.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 4s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 4s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 4s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 4s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 0s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:55:10.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 0s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 0s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 0s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 0s ago 2h 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 4s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 4s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:55:10.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:55:10.732 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:55:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:55:10.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:55:10.799 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:55:11.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:10.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:11.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:10.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:11.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:10.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:11.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:55:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:55:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:55:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:55:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:55:11.078 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: pgmap v4657: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: from='client.54597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3736045011' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: pgmap v4657: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:11.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: from='client.54597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3736045011' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:11.404 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:55:11.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:11.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:11.404 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:11.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: pgmap v4657: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: from='client.54597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3736045011' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:12.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:12 smithi078 ceph-mon[99959]: from='client.54603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:12.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:12 smithi078 ceph-mon[99959]: from='client.54615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:12.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3654445218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:12.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:12 smithi078 ceph-mon[91136]: from='client.54603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:12.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:12 smithi078 ceph-mon[91136]: from='client.54615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:12.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3654445218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:12 smithi146 ceph-mon[103278]: from='client.54603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:12 smithi146 ceph-mon[103278]: from='client.54615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3654445218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:13 smithi146 ceph-mon[103278]: pgmap v4658: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:13 smithi078 ceph-mon[91136]: pgmap v4658: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:13 smithi078 ceph-mon[99959]: pgmap v4658: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:15 smithi146 ceph-mon[103278]: pgmap v4659: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:15 smithi078 ceph-mon[91136]: pgmap v4659: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:15 smithi078 ceph-mon[99959]: pgmap v4659: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:17 smithi078 ceph-mon[91136]: pgmap v4660: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:17 smithi078 ceph-mon[99959]: pgmap v4660: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:17 smithi146 ceph-mon[103278]: pgmap v4660: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:19 smithi146 ceph-mon[103278]: pgmap v4661: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:19 smithi078 ceph-mon[99959]: pgmap v4661: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:19 smithi078 ceph-mon[91136]: pgmap v4661: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:55:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:55:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:20.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:20.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:20.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:21 smithi146 ceph-mon[103278]: pgmap v4662: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:21 smithi078 ceph-mon[99959]: pgmap v4662: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:21 smithi078 ceph-mon[91136]: pgmap v4662: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:23.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:23 smithi146 ceph-mon[103278]: pgmap v4663: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:23 smithi078 ceph-mon[91136]: pgmap v4663: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:23 smithi078 ceph-mon[99959]: pgmap v4663: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:25.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:25 smithi146 ceph-mon[103278]: pgmap v4664: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:25 smithi078 ceph-mon[91136]: pgmap v4664: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:25 smithi078 ceph-mon[99959]: pgmap v4664: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:26 smithi146 ceph-mon[103278]: pgmap v4665: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[99959]: pgmap v4665: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:26 smithi078 ceph-mon[91136]: pgmap v4665: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:28 smithi146 ceph-mon[103278]: pgmap v4666: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:28 smithi078 ceph-mon[91136]: pgmap v4666: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:28 smithi078 ceph-mon[99959]: pgmap v4666: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:31.026 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:55:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:55:31.027 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:30.774Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:31.027 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:30.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:31.027 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:30.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[99959]: pgmap v4667: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[91136]: pgmap v4667: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:31 smithi146 ceph-mon[103278]: pgmap v4667: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:33 smithi078 ceph-mon[91136]: pgmap v4668: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:33 smithi078 ceph-mon[99959]: pgmap v4668: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:33 smithi146 ceph-mon[103278]: pgmap v4668: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:35 smithi078 ceph-mon[91136]: pgmap v4669: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:35 smithi078 ceph-mon[99959]: pgmap v4669: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:35.287 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:35 smithi146 ceph-mon[103278]: pgmap v4669: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:37 smithi078 ceph-mon[91136]: pgmap v4670: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:37 smithi078 ceph-mon[99959]: pgmap v4670: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:37 smithi146 ceph-mon[103278]: pgmap v4670: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:39 smithi146 ceph-mon[103278]: pgmap v4671: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:39 smithi078 ceph-mon[91136]: pgmap v4671: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:39 smithi078 ceph-mon[99959]: pgmap v4671: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:55:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:55:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:40.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:40.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:40.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:41 smithi146 ceph-mon[103278]: pgmap v4672: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:41 smithi078 ceph-mon[99959]: pgmap v4672: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:41 smithi078 ceph-mon[91136]: pgmap v4672: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:41.706 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:55:42.234 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:55:42.234 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 2h - - 2024-09-20T22:55:42.234 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 2h - - 2024-09-20T22:55:42.234 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 16s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:55:42.234 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 11s ago 2h 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:55:42.234 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 16s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 11s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 16s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 16s ago 2h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 11s ago 2h 189M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 16s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 16s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 11s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 16s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 16s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 16s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 16s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 11s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 11s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:55:42.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 11s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:55:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 11s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:55:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 11s ago 2h 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:55:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 16s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:55:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:55:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 16s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:55:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:55:42.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:55:42.549 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:55:42.815 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:55:42.815 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:55:42.816 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:55:43.125 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:55:43.125 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:55:43.125 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:55:43.125 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[99959]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[99959]: from='client.74552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[99959]: pgmap v4673: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[99959]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2965247076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[91136]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[91136]: from='client.74552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[91136]: pgmap v4673: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[91136]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2965247076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:43.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:43 smithi146 ceph-mon[103278]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:43 smithi146 ceph-mon[103278]: from='client.74552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:43 smithi146 ceph-mon[103278]: pgmap v4673: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:43.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:43 smithi146 ceph-mon[103278]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:43.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2965247076' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:55:44.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:44 smithi146 ceph-mon[103278]: from='client.74570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:44.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3564365816' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:44.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:44 smithi078 ceph-mon[91136]: from='client.74570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3564365816' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:44 smithi078 ceph-mon[99959]: from='client.74570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:55:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3564365816' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:55:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:45.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:45 smithi146 ceph-mon[103278]: pgmap v4674: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:45 smithi078 ceph-mon[91136]: pgmap v4674: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:45 smithi078 ceph-mon[99959]: pgmap v4674: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:46 smithi146 ceph-mon[103278]: pgmap v4675: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[91136]: pgmap v4675: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:46 smithi078 ceph-mon[99959]: pgmap v4675: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:48.206 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:48.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:48 smithi146 ceph-mon[103278]: pgmap v4676: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:48 smithi078 ceph-mon[99959]: pgmap v4676: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:48 smithi078 ceph-mon[91136]: pgmap v4676: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:51.033 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:55:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:55:51.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:50.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:51.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:50.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:55:51.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:55:50.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[99959]: pgmap v4677: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[91136]: pgmap v4677: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:51 smithi146 ceph-mon[103278]: pgmap v4677: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:55:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:53 smithi078 ceph-mon[91136]: pgmap v4678: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:53.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:53 smithi078 ceph-mon[99959]: pgmap v4678: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:53 smithi146 ceph-mon[103278]: pgmap v4678: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:55 smithi146 ceph-mon[103278]: pgmap v4679: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:55 smithi078 ceph-mon[91136]: pgmap v4679: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:55 smithi078 ceph-mon[99959]: pgmap v4679: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:57 smithi078 ceph-mon[91136]: pgmap v4680: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:57 smithi078 ceph-mon[99959]: pgmap v4680: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:57.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:57 smithi146 ceph-mon[103278]: pgmap v4680: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:55:58.501 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:59 smithi146 ceph-mon[103278]: pgmap v4681: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:59 smithi078 ceph-mon[99959]: pgmap v4681: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:55:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:59 smithi078 ceph-mon[91136]: pgmap v4681: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:55:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:56:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:56:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:00.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:00.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:00.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:01 smithi146 ceph-mon[103278]: pgmap v4682: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:01 smithi078 ceph-mon[91136]: pgmap v4682: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:01 smithi078 ceph-mon[99959]: pgmap v4682: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:03 smithi146 ceph-mon[103278]: pgmap v4683: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:03 smithi078 ceph-mon[91136]: pgmap v4683: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:03 smithi078 ceph-mon[99959]: pgmap v4683: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:05.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:05 smithi146 ceph-mon[103278]: pgmap v4684: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:05 smithi078 ceph-mon[91136]: pgmap v4684: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:05 smithi078 ceph-mon[99959]: pgmap v4684: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[99959]: pgmap v4685: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[91136]: pgmap v4685: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:07 smithi146 ceph-mon[103278]: pgmap v4685: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:09 smithi146 ceph-mon[103278]: pgmap v4686: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:09 smithi078 ceph-mon[91136]: pgmap v4686: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:09 smithi078 ceph-mon[99959]: pgmap v4686: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:56:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T22:56:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:10.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:10.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:10.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:11 smithi146 ceph-mon[103278]: pgmap v4687: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[91136]: pgmap v4687: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[99959]: pgmap v4687: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:13.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:13 smithi078 ceph-mon[91136]: pgmap v4688: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:13.140 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:13 smithi078 ceph-mon[99959]: pgmap v4688: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:13.426 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:56:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:13 smithi146 ceph-mon[103278]: pgmap v4688: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 2h - - 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 2h - - 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 3s ago 2h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 7s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 3s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 7s ago 2h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:56:13.983 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 3s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 3s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 3s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 3s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 3s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 3s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 3s ago 2h 75.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:56:13.984 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:56:14.301 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:56:14.301 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:56:14.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:56:14.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:56:14.347 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:14.347 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:56:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:56:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:56:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:56:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:56:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:56:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:56:14.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:56:14.896 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:56:14.896 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:14.896 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[99959]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[99959]: from='client.74588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[99959]: from='client.54675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[99959]: pgmap v4689: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1152315014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3767765863' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[91136]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[91136]: from='client.74588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[91136]: from='client.54675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[91136]: pgmap v4689: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1152315014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:15.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3767765863' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:15.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:15 smithi146 ceph-mon[103278]: from='client.54663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:15 smithi146 ceph-mon[103278]: from='client.74588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:15 smithi146 ceph-mon[103278]: from='client.54675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:15.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:15 smithi146 ceph-mon[103278]: pgmap v4689: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:15.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1152315014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:15.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3767765863' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:16 smithi146 ceph-mon[103278]: from='client.54687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:16 smithi078 ceph-mon[91136]: from='client.54687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:16 smithi078 ceph-mon[99959]: from='client.54687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:17 smithi078 ceph-mon[91136]: pgmap v4690: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:17 smithi078 ceph-mon[99959]: pgmap v4690: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:17 smithi146 ceph-mon[103278]: pgmap v4690: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:19.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:19 smithi078 ceph-mon[99959]: pgmap v4691: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:19.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:19 smithi078 ceph-mon[91136]: pgmap v4691: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:19 smithi146 ceph-mon[103278]: pgmap v4691: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:20.779 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:56:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:56:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:20.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:20.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:20.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:21 smithi146 ceph-mon[103278]: pgmap v4692: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:21 smithi078 ceph-mon[99959]: pgmap v4692: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:21 smithi078 ceph-mon[91136]: pgmap v4692: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:23 smithi146 ceph-mon[103278]: pgmap v4693: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:23 smithi078 ceph-mon[91136]: pgmap v4693: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:23.550 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:23 smithi078 ceph-mon[99959]: pgmap v4693: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:25.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:25 smithi146 ceph-mon[103278]: pgmap v4694: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:25 smithi078 ceph-mon[99959]: pgmap v4694: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:25 smithi078 ceph-mon[91136]: pgmap v4694: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:27 smithi146 ceph-mon[103278]: pgmap v4695: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[91136]: pgmap v4695: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[99959]: pgmap v4695: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:29 smithi146 ceph-mon[103278]: pgmap v4696: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:29 smithi078 ceph-mon[91136]: pgmap v4696: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:29 smithi078 ceph-mon[99959]: pgmap v4696: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:30.780 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:56:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:56:31.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:30.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:30.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:30.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:31 smithi146 ceph-mon[103278]: pgmap v4697: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[91136]: pgmap v4697: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[99959]: pgmap v4697: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:33 smithi146 ceph-mon[103278]: pgmap v4698: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:33.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:33 smithi078 ceph-mon[99959]: pgmap v4698: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:33.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:33 smithi078 ceph-mon[91136]: pgmap v4698: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:35 smithi146 ceph-mon[103278]: pgmap v4699: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:35 smithi078 ceph-mon[91136]: pgmap v4699: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:35 smithi078 ceph-mon[99959]: pgmap v4699: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:37 smithi146 ceph-mon[103278]: pgmap v4700: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:37 smithi078 ceph-mon[91136]: pgmap v4700: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:37 smithi078 ceph-mon[99959]: pgmap v4700: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:39 smithi146 ceph-mon[103278]: pgmap v4701: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:39 smithi078 ceph-mon[91136]: pgmap v4701: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:39 smithi078 ceph-mon[99959]: pgmap v4701: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:56:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:56:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:40.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:40.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:40.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:41 smithi146 ceph-mon[103278]: pgmap v4702: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:41 smithi078 ceph-mon[91136]: pgmap v4702: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:41 smithi078 ceph-mon[99959]: pgmap v4702: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:43 smithi146 ceph-mon[103278]: pgmap v4703: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:43 smithi078 ceph-mon[91136]: pgmap v4703: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:43 smithi078 ceph-mon[99959]: pgmap v4703: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:56:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:45.210 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:56:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:45 smithi146 ceph-mon[103278]: pgmap v4704: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:45 smithi078 ceph-mon[99959]: pgmap v4704: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:45 smithi078 ceph-mon[91136]: pgmap v4704: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 2h - - 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 2h - - 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 19s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 14s ago 2h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 19s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 14s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:56:45.749 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 19s ago 2h 571M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 19s ago 2h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 14s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 19s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 19s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 14s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 19s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 19s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 19s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 19s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 14s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 14s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 14s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 14s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 14s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:56:45.751 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 19s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:56:45.751 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:56:45.751 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 19s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:56:45.751 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:56:46.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:56:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:56:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:56:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:56:46.063 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:56:46.322 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:46 smithi078 ceph-mon[99959]: from='client.54699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:46.322 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1101759952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:46.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:46 smithi078 ceph-mon[91136]: from='client.54699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:46.322 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1101759952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:56:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:56:46.340 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:56:46.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:46 smithi146 ceph-mon[103278]: from='client.54699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:46.490 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1101759952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:56:46.648 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:56:46.649 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:56:46.649 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: from='client.74624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: from='client.54711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: pgmap v4705: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1242349882' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: from='client.74624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: from='client.54711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: pgmap v4705: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1242349882' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: from='client.74624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: from='client.54711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: pgmap v4705: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:47.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1242349882' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:56:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:48 smithi146 ceph-mon[103278]: from='client.54723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:48 smithi078 ceph-mon[99959]: from='client.54723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:48 smithi078 ceph-mon[91136]: from='client.54723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:56:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:49 smithi146 ceph-mon[103278]: pgmap v4706: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:49 smithi078 ceph-mon[91136]: pgmap v4706: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:49 smithi078 ceph-mon[99959]: pgmap v4706: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:56:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:56:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:50.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:50.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:56:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:56:50.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:56:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:51 smithi146 ceph-mon[103278]: pgmap v4707: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[91136]: pgmap v4707: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[99959]: pgmap v4707: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:56:53.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:53 smithi078 ceph-mon[91136]: pgmap v4708: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:53.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:53 smithi078 ceph-mon[99959]: pgmap v4708: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:53 smithi146 ceph-mon[103278]: pgmap v4708: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:55.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:55 smithi078 ceph-mon[99959]: pgmap v4709: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:55.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:55 smithi078 ceph-mon[91136]: pgmap v4709: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:55 smithi146 ceph-mon[103278]: pgmap v4709: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:57 smithi146 ceph-mon[103278]: pgmap v4710: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:57 smithi078 ceph-mon[91136]: pgmap v4710: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:57 smithi078 ceph-mon[99959]: pgmap v4710: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:56:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:59 smithi078 ceph-mon[91136]: pgmap v4711: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:56:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:59 smithi078 ceph-mon[99959]: pgmap v4711: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:56:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:56:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:59 smithi146 ceph-mon[103278]: pgmap v4711: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:56:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:56:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:00 smithi078 ceph-mon[91136]: pgmap v4712: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:00 smithi078 ceph-mon[99959]: pgmap v4712: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:00 smithi146 ceph-mon[103278]: pgmap v4712: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:57:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:57:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:00.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:00.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:00.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:03 smithi146 ceph-mon[103278]: pgmap v4713: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:03 smithi078 ceph-mon[99959]: pgmap v4713: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:03 smithi078 ceph-mon[91136]: pgmap v4713: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:05.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:05 smithi146 ceph-mon[103278]: pgmap v4714: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:05 smithi078 ceph-mon[99959]: pgmap v4714: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:05 smithi078 ceph-mon[91136]: pgmap v4714: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:07 smithi146 ceph-mon[103278]: pgmap v4715: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[91136]: pgmap v4715: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[99959]: pgmap v4715: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:09 smithi146 ceph-mon[103278]: pgmap v4716: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:09 smithi078 ceph-mon[99959]: pgmap v4716: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:09 smithi078 ceph-mon[91136]: pgmap v4716: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:57:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T22:57:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:10.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:10.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:10.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:11 smithi146 ceph-mon[103278]: pgmap v4717: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[91136]: pgmap v4717: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[99959]: pgmap v4717: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:13 smithi146 ceph-mon[103278]: pgmap v4718: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:13 smithi078 ceph-mon[91136]: pgmap v4718: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:13 smithi078 ceph-mon[99959]: pgmap v4718: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:14.112 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:14.112 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:15.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:15 smithi078 ceph-mon[91136]: pgmap v4719: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:15.136 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:15 smithi078 ceph-mon[99959]: pgmap v4719: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:15 smithi146 ceph-mon[103278]: pgmap v4719: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:16.941 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 2h - - 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 2h - - 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 11s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 6s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 11s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 6s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 11s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:57:17.492 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 11s ago 2h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 6s ago 2h 190M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 11s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 11s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 6s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 11s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 11s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 11s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 11s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 6s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 6s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 6s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 6s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 6s ago 2h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 11s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:57:17.493 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 11s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:57:17.494 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:57:17.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:17 smithi078 ceph-mon[99959]: pgmap v4720: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:17.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:17 smithi078 ceph-mon[91136]: pgmap v4720: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:17 smithi146 ceph-mon[103278]: pgmap v4720: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:57:17.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:57:17.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:17.812 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:57:17.812 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:57:17.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:57:17.812 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:57:18.090 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:57:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:57:18.091 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:57:18.404 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:57:18.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:18.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:18.404 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:18.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:18.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[91136]: from='client.74654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:18.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:18.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[91136]: from='client.54741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:18.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3011471883' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:18.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[99959]: from='client.74654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:18.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:18.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[99959]: from='client.54741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:18.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3011471883' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:18 smithi146 ceph-mon[103278]: from='client.74654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:18 smithi146 ceph-mon[103278]: from='client.54741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3011471883' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:19 smithi146 ceph-mon[103278]: from='client.54747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:19 smithi146 ceph-mon[103278]: pgmap v4721: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:19 smithi146 ceph-mon[103278]: from='client.54759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4149808134' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[91136]: from='client.54747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[91136]: pgmap v4721: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[91136]: from='client.54759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4149808134' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[99959]: from='client.54747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[99959]: pgmap v4721: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[99959]: from='client.54759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4149808134' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:57:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T22:57:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:20.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:20.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:20.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:21 smithi146 ceph-mon[103278]: pgmap v4722: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:21 smithi078 ceph-mon[99959]: pgmap v4722: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:21 smithi078 ceph-mon[91136]: pgmap v4722: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:23 smithi146 ceph-mon[103278]: pgmap v4723: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:23 smithi078 ceph-mon[99959]: pgmap v4723: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:23 smithi078 ceph-mon[91136]: pgmap v4723: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:25 smithi146 ceph-mon[103278]: pgmap v4724: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:25 smithi078 ceph-mon[99959]: pgmap v4724: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:25 smithi078 ceph-mon[91136]: pgmap v4724: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[99959]: pgmap v4725: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[91136]: pgmap v4725: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.344 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:27 smithi146 ceph-mon[103278]: pgmap v4725: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:27.344 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.344 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:27.344 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:29 smithi146 ceph-mon[103278]: pgmap v4726: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:29 smithi078 ceph-mon[91136]: pgmap v4726: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:29 smithi078 ceph-mon[99959]: pgmap v4726: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:57:30] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:57:31.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:30.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:31.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:30.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:31.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:30.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:31 smithi146 ceph-mon[103278]: pgmap v4727: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:31 smithi078 ceph-mon[91136]: pgmap v4727: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:31 smithi078 ceph-mon[99959]: pgmap v4727: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:33 smithi146 ceph-mon[103278]: pgmap v4728: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:33 smithi078 ceph-mon[91136]: pgmap v4728: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:33 smithi078 ceph-mon[99959]: pgmap v4728: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:35 smithi146 ceph-mon[103278]: pgmap v4729: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:35 smithi078 ceph-mon[91136]: pgmap v4729: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:35 smithi078 ceph-mon[99959]: pgmap v4729: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:37 smithi146 ceph-mon[103278]: pgmap v4730: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:37 smithi078 ceph-mon[91136]: pgmap v4730: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:37 smithi078 ceph-mon[99959]: pgmap v4730: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:39 smithi146 ceph-mon[103278]: pgmap v4731: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:39 smithi078 ceph-mon[91136]: pgmap v4731: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:39 smithi078 ceph-mon[99959]: pgmap v4731: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:57:40] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T22:57:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:40.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:40.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:40.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:41 smithi146 ceph-mon[103278]: pgmap v4732: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:41 smithi078 ceph-mon[99959]: pgmap v4732: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:41 smithi078 ceph-mon[91136]: pgmap v4732: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:43 smithi146 ceph-mon[103278]: pgmap v4733: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:43 smithi078 ceph-mon[91136]: pgmap v4733: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:43 smithi078 ceph-mon[99959]: pgmap v4733: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:45 smithi146 ceph-mon[103278]: pgmap v4734: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:45 smithi078 ceph-mon[99959]: pgmap v4734: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:45 smithi078 ceph-mon[91136]: pgmap v4734: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:47.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:47 smithi146 ceph-mon[103278]: pgmap v4735: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:47.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[91136]: pgmap v4735: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[99959]: pgmap v4735: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:48.705 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:57:49.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:49 smithi078 ceph-mon[99959]: pgmap v4736: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:49 smithi078 ceph-mon[91136]: pgmap v4736: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 2h - - 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 2h - - 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 2s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 18s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 2s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 18s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 2s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 2s ago 2h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 18s ago 2h 191M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 2s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 2s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 18s ago 2h 14.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:57:49.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 2s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 2s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 2s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 2s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 18s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 18s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 18s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 18s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 18s ago 2h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 2s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 2s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:57:49.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:57:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:49 smithi146 ceph-mon[103278]: pgmap v4736: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:49.605 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:57:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:57:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:57:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:57:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:57:49.606 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:57:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:57:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:57:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:57:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:57:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:57:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:57:49.882 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:57:50.192 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:57:50.192 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:57:50.192 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:57:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:50 smithi146 ceph-mon[103278]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:50 smithi146 ceph-mon[103278]: from='client.54777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3789454628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1104047085' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[91136]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[91136]: from='client.54777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3789454628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1104047085' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[99959]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[99959]: from='client.54777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3789454628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:57:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1104047085' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:57:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:57:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:57:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:50.784Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:50.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:57:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:57:50.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:57:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:51 smithi146 ceph-mon[103278]: from='client.74702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:51 smithi146 ceph-mon[103278]: from='client.54795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:51 smithi146 ceph-mon[103278]: pgmap v4737: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:51 smithi078 ceph-mon[91136]: from='client.74702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:51 smithi078 ceph-mon[91136]: from='client.54795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:51 smithi078 ceph-mon[91136]: pgmap v4737: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:51 smithi078 ceph-mon[99959]: from='client.74702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:51 smithi078 ceph-mon[99959]: from='client.54795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:57:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:51 smithi078 ceph-mon[99959]: pgmap v4737: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:52 smithi146 ceph-mon[103278]: pgmap v4738: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[91136]: pgmap v4738: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:57:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:52 smithi078 ceph-mon[99959]: pgmap v4738: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:55.340 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:55 smithi078 ceph-mon[91136]: pgmap v4739: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:55.340 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:55 smithi078 ceph-mon[99959]: pgmap v4739: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:55 smithi146 ceph-mon[103278]: pgmap v4739: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:57.333 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:57 smithi146 ceph-mon[103278]: pgmap v4740: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:57 smithi078 ceph-mon[99959]: pgmap v4740: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:57.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:57 smithi078 ceph-mon[91136]: pgmap v4740: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:57:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:58.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:59 smithi146 ceph-mon[103278]: pgmap v4741: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:57:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:59 smithi078 ceph-mon[91136]: pgmap v4741: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:57:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:57:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:59 smithi078 ceph-mon[99959]: pgmap v4741: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:57:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:57:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:58:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:58:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:00.785Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:00.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:01.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:00.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:01 smithi146 ceph-mon[103278]: pgmap v4742: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:01.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:01 smithi078 ceph-mon[91136]: pgmap v4742: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:01.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:01 smithi078 ceph-mon[99959]: pgmap v4742: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:03 smithi146 ceph-mon[103278]: pgmap v4743: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:03 smithi078 ceph-mon[91136]: pgmap v4743: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:03 smithi078 ceph-mon[99959]: pgmap v4743: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:05 smithi146 ceph-mon[103278]: pgmap v4744: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:05 smithi078 ceph-mon[91136]: pgmap v4744: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:05 smithi078 ceph-mon[99959]: pgmap v4744: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:07.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:07 smithi146 ceph-mon[103278]: pgmap v4745: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:07.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[91136]: pgmap v4745: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[99959]: pgmap v4745: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:07.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:08.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:09 smithi146 ceph-mon[103278]: pgmap v4746: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:09 smithi078 ceph-mon[91136]: pgmap v4746: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:09 smithi078 ceph-mon[99959]: pgmap v4746: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:58:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:58:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:10.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:10.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:10.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:11.436 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:11 smithi146 ceph-mon[103278]: pgmap v4747: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:11 smithi078 ceph-mon[91136]: pgmap v4747: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:11 smithi078 ceph-mon[99959]: pgmap v4747: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:12 smithi146 ceph-mon[103278]: pgmap v4748: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[99959]: pgmap v4748: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:12 smithi078 ceph-mon[91136]: pgmap v4748: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:14 smithi146 ceph-mon[103278]: pgmap v4749: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:14 smithi078 ceph-mon[99959]: pgmap v4749: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:14 smithi078 ceph-mon[91136]: pgmap v4749: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:17 smithi146 ceph-mon[103278]: pgmap v4750: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:17 smithi078 ceph-mon[91136]: pgmap v4750: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:17 smithi078 ceph-mon[99959]: pgmap v4750: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:18.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:19 smithi146 ceph-mon[103278]: pgmap v4751: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:19 smithi078 ceph-mon[99959]: pgmap v4751: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:19 smithi078 ceph-mon[91136]: pgmap v4751: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:20.476 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:58:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:58:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T22:58:21.004 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 2h - - 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 2h - - 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 14s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 9s ago 2h 87.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 9s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 2h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 9s ago 2h 192M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 14s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 9s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:58:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 14s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 9s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 9s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 9s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 9s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 9s ago 2h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 14s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:58:21.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:58:21.094 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:20.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:21.094 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:20.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:21.095 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:20.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:21.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:58:21.317 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:58:21.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:21.318 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:58:21.318 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:58:21.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:58:21.318 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:58:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:21 smithi146 ceph-mon[103278]: pgmap v4752: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:21 smithi078 ceph-mon[99959]: pgmap v4752: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:21 smithi078 ceph-mon[91136]: pgmap v4752: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:21.586 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:58:21.586 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:58:21.586 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:58:21.586 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:58:21.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:58:21.895 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:58:21.895 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:21.895 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:21.895 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:21.895 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[99959]: from='client.74726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[99959]: from='client.74732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[99959]: from='client.74738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2143905078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2894842937' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[91136]: from='client.74726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[91136]: from='client.74732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[91136]: from='client.74738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2143905078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2894842937' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:22 smithi146 ceph-mon[103278]: from='client.74726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:22 smithi146 ceph-mon[103278]: from='client.74732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:22 smithi146 ceph-mon[103278]: from='client.74738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2143905078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2894842937' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:23 smithi146 ceph-mon[103278]: from='client.54831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:23 smithi146 ceph-mon[103278]: pgmap v4753: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:23 smithi078 ceph-mon[91136]: from='client.54831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:23 smithi078 ceph-mon[91136]: pgmap v4753: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:23 smithi078 ceph-mon[99959]: from='client.54831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:23 smithi078 ceph-mon[99959]: pgmap v4753: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:25 smithi146 ceph-mon[103278]: pgmap v4754: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:25 smithi078 ceph-mon[91136]: pgmap v4754: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:25 smithi078 ceph-mon[99959]: pgmap v4754: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[91136]: pgmap v4755: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[99959]: pgmap v4755: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:27 smithi146 ceph-mon[103278]: pgmap v4755: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:27.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:27.366 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:28.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:29.334 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:29 smithi146 ceph-mon[103278]: pgmap v4756: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:29.334 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:29 smithi078 ceph-mon[91136]: pgmap v4756: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:29 smithi078 ceph-mon[99959]: pgmap v4756: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:58:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:58:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:30.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:30.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:30.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:31 smithi146 ceph-mon[103278]: pgmap v4757: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:31 smithi078 ceph-mon[91136]: pgmap v4757: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:31 smithi078 ceph-mon[99959]: pgmap v4757: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:32 smithi146 ceph-mon[103278]: pgmap v4758: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[91136]: pgmap v4758: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:33.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:32 smithi078 ceph-mon[99959]: pgmap v4758: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:35 smithi146 ceph-mon[103278]: pgmap v4759: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:35 smithi078 ceph-mon[91136]: pgmap v4759: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:35 smithi078 ceph-mon[99959]: pgmap v4759: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:37 smithi078 ceph-mon[99959]: pgmap v4760: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:37 smithi078 ceph-mon[91136]: pgmap v4760: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:37 smithi146 ceph-mon[103278]: pgmap v4760: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:39 smithi146 ceph-mon[103278]: pgmap v4761: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:39 smithi078 ceph-mon[91136]: pgmap v4761: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:39 smithi078 ceph-mon[99959]: pgmap v4761: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:58:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:58:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:40.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:40.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:40.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:41 smithi146 ceph-mon[103278]: pgmap v4762: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:41 smithi078 ceph-mon[99959]: pgmap v4762: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:41 smithi078 ceph-mon[91136]: pgmap v4762: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:43 smithi146 ceph-mon[103278]: pgmap v4763: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:43 smithi078 ceph-mon[91136]: pgmap v4763: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:43 smithi078 ceph-mon[99959]: pgmap v4763: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:45 smithi146 ceph-mon[103278]: pgmap v4764: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:45 smithi078 ceph-mon[99959]: pgmap v4764: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:45 smithi078 ceph-mon[91136]: pgmap v4764: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[91136]: pgmap v4765: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[99959]: pgmap v4765: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:47 smithi146 ceph-mon[103278]: pgmap v4765: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:48.415 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:49.356 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:49 smithi146 ceph-mon[103278]: pgmap v4766: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:49 smithi078 ceph-mon[91136]: pgmap v4766: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:49 smithi078 ceph-mon[99959]: pgmap v4766: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:58:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:58:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:50.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:50.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:58:51.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:58:50.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:58:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:51 smithi146 ceph-mon[103278]: pgmap v4767: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:51 smithi078 ceph-mon[91136]: pgmap v4767: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:51 smithi078 ceph-mon[99959]: pgmap v4767: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:52.191 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 2h - - 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 2h - - 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 5s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 1s ago 2h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 5s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 1s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 5s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 5s ago 2h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:58:52.721 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 1s ago 2h 193M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 5s ago 2h 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 5s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 1s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 5s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 5s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 5s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 5s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 1s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 1s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 1s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 1s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 1s ago 2h 77.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 5s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:58:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 5s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:58:52.723 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:58:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:52 smithi146 ceph-mon[103278]: pgmap v4768: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:52 smithi146 ceph-mon[103278]: from='client.54843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:58:53.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:58:53.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:58:53.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[99959]: pgmap v4768: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:53.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[99959]: from='client.54843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:58:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[91136]: pgmap v4768: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:52 smithi078 ceph-mon[91136]: from='client.54843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:58:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:58:53.305 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:58:53.612 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:58:53.613 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:58:53.613 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:58:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:53 smithi146 ceph-mon[103278]: from='client.54849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:53 smithi146 ceph-mon[103278]: from='client.54855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2324875397' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/253245689' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[99959]: from='client.54849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[99959]: from='client.54855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2324875397' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/253245689' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[91136]: from='client.54849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[91136]: from='client.54855 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2324875397' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:58:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/253245689' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:58:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:54 smithi146 ceph-mon[103278]: from='client.54867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:54 smithi146 ceph-mon[103278]: pgmap v4769: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:54 smithi078 ceph-mon[91136]: from='client.54867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:54 smithi078 ceph-mon[91136]: pgmap v4769: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:54 smithi078 ceph-mon[99959]: from='client.54867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:58:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:54 smithi078 ceph-mon[99959]: pgmap v4769: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:57 smithi146 ceph-mon[103278]: pgmap v4770: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:57 smithi078 ceph-mon[91136]: pgmap v4770: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:57 smithi078 ceph-mon[99959]: pgmap v4770: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:58.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:58.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:59.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:59 smithi078 ceph-mon[91136]: pgmap v4771: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:59.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:59.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:59 smithi078 ceph-mon[99959]: pgmap v4771: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:59.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:59 smithi146 ceph-mon[103278]: pgmap v4771: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:59:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:59:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:00.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:00.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:00.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:01 smithi146 ceph-mon[103278]: pgmap v4772: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:01 smithi078 ceph-mon[99959]: pgmap v4772: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:01 smithi078 ceph-mon[91136]: pgmap v4772: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:03 smithi146 ceph-mon[103278]: pgmap v4773: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:03 smithi078 ceph-mon[99959]: pgmap v4773: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:03 smithi078 ceph-mon[91136]: pgmap v4773: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:05 smithi146 ceph-mon[103278]: pgmap v4774: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:05 smithi078 ceph-mon[99959]: pgmap v4774: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:05 smithi078 ceph-mon[91136]: pgmap v4774: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:07 smithi146 ceph-mon[103278]: pgmap v4775: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:07 smithi078 ceph-mon[91136]: pgmap v4775: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:07 smithi078 ceph-mon[99959]: pgmap v4775: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:09.494 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:09 smithi146 ceph-mon[103278]: pgmap v4776: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:09 smithi078 ceph-mon[91136]: pgmap v4776: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:09 smithi078 ceph-mon[99959]: pgmap v4776: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:59:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:59:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:10.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:10.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:10.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:11 smithi146 ceph-mon[103278]: pgmap v4777: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:11 smithi078 ceph-mon[99959]: pgmap v4777: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:11 smithi078 ceph-mon[91136]: pgmap v4777: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:12 smithi146 ceph-mon[103278]: pgmap v4778: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[91136]: pgmap v4778: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:12 smithi078 ceph-mon[99959]: pgmap v4778: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:14 smithi146 ceph-mon[103278]: pgmap v4779: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:14 smithi078 ceph-mon[91136]: pgmap v4779: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:14 smithi078 ceph-mon[99959]: pgmap v4779: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:17 smithi146 ceph-mon[103278]: pgmap v4780: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:17 smithi078 ceph-mon[99959]: pgmap v4780: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:17 smithi078 ceph-mon[91136]: pgmap v4780: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:18.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:18.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:19 smithi078 ceph-mon[91136]: pgmap v4781: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:19 smithi078 ceph-mon[99959]: pgmap v4781: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:19 smithi146 ceph-mon[103278]: pgmap v4781: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:59:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:59:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:20.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:20.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:20.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:21 smithi146 ceph-mon[103278]: pgmap v4782: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:21 smithi078 ceph-mon[91136]: pgmap v4782: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:21 smithi078 ceph-mon[99959]: pgmap v4782: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:23 smithi146 ceph-mon[103278]: pgmap v4783: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:23 smithi078 ceph-mon[91136]: pgmap v4783: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:23 smithi078 ceph-mon[99959]: pgmap v4783: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:23.911 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:59:24.438 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 2h - - 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 2h - - 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 17s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 12s ago 2h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 12s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 2h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 12s ago 2h 194M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:59:24.439 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 17s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 12s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 12s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 12s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 12s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 12s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 12s ago 2h 77.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:59:24.440 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:24.748 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:59:24.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:59:24.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:59:24.749 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:59:25.018 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:59:25.019 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:59:25.019 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:59:25.019 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:59:25.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:59:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[99959]: from='client.54879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[99959]: pgmap v4784: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[99959]: from='client.74804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2675292612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[91136]: from='client.54879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[91136]: pgmap v4784: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[91136]: from='client.74804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2675292612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:25.331 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:59:25.332 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:25.332 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:25.332 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:25.332 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:25 smithi146 ceph-mon[103278]: from='client.54879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:25 smithi146 ceph-mon[103278]: pgmap v4784: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:25 smithi146 ceph-mon[103278]: from='client.74804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2675292612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:26 smithi146 ceph-mon[103278]: from='client.54891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:26 smithi146 ceph-mon[103278]: from='client.74822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3006180328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:26 smithi078 ceph-mon[91136]: from='client.54891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:26 smithi078 ceph-mon[91136]: from='client.74822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3006180328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:26 smithi078 ceph-mon[99959]: from='client.54891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:26 smithi078 ceph-mon[99959]: from='client.74822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3006180328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:27.367 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:27 smithi078 ceph-mon[91136]: pgmap v4785: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:27.368 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:27 smithi078 ceph-mon[99959]: pgmap v4785: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:27.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:27 smithi146 ceph-mon[103278]: pgmap v4785: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:29.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:29 smithi146 ceph-mon[103278]: pgmap v4786: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:29.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:29 smithi078 ceph-mon[91136]: pgmap v4786: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:29 smithi078 ceph-mon[99959]: pgmap v4786: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:59:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:59:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:30.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:30.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:30.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:31 smithi146 ceph-mon[103278]: pgmap v4787: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:31 smithi078 ceph-mon[99959]: pgmap v4787: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:31 smithi078 ceph-mon[91136]: pgmap v4787: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:32 smithi146 ceph-mon[103278]: pgmap v4788: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[99959]: pgmap v4788: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:33.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:32 smithi078 ceph-mon[91136]: pgmap v4788: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:35 smithi146 ceph-mon[103278]: pgmap v4789: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:35 smithi078 ceph-mon[91136]: pgmap v4789: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:35 smithi078 ceph-mon[99959]: pgmap v4789: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:37 smithi146 ceph-mon[103278]: pgmap v4790: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:37 smithi078 ceph-mon[91136]: pgmap v4790: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:37 smithi078 ceph-mon[99959]: pgmap v4790: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:38.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:38.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:39.442 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:39 smithi078 ceph-mon[99959]: pgmap v4791: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:39.442 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:39 smithi078 ceph-mon[91136]: pgmap v4791: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:39 smithi146 ceph-mon[103278]: pgmap v4791: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:40.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:59:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T22:59:40.925 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:40.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:40.925 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:40.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:40.925 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:40.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:41 smithi078 ceph-mon[91136]: pgmap v4792: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:41 smithi078 ceph-mon[99959]: pgmap v4792: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:41 smithi146 ceph-mon[103278]: pgmap v4792: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:43 smithi146 ceph-mon[103278]: pgmap v4793: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:43 smithi078 ceph-mon[91136]: pgmap v4793: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:43 smithi078 ceph-mon[99959]: pgmap v4793: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:45 smithi146 ceph-mon[103278]: pgmap v4794: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:45 smithi078 ceph-mon[91136]: pgmap v4794: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:45 smithi078 ceph-mon[99959]: pgmap v4794: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:47.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:47 smithi078 ceph-mon[99959]: pgmap v4795: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:47.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:47 smithi078 ceph-mon[91136]: pgmap v4795: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:47 smithi146 ceph-mon[103278]: pgmap v4795: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:49 smithi078 ceph-mon[99959]: pgmap v4796: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:49 smithi078 ceph-mon[91136]: pgmap v4796: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:49 smithi146 ceph-mon[103278]: pgmap v4796: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 22:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:22:59:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T22:59:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:50.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:50.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T22:59:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 22:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T22:59:50.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T22:59:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:51 smithi146 ceph-mon[103278]: pgmap v4797: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:51 smithi078 ceph-mon[91136]: pgmap v4797: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:51 smithi078 ceph-mon[99959]: pgmap v4797: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:53 smithi146 ceph-mon[103278]: pgmap v4798: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[99959]: pgmap v4798: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[91136]: pgmap v4798: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T22:59:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:55 smithi146 ceph-mon[103278]: pgmap v4799: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:55 smithi078 ceph-mon[99959]: pgmap v4799: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:55 smithi078 ceph-mon[91136]: pgmap v4799: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:55.629 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 2h - - 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 2h - - 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 8s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 4s ago 2h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 8s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 4s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T22:59:56.158 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 8s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 8s ago 2h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 4s ago 2h 195M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 8s ago 2h 202M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 8s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 4s ago 2h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 8s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 8s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 8s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 8s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 4s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 4s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 4s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 4s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 4s ago 2h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T22:59:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 8s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T22:59:56.160 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T22:59:56.160 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 8s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T22:59:56.160 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T22:59:56.467 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:59:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T22:59:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T22:59:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T22:59:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T22:59:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T22:59:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T22:59:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T22:59:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T22:59:56.469 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:59:56.733 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:59:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T22:59:56.735 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T22:59:57.041 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T22:59:57.041 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T22:59:57.041 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:57.041 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T22:59:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:57 smithi146 ceph-mon[103278]: from='client.74834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:57 smithi146 ceph-mon[103278]: from='client.74840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:57 smithi146 ceph-mon[103278]: pgmap v4800: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:57 smithi146 ceph-mon[103278]: from='client.74846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/807505101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3201132771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[99959]: from='client.74834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[99959]: from='client.74840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[99959]: pgmap v4800: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[99959]: from='client.74846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/807505101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3201132771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[91136]: from='client.74834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[91136]: from='client.74840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[91136]: pgmap v4800: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[91136]: from='client.74846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/807505101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T22:59:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3201132771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T22:59:58.334 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:58 smithi146 ceph-mon[103278]: from='client.74858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:58.334 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:58 smithi078 ceph-mon[99959]: from='client.74858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:58 smithi078 ceph-mon[91136]: from='client.74858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:59:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:59.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:59 smithi078 ceph-mon[91136]: pgmap v4801: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:59.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 22:59:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:59.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:59 smithi078 ceph-mon[99959]: pgmap v4801: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:59.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 22:59:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T22:59:59.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:59 smithi146 ceph-mon[103278]: pgmap v4801: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:59.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 22:59:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T22:59:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:00.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:00:00] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T23:00:00.899 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:00.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:00.899 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:00.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:00.899 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:00.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:01.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:01 smithi078 ceph-mon[91136]: pgmap v4802: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:01.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:01 smithi078 ceph-mon[99959]: pgmap v4802: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:01 smithi146 ceph-mon[103278]: pgmap v4802: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:03 smithi146 ceph-mon[103278]: pgmap v4803: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:03 smithi078 ceph-mon[91136]: pgmap v4803: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:03 smithi078 ceph-mon[99959]: pgmap v4803: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:05 smithi146 ceph-mon[103278]: pgmap v4804: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:05 smithi078 ceph-mon[91136]: pgmap v4804: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:05 smithi078 ceph-mon[99959]: pgmap v4804: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:07 smithi146 ceph-mon[103278]: pgmap v4805: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:07 smithi078 ceph-mon[99959]: pgmap v4805: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:07 smithi078 ceph-mon[91136]: pgmap v4805: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:09 smithi146 ceph-mon[103278]: pgmap v4806: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:09 smithi078 ceph-mon[99959]: pgmap v4806: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:09 smithi078 ceph-mon[91136]: pgmap v4806: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:00:10] "GET /metrics HTTP/1.1" 200 38983 "" "Prometheus/2.43.0" 2024-09-20T23:00:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:10.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:10.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:10.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:11.360 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:11 smithi146 ceph-mon[103278]: pgmap v4807: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:11 smithi078 ceph-mon[91136]: pgmap v4807: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:11 smithi078 ceph-mon[99959]: pgmap v4807: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:13 smithi146 ceph-mon[103278]: pgmap v4808: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[91136]: pgmap v4808: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[99959]: pgmap v4808: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:15 smithi146 ceph-mon[103278]: pgmap v4809: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:15 smithi078 ceph-mon[91136]: pgmap v4809: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:15 smithi078 ceph-mon[99959]: pgmap v4809: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:17 smithi146 ceph-mon[103278]: pgmap v4810: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:17 smithi078 ceph-mon[99959]: pgmap v4810: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:17 smithi078 ceph-mon[91136]: pgmap v4810: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:19 smithi146 ceph-mon[103278]: pgmap v4811: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:19 smithi078 ceph-mon[99959]: pgmap v4811: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:19 smithi078 ceph-mon[91136]: pgmap v4811: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:00:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:00:21.065 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:20.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:21.065 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:20.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:21.065 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:20.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:21.318 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:21 smithi078 ceph-mon[99959]: pgmap v4812: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:21.318 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:21 smithi078 ceph-mon[91136]: pgmap v4812: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:21 smithi146 ceph-mon[103278]: pgmap v4812: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:23 smithi146 ceph-mon[103278]: pgmap v4813: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:23 smithi078 ceph-mon[99959]: pgmap v4813: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:23 smithi078 ceph-mon[91136]: pgmap v4813: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:25 smithi146 ceph-mon[103278]: pgmap v4814: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:25 smithi078 ceph-mon[91136]: pgmap v4814: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:25.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:25 smithi078 ceph-mon[99959]: pgmap v4814: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:27.336 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:00:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:27 smithi146 ceph-mon[103278]: pgmap v4815: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:27 smithi078 ceph-mon[91136]: pgmap v4815: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:27 smithi078 ceph-mon[99959]: pgmap v4815: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:27.874 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:00:27.874 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 2h - - 2024-09-20T23:00:27.874 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 2h - - 2024-09-20T23:00:27.874 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 0s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:00:27.874 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 15s ago 2h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 0s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 15s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 0s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 0s ago 2h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 15s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 0s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 0s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 15s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 0s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 0s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 0s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 0s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 15s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 15s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:00:27.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 15s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:00:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 15s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:00:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 15s ago 2h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:00:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 0s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:00:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:00:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 0s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:00:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:00:28.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:00:28.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:28.188 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:00:28.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:00:28.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:00:28.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:00:28.470 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:00:28.471 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:00:28.471 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:00:28.471 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:00:28.471 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:00:28.471 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:00:28.471 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:00:28.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:28.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/452122366' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:28.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:28.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/452122366' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/452122366' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:00:28.783 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:00:28.784 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:00:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:00:28.784 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:00:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:00:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:29 smithi146 ceph-mon[103278]: from='client.54951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:29 smithi146 ceph-mon[103278]: from='client.74876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:29 smithi146 ceph-mon[103278]: from='client.74882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:29 smithi146 ceph-mon[103278]: pgmap v4816: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:29.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4112074889' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[99959]: from='client.54951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[99959]: from='client.74876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[99959]: from='client.74882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[99959]: pgmap v4816: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4112074889' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[91136]: from='client.54951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[91136]: from='client.74876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[91136]: from='client.74882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[91136]: pgmap v4816: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:29.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4112074889' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:00:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:30 smithi146 ceph-mon[103278]: from='client.74894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:30 smithi078 ceph-mon[91136]: from='client.74894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:30 smithi078 ceph-mon[99959]: from='client.74894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:00:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:00:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:00:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:30.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:30.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:30.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:31 smithi146 ceph-mon[103278]: pgmap v4817: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:31 smithi078 ceph-mon[91136]: pgmap v4817: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:31 smithi078 ceph-mon[99959]: pgmap v4817: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:33 smithi146 ceph-mon[103278]: pgmap v4818: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[91136]: pgmap v4818: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[99959]: pgmap v4818: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:35 smithi146 ceph-mon[103278]: pgmap v4819: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:35 smithi078 ceph-mon[91136]: pgmap v4819: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:35 smithi078 ceph-mon[99959]: pgmap v4819: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:37 smithi146 ceph-mon[103278]: pgmap v4820: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:37 smithi078 ceph-mon[91136]: pgmap v4820: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:37 smithi078 ceph-mon[99959]: pgmap v4820: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:39 smithi146 ceph-mon[103278]: pgmap v4821: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:39 smithi078 ceph-mon[91136]: pgmap v4821: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:39 smithi078 ceph-mon[99959]: pgmap v4821: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:40.682 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:00:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:00:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:40.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:40.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:40.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:41 smithi146 ceph-mon[103278]: pgmap v4822: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:41 smithi078 ceph-mon[99959]: pgmap v4822: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:41 smithi078 ceph-mon[91136]: pgmap v4822: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:43 smithi146 ceph-mon[103278]: pgmap v4823: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:43 smithi078 ceph-mon[91136]: pgmap v4823: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:43 smithi078 ceph-mon[99959]: pgmap v4823: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:45 smithi146 ceph-mon[103278]: pgmap v4824: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:45 smithi078 ceph-mon[91136]: pgmap v4824: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:45 smithi078 ceph-mon[99959]: pgmap v4824: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:47 smithi078 ceph-mon[91136]: pgmap v4825: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:47 smithi078 ceph-mon[99959]: pgmap v4825: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:47 smithi146 ceph-mon[103278]: pgmap v4825: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:49 smithi078 ceph-mon[91136]: pgmap v4826: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:49 smithi078 ceph-mon[99959]: pgmap v4826: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:49 smithi146 ceph-mon[103278]: pgmap v4826: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:00:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:00:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:00:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:50.796Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:50.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:00:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:00:50.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:00:51.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:51 smithi146 ceph-mon[103278]: pgmap v4827: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:51 smithi078 ceph-mon[91136]: pgmap v4827: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:51 smithi078 ceph-mon[99959]: pgmap v4827: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[91136]: pgmap v4828: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[99959]: pgmap v4828: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:53 smithi146 ceph-mon[103278]: pgmap v4828: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:00:53.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:53.639 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:00:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:55 smithi078 ceph-mon[91136]: pgmap v4829: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:00:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:55 smithi078 ceph-mon[99959]: pgmap v4829: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:00:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:55 smithi146 ceph-mon[103278]: pgmap v4829: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:00:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:57 smithi078 ceph-mon[91136]: pgmap v4830: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:00:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:57 smithi078 ceph-mon[99959]: pgmap v4830: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:00:57.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:57 smithi146 ceph-mon[103278]: pgmap v4830: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:00:58.579 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:00:59.086 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:00:59.352 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:59 smithi078 ceph-mon[99959]: pgmap v4831: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:00:59.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:59.353 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:59 smithi078 ceph-mon[91136]: pgmap v4831: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:00:59.353 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:59.617 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 2h - - 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 2h - - 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 11s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 7s ago 2h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 11s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 7s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 11s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 11s ago 2h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 7s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 11s ago 2h 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 11s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:00:59.618 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 7s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 11s ago 2h 76.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 11s ago 2h 78.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 11s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 11s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 7s ago 2h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 7s ago 2h 70.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 7s ago 2h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 7s ago 2h 70.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 7s ago 2h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 11s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 7s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 11s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:00:59.619 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 7s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:00:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:59 smithi146 ceph-mon[103278]: pgmap v4831: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:00:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:00:59.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:00:59.934 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:01:00.201 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:01:00.202 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:01:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:01:00.203 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:01:00.203 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:01:00.513 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:01:00.514 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:00.514 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:00.514 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:00.514 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:00.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-mon[91136]: from='client.54987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:00.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2394943730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:00.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-mon[99959]: from='client.54987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:00.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2394943730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:00 smithi146 ceph-mon[103278]: from='client.54987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2394943730' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:01:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:01:01.105 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:00.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:01.105 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:00.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:01.105 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:00.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:01.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[91136]: from='client.74912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[91136]: from='client.54999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[91136]: pgmap v4832: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[91136]: from='client.55011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/281932688' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[99959]: from='client.74912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[99959]: from='client.54999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[99959]: pgmap v4832: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[99959]: from='client.55011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/281932688' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:01 smithi146 ceph-mon[103278]: from='client.74912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:01 smithi146 ceph-mon[103278]: from='client.54999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:01 smithi146 ceph-mon[103278]: pgmap v4832: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:01:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:01 smithi146 ceph-mon[103278]: from='client.55011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/281932688' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:02.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:02 smithi078 ceph-mon[99959]: pgmap v4833: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:01:02.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:02 smithi078 ceph-mon[91136]: pgmap v4833: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:01:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:02 smithi146 ceph-mon[103278]: pgmap v4833: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:01:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:05 smithi146 ceph-mon[103278]: pgmap v4834: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:01:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:05 smithi078 ceph-mon[91136]: pgmap v4834: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:01:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:05 smithi078 ceph-mon[99959]: pgmap v4834: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:01:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:07 smithi146 ceph-mon[103278]: pgmap v4835: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:01:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:07 smithi078 ceph-mon[91136]: pgmap v4835: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:01:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:07 smithi078 ceph-mon[99959]: pgmap v4835: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:01:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:09 smithi146 ceph-mon[103278]: pgmap v4836: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:01:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:09 smithi078 ceph-mon[91136]: pgmap v4836: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:01:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:09 smithi078 ceph-mon[99959]: pgmap v4836: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:01:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:01:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:01:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:10.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:10.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:10.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:11 smithi146 ceph-mon[103278]: pgmap v4837: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:01:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:11 smithi078 ceph-mon[99959]: pgmap v4837: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:01:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:11 smithi078 ceph-mon[91136]: pgmap v4837: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:01:13.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:13 smithi146 ceph-mon[103278]: pgmap v4838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T23:01:13.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.367 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[91136]: pgmap v4838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[99959]: pgmap v4838: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:15 smithi146 ceph-mon[103278]: pgmap v4839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-20T23:01:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:15 smithi078 ceph-mon[91136]: pgmap v4839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-20T23:01:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:15 smithi078 ceph-mon[99959]: pgmap v4839: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-20T23:01:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:17 smithi146 ceph-mon[103278]: pgmap v4840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T23:01:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:17 smithi078 ceph-mon[91136]: pgmap v4840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T23:01:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:17 smithi078 ceph-mon[99959]: pgmap v4840: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T23:01:18.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:18.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:19 smithi146 ceph-mon[103278]: pgmap v4841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:01:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:19 smithi078 ceph-mon[99959]: pgmap v4841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:01:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:19 smithi078 ceph-mon[91136]: pgmap v4841: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:01:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:01:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:01:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:20.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:20.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:20.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:21 smithi146 ceph-mon[103278]: pgmap v4842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:01:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:21 smithi078 ceph-mon[99959]: pgmap v4842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:01:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:21 smithi078 ceph-mon[91136]: pgmap v4842: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:01:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:23 smithi146 ceph-mon[103278]: pgmap v4843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:01:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:23 smithi078 ceph-mon[91136]: pgmap v4843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:01:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:23 smithi078 ceph-mon[99959]: pgmap v4843: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:01:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:25 smithi146 ceph-mon[103278]: pgmap v4844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:01:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:25 smithi078 ceph-mon[91136]: pgmap v4844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:01:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:25 smithi078 ceph-mon[99959]: pgmap v4844: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:01:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:27 smithi146 ceph-mon[103278]: pgmap v4845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-20T23:01:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:27 smithi078 ceph-mon[91136]: pgmap v4845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-20T23:01:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:27 smithi078 ceph-mon[99959]: pgmap v4845: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-20T23:01:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:29 smithi146 ceph-mon[103278]: pgmap v4846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T23:01:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:29 smithi078 ceph-mon[99959]: pgmap v4846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T23:01:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:29 smithi078 ceph-mon[91136]: pgmap v4846: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T23:01:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:30.806 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:01:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:01:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:01:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:30.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:30.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:30.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 2h - - 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 2h - - 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 3s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 18s ago 2h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 3s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 18s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:01:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 3s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 3s ago 2h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 18s ago 2h 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 3s ago 2h 202M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 3s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 18s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 3s ago 2h 76.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 3s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 3s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 3s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 18s ago 2h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 18s ago 2h 70.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 18s ago 2h 74.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 18s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:01:31.341 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 18s ago 2h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:01:31.342 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 3s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:01:31.342 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:01:31.342 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 3s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:01:31.342 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:01:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:31 smithi146 ceph-mon[103278]: pgmap v4847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:31.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:31 smithi078 ceph-mon[99959]: pgmap v4847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:31.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:31 smithi078 ceph-mon[91136]: pgmap v4847: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:01:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:01:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:01:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:01:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:01:31.651 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:01:31.919 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:01:31.920 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:01:31.920 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:01:31.920 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:01:31.920 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:01:32.225 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:01:32.225 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:32.225 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:01:32.225 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:32.225 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:01:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:32 smithi146 ceph-mon[103278]: from='client.55023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:32 smithi146 ceph-mon[103278]: from='client.55029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3686817628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:32 smithi078 ceph-mon[91136]: from='client.55023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:32 smithi078 ceph-mon[91136]: from='client.55029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3686817628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:32 smithi078 ceph-mon[99959]: from='client.55023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:32 smithi078 ceph-mon[99959]: from='client.55029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3686817628' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:01:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: from='client.74954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: from='client.74966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: pgmap v4848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3599521082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: from='client.74954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: from='client.74966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: pgmap v4848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3599521082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: from='client.74954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: from='client.74966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: pgmap v4848: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3599521082' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:01:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:35 smithi146 ceph-mon[103278]: pgmap v4849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:35 smithi078 ceph-mon[91136]: pgmap v4849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:35 smithi078 ceph-mon[99959]: pgmap v4849: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:37 smithi146 ceph-mon[103278]: pgmap v4850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:37 smithi078 ceph-mon[99959]: pgmap v4850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:37 smithi078 ceph-mon[91136]: pgmap v4850: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:39 smithi146 ceph-mon[103278]: pgmap v4851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:39 smithi078 ceph-mon[91136]: pgmap v4851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:39 smithi078 ceph-mon[99959]: pgmap v4851: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:01:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:01:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:40.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:40.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:40.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:41 smithi146 ceph-mon[103278]: pgmap v4852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:41 smithi078 ceph-mon[91136]: pgmap v4852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:41 smithi078 ceph-mon[99959]: pgmap v4852: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:43 smithi078 ceph-mon[91136]: pgmap v4853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:43 smithi078 ceph-mon[99959]: pgmap v4853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:43 smithi146 ceph-mon[103278]: pgmap v4853: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:44.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:44.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:45 smithi146 ceph-mon[103278]: pgmap v4854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:45 smithi078 ceph-mon[91136]: pgmap v4854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:45 smithi078 ceph-mon[99959]: pgmap v4854: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:47 smithi146 ceph-mon[103278]: pgmap v4855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:47 smithi078 ceph-mon[91136]: pgmap v4855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:47 smithi078 ceph-mon[99959]: pgmap v4855: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:49 smithi146 ceph-mon[103278]: pgmap v4856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[99959]: pgmap v4856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[91136]: pgmap v4856: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:01:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:01:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:50.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:50.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:01:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:01:50.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:01:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:51 smithi146 ceph-mon[103278]: pgmap v4857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:51 smithi078 ceph-mon[91136]: pgmap v4857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:51 smithi078 ceph-mon[99959]: pgmap v4857: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:53 smithi146 ceph-mon[103278]: pgmap v4858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[91136]: pgmap v4858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[99959]: pgmap v4858: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:01:55.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:55 smithi146 ceph-mon[103278]: pgmap v4859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:55 smithi078 ceph-mon[91136]: pgmap v4859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:55 smithi078 ceph-mon[99959]: pgmap v4859: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:57 smithi146 ceph-mon[103278]: pgmap v4860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:57 smithi078 ceph-mon[99959]: pgmap v4860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:57 smithi078 ceph-mon[91136]: pgmap v4860: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:01:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:01:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:59 smithi146 ceph-mon[103278]: pgmap v4861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:01:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:59 smithi078 ceph-mon[99959]: pgmap v4861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:01:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:01:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:59 smithi078 ceph-mon[91136]: pgmap v4861: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:01:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:01:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:02:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:02:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:00.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:00.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:00.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:01 smithi146 ceph-mon[103278]: pgmap v4862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:01 smithi078 ceph-mon[91136]: pgmap v4862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:01 smithi078 ceph-mon[99959]: pgmap v4862: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:02.530 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 2h - - 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 2h - - 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 14s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 10s ago 2h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 10s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 2h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 10s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 14s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 10s ago 2h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 14s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 10s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 10s ago 2h 70.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 10s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 10s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 10s ago 2h 77.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 10s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 14s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 10s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:02:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:02:03.387 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:02:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:03 smithi078 ceph-mon[99959]: pgmap v4863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:03 smithi078 ceph-mon[91136]: pgmap v4863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:03.659 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:02:03.659 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:02:03.659 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:02:03.659 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:02:03.659 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:02:03.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:02:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:02:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:03 smithi146 ceph-mon[103278]: pgmap v4863: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:03.979 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:02:03.979 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:02:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:02:03.979 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:02:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:02:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[91136]: from='client.55059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[91136]: from='client.74984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[91136]: from='client.74990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/903283435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1122765471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:04.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[99959]: from='client.55059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[99959]: from='client.74984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[99959]: from='client.74990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/903283435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:04.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1122765471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:04 smithi146 ceph-mon[103278]: from='client.55059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:04 smithi146 ceph-mon[103278]: from='client.74984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:04 smithi146 ceph-mon[103278]: from='client.74990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/903283435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1122765471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:05.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:05 smithi078 ceph-mon[99959]: from='client.75002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:05.420 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:05 smithi078 ceph-mon[99959]: pgmap v4864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:05.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:05 smithi078 ceph-mon[91136]: from='client.75002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:05.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:05 smithi078 ceph-mon[91136]: pgmap v4864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:05 smithi146 ceph-mon[103278]: from='client.75002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:05 smithi146 ceph-mon[103278]: pgmap v4864: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:07 smithi078 ceph-mon[91136]: pgmap v4865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:07 smithi078 ceph-mon[99959]: pgmap v4865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:07 smithi146 ceph-mon[103278]: pgmap v4865: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[99959]: pgmap v4866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[91136]: pgmap v4866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:09 smithi146 ceph-mon[103278]: pgmap v4866: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:02:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:02:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:10.801Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:10.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:10.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:11 smithi078 ceph-mon[91136]: pgmap v4867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:11 smithi078 ceph-mon[99959]: pgmap v4867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:11 smithi146 ceph-mon[103278]: pgmap v4867: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:13.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:13 smithi146 ceph-mon[103278]: pgmap v4868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:13.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[91136]: pgmap v4868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[99959]: pgmap v4868: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:15.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:15 smithi146 ceph-mon[103278]: pgmap v4869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:15 smithi078 ceph-mon[91136]: pgmap v4869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:15 smithi078 ceph-mon[99959]: pgmap v4869: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:17 smithi146 ceph-mon[103278]: pgmap v4870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:17 smithi078 ceph-mon[91136]: pgmap v4870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:17 smithi078 ceph-mon[99959]: pgmap v4870: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:18.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:18.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:19 smithi078 ceph-mon[91136]: pgmap v4871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:19 smithi078 ceph-mon[99959]: pgmap v4871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:19 smithi146 ceph-mon[103278]: pgmap v4871: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:02:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:02:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:20.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:20.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:20.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:21 smithi146 ceph-mon[103278]: pgmap v4872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:21 smithi078 ceph-mon[91136]: pgmap v4872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:21 smithi078 ceph-mon[99959]: pgmap v4872: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:23.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:23 smithi078 ceph-mon[99959]: pgmap v4873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:23.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:23 smithi078 ceph-mon[91136]: pgmap v4873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:23 smithi146 ceph-mon[103278]: pgmap v4873: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:24.670 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:24 smithi078 ceph-mon[91136]: pgmap v4874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:24.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:24 smithi078 ceph-mon[99959]: pgmap v4874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:24 smithi146 ceph-mon[103278]: pgmap v4874: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:27 smithi146 ceph-mon[103278]: pgmap v4875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:27 smithi078 ceph-mon[91136]: pgmap v4875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:27 smithi078 ceph-mon[99959]: pgmap v4875: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:28.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:28.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:28.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:29 smithi146 ceph-mon[103278]: pgmap v4876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[91136]: pgmap v4876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[99959]: pgmap v4876: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:02:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:02:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:30.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:30.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:30.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:31 smithi146 ceph-mon[103278]: pgmap v4877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:31 smithi078 ceph-mon[91136]: pgmap v4877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:31 smithi078 ceph-mon[99959]: pgmap v4877: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:33 smithi146 ceph-mon[103278]: pgmap v4878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[91136]: pgmap v4878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[99959]: pgmap v4878: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:33.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:34.273 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 2h - - 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 2h - - 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 6s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 1s ago 2h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 6s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 1s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 6s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:02:34.800 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 6s ago 2h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 1s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 6s ago 2h 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 6s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 1s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 6s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 6s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 6s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 6s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 1s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 1s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 1s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 1s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 1s ago 2h 78.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 6s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:02:34.801 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:02:34.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 6s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:02:34.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 1s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:02:35.110 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:02:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:02:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:02:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:35.110 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:02:35.111 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:02:35.377 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:02:35.377 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:02:35.378 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:02:35.379 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:35 smithi078 ceph-mon[99959]: pgmap v4879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:35.379 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:35 smithi078 ceph-mon[99959]: from='client.75014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:35.379 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2514260627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:35.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:35 smithi078 ceph-mon[91136]: pgmap v4879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:35.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:35 smithi078 ceph-mon[91136]: from='client.75014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:35.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2514260627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:35 smithi146 ceph-mon[103278]: pgmap v4879: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:35 smithi146 ceph-mon[103278]: from='client.75014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2514260627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:02:35.684 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:02:35.684 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:02:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:02:35.684 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:02:35.684 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:02:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:36 smithi146 ceph-mon[103278]: from='client.55101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:36 smithi146 ceph-mon[103278]: from='client.55107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3963771444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:36 smithi078 ceph-mon[91136]: from='client.55101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:36 smithi078 ceph-mon[91136]: from='client.55107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3963771444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:36 smithi078 ceph-mon[99959]: from='client.55101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:36 smithi078 ceph-mon[99959]: from='client.55107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3963771444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:02:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:37 smithi146 ceph-mon[103278]: from='client.75038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:37 smithi146 ceph-mon[103278]: pgmap v4880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:37 smithi078 ceph-mon[91136]: from='client.75038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:37 smithi078 ceph-mon[91136]: pgmap v4880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:37 smithi078 ceph-mon[99959]: from='client.75038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:02:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:37 smithi078 ceph-mon[99959]: pgmap v4880: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:39 smithi146 ceph-mon[103278]: pgmap v4881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:39 smithi078 ceph-mon[91136]: pgmap v4881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:39 smithi078 ceph-mon[99959]: pgmap v4881: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:02:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:02:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:40.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:40.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:40.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:41 smithi146 ceph-mon[103278]: pgmap v4882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:41 smithi078 ceph-mon[91136]: pgmap v4882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:41 smithi078 ceph-mon[99959]: pgmap v4882: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:43 smithi146 ceph-mon[103278]: pgmap v4883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:43 smithi078 ceph-mon[91136]: pgmap v4883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:43 smithi078 ceph-mon[99959]: pgmap v4883: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:45 smithi146 ceph-mon[103278]: pgmap v4884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:45 smithi078 ceph-mon[91136]: pgmap v4884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:45 smithi078 ceph-mon[99959]: pgmap v4884: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:47 smithi146 ceph-mon[103278]: pgmap v4885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:47 smithi078 ceph-mon[91136]: pgmap v4885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:47 smithi078 ceph-mon[99959]: pgmap v4885: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:49 smithi146 ceph-mon[103278]: pgmap v4886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[99959]: pgmap v4886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[91136]: pgmap v4886: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:49.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:02:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:02:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:50.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:50.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:02:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:02:50.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:02:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:51 smithi146 ceph-mon[103278]: pgmap v4887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:51 smithi078 ceph-mon[91136]: pgmap v4887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:51 smithi078 ceph-mon[99959]: pgmap v4887: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:53 smithi146 ceph-mon[103278]: pgmap v4888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[91136]: pgmap v4888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[99959]: pgmap v4888: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:02:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:55 smithi146 ceph-mon[103278]: pgmap v4889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:55 smithi078 ceph-mon[91136]: pgmap v4889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:55 smithi078 ceph-mon[99959]: pgmap v4889: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:57 smithi146 ceph-mon[103278]: pgmap v4890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:57 smithi078 ceph-mon[91136]: pgmap v4890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:57 smithi078 ceph-mon[99959]: pgmap v4890: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:02:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:02:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:59 smithi146 ceph-mon[103278]: pgmap v4891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:59.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:59 smithi078 ceph-mon[99959]: pgmap v4891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:59.526 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:02:59.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:59 smithi078 ceph-mon[91136]: pgmap v4891: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:02:59.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:03:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:03:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:00.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:00.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:00.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:01 smithi146 ceph-mon[103278]: pgmap v4892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:01 smithi078 ceph-mon[91136]: pgmap v4892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:01 smithi078 ceph-mon[99959]: pgmap v4892: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:03 smithi146 ceph-mon[103278]: pgmap v4893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:03 smithi078 ceph-mon[91136]: pgmap v4893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:03 smithi078 ceph-mon[99959]: pgmap v4893: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:05 smithi146 ceph-mon[103278]: pgmap v4894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:05 smithi078 ceph-mon[91136]: pgmap v4894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:05 smithi078 ceph-mon[99959]: pgmap v4894: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:05.980 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:03:06.525 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 2h - - 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 2h - - 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 17s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 13s ago 2h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 13s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 2h 309M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 13s ago 2h 198M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 17s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 13s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:03:06.526 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 13s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 13s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 13s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 13s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 13s ago 2h 78.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:03:06.527 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:03:06.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:03:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:03:06.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:03:06.849 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:03:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:03:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:03:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:03:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:03:07.142 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:03:07.462 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:03:07.462 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:07.462 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:07.462 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:07.462 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[99959]: from='client.75050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[99959]: pgmap v4895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[99959]: from='client.55137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1793173951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[91136]: from='client.75050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[91136]: pgmap v4895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[91136]: from='client.55137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:07.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1793173951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:07 smithi146 ceph-mon[103278]: from='client.75050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:07 smithi146 ceph-mon[103278]: pgmap v4895: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:07 smithi146 ceph-mon[103278]: from='client.55137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1793173951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:08 smithi146 ceph-mon[103278]: from='client.55143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:08 smithi146 ceph-mon[103278]: from='client.55155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:08.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3837078679' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[91136]: from='client.55143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[91136]: from='client.55155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3837078679' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[99959]: from='client.55143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[99959]: from='client.55155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3837078679' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:09 smithi146 ceph-mon[103278]: pgmap v4896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[91136]: pgmap v4896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[99959]: pgmap v4896: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:03:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:03:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:10.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:10.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:10.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:11 smithi078 ceph-mon[91136]: pgmap v4897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:11 smithi078 ceph-mon[99959]: pgmap v4897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:11 smithi146 ceph-mon[103278]: pgmap v4897: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:13 smithi078 ceph-mon[91136]: pgmap v4898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:13 smithi078 ceph-mon[99959]: pgmap v4898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:13 smithi146 ceph-mon[103278]: pgmap v4898: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:15 smithi078 ceph-mon[91136]: pgmap v4899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:15 smithi078 ceph-mon[99959]: pgmap v4899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:15.605 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:15 smithi146 ceph-mon[103278]: pgmap v4899: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:17 smithi078 ceph-mon[91136]: pgmap v4900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:17 smithi078 ceph-mon[99959]: pgmap v4900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:17 smithi146 ceph-mon[103278]: pgmap v4900: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:19 smithi078 ceph-mon[91136]: pgmap v4901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:19 smithi078 ceph-mon[99959]: pgmap v4901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:19 smithi146 ceph-mon[103278]: pgmap v4901: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:03:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:03:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:20.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:20.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:20.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:21 smithi146 ceph-mon[103278]: pgmap v4902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:21 smithi078 ceph-mon[91136]: pgmap v4902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:21 smithi078 ceph-mon[99959]: pgmap v4902: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:23 smithi146 ceph-mon[103278]: pgmap v4903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:23 smithi078 ceph-mon[99959]: pgmap v4903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:23 smithi078 ceph-mon[91136]: pgmap v4903: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:25.656 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:25 smithi078 ceph-mon[91136]: pgmap v4904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:25.657 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:25 smithi078 ceph-mon[99959]: pgmap v4904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:25 smithi146 ceph-mon[103278]: pgmap v4904: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:27 smithi078 ceph-mon[99959]: pgmap v4905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:27 smithi078 ceph-mon[91136]: pgmap v4905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:27.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:27 smithi146 ceph-mon[103278]: pgmap v4905: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:28.748 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:28.748 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:29 smithi146 ceph-mon[103278]: pgmap v4906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[99959]: pgmap v4906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[91136]: pgmap v4906: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:29.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:03:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:03:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:30.805Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:30.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:30.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:31 smithi146 ceph-mon[103278]: pgmap v4907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:31 smithi078 ceph-mon[91136]: pgmap v4907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:31 smithi078 ceph-mon[99959]: pgmap v4907: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:33 smithi146 ceph-mon[103278]: pgmap v4908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:33 smithi078 ceph-mon[91136]: pgmap v4908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:33 smithi078 ceph-mon[99959]: pgmap v4908: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:34 smithi146 ceph-mon[103278]: pgmap v4909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[91136]: pgmap v4909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:34 smithi078 ceph-mon[99959]: pgmap v4909: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:37.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:37 smithi146 ceph-mon[103278]: pgmap v4910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:37 smithi078 ceph-mon[91136]: pgmap v4910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:37 smithi078 ceph-mon[99959]: pgmap v4910: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:37.791 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 2h - - 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 2h - - 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 9s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 4s ago 2h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 9s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 4s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 9s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 9s ago 2h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 4s ago 2h 196M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:03:38.322 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 9s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 9s ago 2h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 4s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 9s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 9s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 9s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 9s ago 2h 79.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 4s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 4s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 4s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 4s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 4s ago 2h 79.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 9s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:03:38.323 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 9s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:03:38.324 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 4s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:03:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:38.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:03:38.634 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:03:38.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:03:38.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:03:38.635 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:03:38.903 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:03:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:03:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:03:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:03:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:03:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:03:38.904 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:03:39.214 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:03:39.214 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:03:39.214 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:03:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:39 smithi146 ceph-mon[103278]: from='client.55167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:39 smithi146 ceph-mon[103278]: from='client.55173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:39 smithi146 ceph-mon[103278]: pgmap v4911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:39 smithi146 ceph-mon[103278]: from='client.55179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1698865382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[91136]: from='client.55167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[91136]: from='client.55173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[91136]: pgmap v4911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[91136]: from='client.55179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1698865382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[99959]: from='client.55167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[99959]: from='client.55173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[99959]: pgmap v4911: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[99959]: from='client.55179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1698865382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:03:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:40 smithi146 ceph-mon[103278]: from='client.75110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2166364804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-mon[91136]: from='client.75110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2166364804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-mon[99959]: from='client.75110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:03:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2166364804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:03:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:03:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:03:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:40.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:40.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:40.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:41 smithi146 ceph-mon[103278]: pgmap v4912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:41 smithi078 ceph-mon[91136]: pgmap v4912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:41 smithi078 ceph-mon[99959]: pgmap v4912: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:43 smithi146 ceph-mon[103278]: pgmap v4913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:43 smithi078 ceph-mon[91136]: pgmap v4913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:43 smithi078 ceph-mon[99959]: pgmap v4913: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:45 smithi146 ceph-mon[103278]: pgmap v4914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:45 smithi078 ceph-mon[91136]: pgmap v4914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:45 smithi078 ceph-mon[99959]: pgmap v4914: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:47 smithi146 ceph-mon[103278]: pgmap v4915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:47 smithi078 ceph-mon[91136]: pgmap v4915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:47 smithi078 ceph-mon[99959]: pgmap v4915: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:48.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:48.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:49 smithi146 ceph-mon[103278]: pgmap v4916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[91136]: pgmap v4916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[99959]: pgmap v4916: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:49.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:49.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:03:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:03:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:50.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:50.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:03:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:03:50.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:03:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:51 smithi146 ceph-mon[103278]: pgmap v4917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:51 smithi078 ceph-mon[91136]: pgmap v4917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:51 smithi078 ceph-mon[99959]: pgmap v4917: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:53 smithi146 ceph-mon[103278]: pgmap v4918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:53 smithi078 ceph-mon[91136]: pgmap v4918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:53 smithi078 ceph-mon[99959]: pgmap v4918: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:54 smithi146 ceph-mon[103278]: pgmap v4919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[91136]: pgmap v4919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:03:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:54 smithi078 ceph-mon[99959]: pgmap v4919: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:57.492 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:57 smithi146 ceph-mon[103278]: pgmap v4920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:57 smithi078 ceph-mon[91136]: pgmap v4920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:57 smithi078 ceph-mon[99959]: pgmap v4920: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:03:58.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:03:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:59 smithi146 ceph-mon[103278]: pgmap v4921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:03:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:59 smithi078 ceph-mon[91136]: pgmap v4921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:03:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:03:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:59 smithi078 ceph-mon[99959]: pgmap v4921: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:03:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:03:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:04:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:04:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:00.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:00.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:00.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:01 smithi146 ceph-mon[103278]: pgmap v4922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:01 smithi078 ceph-mon[99959]: pgmap v4922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:01 smithi078 ceph-mon[91136]: pgmap v4922: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:03 smithi146 ceph-mon[103278]: pgmap v4923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:03 smithi078 ceph-mon[91136]: pgmap v4923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:03 smithi078 ceph-mon[99959]: pgmap v4923: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:05 smithi146 ceph-mon[103278]: pgmap v4924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:05 smithi078 ceph-mon[99959]: pgmap v4924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:05 smithi078 ceph-mon[91136]: pgmap v4924: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:07 smithi146 ceph-mon[103278]: pgmap v4925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:07 smithi078 ceph-mon[91136]: pgmap v4925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:07 smithi078 ceph-mon[99959]: pgmap v4925: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:08.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:08.297 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:09.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[91136]: pgmap v4926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[99959]: pgmap v4926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.507 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:04:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:09 smithi146 ceph-mon[103278]: pgmap v4926: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:10.041 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 2h - - 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 2h - - 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 0s ago 2h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 16s ago 2h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 0s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 16s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 0s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 0s ago 2h 310M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 16s ago 2h 197M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 0s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:04:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 0s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 16s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 0s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 0s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 0s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 0s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 16s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 16s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 16s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 16s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 16s ago 2h 79.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 0s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 16s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 0s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:04:10.043 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 16s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:04:10.368 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:04:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:04:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:04:10.369 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:04:10.648 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:04:10.648 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:04:10.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:04:10.650 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:04:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:04:10.971 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:04:10.972 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:10.972 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:10.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:10.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:10.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:10.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:10.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:10.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:11 smithi146 ceph-mon[103278]: from='client.55203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:11 smithi146 ceph-mon[103278]: from='client.75128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:11 smithi146 ceph-mon[103278]: from='client.55215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:11 smithi146 ceph-mon[103278]: pgmap v4927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3179057681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/662785770' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[91136]: from='client.55203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[91136]: from='client.75128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[91136]: from='client.55215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[91136]: pgmap v4927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3179057681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/662785770' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[99959]: from='client.55203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[99959]: from='client.75128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[99959]: from='client.55215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[99959]: pgmap v4927: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3179057681' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:11.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/662785770' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:12 smithi146 ceph-mon[103278]: from='client.75146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:12 smithi078 ceph-mon[99959]: from='client.75146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:12 smithi078 ceph-mon[91136]: from='client.75146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:13 smithi146 ceph-mon[103278]: pgmap v4928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:13 smithi078 ceph-mon[99959]: pgmap v4928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:13 smithi078 ceph-mon[91136]: pgmap v4928: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:15 smithi146 ceph-mon[103278]: pgmap v4929: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:15 smithi078 ceph-mon[91136]: pgmap v4929: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:15 smithi078 ceph-mon[99959]: pgmap v4929: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:17.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:17 smithi146 ceph-mon[103278]: pgmap v4930: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:17 smithi078 ceph-mon[91136]: pgmap v4930: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:17 smithi078 ceph-mon[99959]: pgmap v4930: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:18.309 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:19 smithi146 ceph-mon[103278]: pgmap v4931: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:19 smithi078 ceph-mon[91136]: pgmap v4931: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:19 smithi078 ceph-mon[99959]: pgmap v4931: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:04:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:04:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:20.809Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:20.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:20.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:21 smithi146 ceph-mon[103278]: pgmap v4932: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:21 smithi078 ceph-mon[91136]: pgmap v4932: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:21 smithi078 ceph-mon[99959]: pgmap v4932: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:23 smithi078 ceph-mon[91136]: pgmap v4933: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:23 smithi078 ceph-mon[99959]: pgmap v4933: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:23 smithi146 ceph-mon[103278]: pgmap v4933: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:25 smithi078 ceph-mon[91136]: pgmap v4934: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:25 smithi078 ceph-mon[99959]: pgmap v4934: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:25 smithi146 ceph-mon[103278]: pgmap v4934: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 23:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T23:04:25.847195559Z level=info msg="Completed cleanup jobs" duration=11.971458ms 2024-09-20T23:04:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:27 smithi078 ceph-mon[91136]: pgmap v4935: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:27 smithi078 ceph-mon[99959]: pgmap v4935: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:27 smithi146 ceph-mon[103278]: pgmap v4935: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[91136]: pgmap v4936: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[99959]: pgmap v4936: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:29 smithi146 ceph-mon[103278]: pgmap v4936: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:04:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:04:31.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:30.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:31.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:30.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:31.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:30.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:31 smithi078 ceph-mon[91136]: pgmap v4937: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:31 smithi078 ceph-mon[99959]: pgmap v4937: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:31 smithi146 ceph-mon[103278]: pgmap v4937: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:33 smithi078 ceph-mon[91136]: pgmap v4938: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:33 smithi078 ceph-mon[99959]: pgmap v4938: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:33 smithi146 ceph-mon[103278]: pgmap v4938: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:34 smithi146 ceph-mon[103278]: pgmap v4939: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[91136]: pgmap v4939: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:34 smithi078 ceph-mon[99959]: pgmap v4939: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:37 smithi146 ceph-mon[103278]: pgmap v4940: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:37 smithi078 ceph-mon[91136]: pgmap v4940: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:37 smithi078 ceph-mon[99959]: pgmap v4940: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:39 smithi146 ceph-mon[103278]: pgmap v4941: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:39 smithi078 ceph-mon[99959]: pgmap v4941: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:39 smithi078 ceph-mon[91136]: pgmap v4941: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:04:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:04:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:40.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:40.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:40.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:41.268 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:04:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:41 smithi146 ceph-mon[103278]: pgmap v4942: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:41 smithi078 ceph-mon[99959]: pgmap v4942: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:41 smithi078 ceph-mon[91136]: pgmap v4942: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:41.801 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:04:41.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 2h - - 2024-09-20T23:04:41.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 2h - - 2024-09-20T23:04:41.801 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 12s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 7s ago 2h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 12s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 7s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 12s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 12s ago 2h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 7s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 12s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:04:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 12s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 7s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 12s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 12s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 12s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 12s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 7s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:04:41.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 7s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 7s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 7s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 7s ago 2h 79.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 12s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 7s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 12s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:04:41.804 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 7s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:04:42.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:04:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:04:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:04:42.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:04:42.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:04:42.377 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:04:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:04:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:04:42.378 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:04:42.378 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:04:42.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:42 smithi078 ceph-mon[91136]: from='client.75158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:42.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2299127678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:42.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:42 smithi078 ceph-mon[99959]: from='client.75158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:42.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2299127678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:42 smithi146 ceph-mon[103278]: from='client.75158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2299127678' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:04:42.690 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:04:42.690 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:42.690 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:04:42.690 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:42.690 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:04:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:43 smithi146 ceph-mon[103278]: from='client.55245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:43 smithi146 ceph-mon[103278]: from='client.75170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:43 smithi146 ceph-mon[103278]: pgmap v4943: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3839999960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[91136]: from='client.55245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[91136]: from='client.75170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[91136]: pgmap v4943: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3839999960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[99959]: from='client.55245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[99959]: from='client.75170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[99959]: pgmap v4943: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3839999960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:04:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:44 smithi146 ceph-mon[103278]: from='client.75182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:44 smithi078 ceph-mon[99959]: from='client.75182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:44 smithi078 ceph-mon[91136]: from='client.75182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:04:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:45 smithi146 ceph-mon[103278]: pgmap v4944: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:45 smithi078 ceph-mon[91136]: pgmap v4944: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:45 smithi078 ceph-mon[99959]: pgmap v4944: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:47 smithi146 ceph-mon[103278]: pgmap v4945: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:47 smithi078 ceph-mon[91136]: pgmap v4945: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:47 smithi078 ceph-mon[99959]: pgmap v4945: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:49 smithi078 ceph-mon[99959]: pgmap v4946: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:49 smithi078 ceph-mon[91136]: pgmap v4946: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:49 smithi146 ceph-mon[103278]: pgmap v4946: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[91136]: pgmap v4947: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-mon[99959]: pgmap v4947: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:50 smithi146 ceph-mon[103278]: pgmap v4947: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:04:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:04:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:50.812Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:50.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:04:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:04:50.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:04:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:53 smithi146 ceph-mon[103278]: pgmap v4948: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:53 smithi078 ceph-mon[91136]: pgmap v4948: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:53 smithi078 ceph-mon[99959]: pgmap v4948: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:55 smithi146 ceph-mon[103278]: pgmap v4949: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[91136]: pgmap v4949: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:04:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:55 smithi078 ceph-mon[99959]: pgmap v4949: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:57 smithi146 ceph-mon[103278]: pgmap v4950: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:57 smithi078 ceph-mon[99959]: pgmap v4950: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:57 smithi078 ceph-mon[91136]: pgmap v4950: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:04:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:59 smithi146 ceph-mon[103278]: pgmap v4951: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:04:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:59 smithi078 ceph-mon[91136]: pgmap v4951: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:04:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:04:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:04:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:59 smithi078 ceph-mon[99959]: pgmap v4951: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:04:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:04:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:05:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:05:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:00.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:00.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:00.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:01 smithi146 ceph-mon[103278]: pgmap v4952: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:01 smithi078 ceph-mon[91136]: pgmap v4952: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:01 smithi078 ceph-mon[99959]: pgmap v4952: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:03 smithi146 ceph-mon[103278]: pgmap v4953: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:03 smithi078 ceph-mon[91136]: pgmap v4953: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:03 smithi078 ceph-mon[99959]: pgmap v4953: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:05 smithi146 ceph-mon[103278]: pgmap v4954: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:05 smithi078 ceph-mon[99959]: pgmap v4954: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:05 smithi078 ceph-mon[91136]: pgmap v4954: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:07 smithi146 ceph-mon[103278]: pgmap v4955: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:07 smithi078 ceph-mon[91136]: pgmap v4955: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:07 smithi078 ceph-mon[99959]: pgmap v4955: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:09.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:09.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:09 smithi078 ceph-mon[99959]: pgmap v4956: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:09.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:09.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:09 smithi078 ceph-mon[91136]: pgmap v4956: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:09 smithi146 ceph-mon[103278]: pgmap v4956: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[91136]: pgmap v4957: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-mon[99959]: pgmap v4957: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:10.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:10.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:10 smithi146 ceph-mon[103278]: pgmap v4957: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:10.852 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:05:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:05:10.853 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:10.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:10.853 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:10.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:10.853 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:10.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:12.987 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:05:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:13 smithi078 ceph-mon[99959]: pgmap v4958: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:13 smithi078 ceph-mon[91136]: pgmap v4958: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 2h - - 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 2h - - 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 4s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 19s ago 2h 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 4s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 19s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:05:13.516 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 4s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 4s ago 2h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 19s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 4s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 4s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 19s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 4s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 4s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 4s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 4s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 19s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 19s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 19s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:05:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 19s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:05:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 19s ago 2h 79.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:05:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 4s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:05:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:05:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 4s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:05:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 19s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:05:13.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:13 smithi146 ceph-mon[103278]: pgmap v4958: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:13.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:05:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:05:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:05:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:05:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:05:13.826 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:05:14.093 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:05:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:05:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:05:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:05:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:05:14.094 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:05:14.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:05:14.405 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:05:14.405 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:14.405 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:14.405 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:14.405 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='client.75194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='client.75200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1791639267' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='client.75194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='client.75200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:14.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1791639267' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:14.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='client.75194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='client.75200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1791639267' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:15 smithi146 ceph-mon[103278]: from='client.55287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:15 smithi146 ceph-mon[103278]: from='client.55299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:15 smithi146 ceph-mon[103278]: pgmap v4959: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1641633206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[91136]: from='client.55287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[91136]: from='client.55299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[91136]: pgmap v4959: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1641633206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[99959]: from='client.55287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[99959]: from='client.55299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[99959]: pgmap v4959: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1641633206' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:17 smithi146 ceph-mon[103278]: pgmap v4960: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:17 smithi078 ceph-mon[91136]: pgmap v4960: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:17 smithi078 ceph-mon[99959]: pgmap v4960: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:19 smithi146 ceph-mon[103278]: pgmap v4961: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:19 smithi078 ceph-mon[99959]: pgmap v4961: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:19 smithi078 ceph-mon[91136]: pgmap v4961: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:05:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:05:21.238 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:20.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:21.239 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:20.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:21.239 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:20.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:21 smithi146 ceph-mon[103278]: pgmap v4962: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:21 smithi078 ceph-mon[99959]: pgmap v4962: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:21 smithi078 ceph-mon[91136]: pgmap v4962: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:23 smithi146 ceph-mon[103278]: pgmap v4963: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:23 smithi078 ceph-mon[99959]: pgmap v4963: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:23 smithi078 ceph-mon[91136]: pgmap v4963: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:25 smithi146 ceph-mon[103278]: pgmap v4964: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:25 smithi078 ceph-mon[99959]: pgmap v4964: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:25 smithi078 ceph-mon[91136]: pgmap v4964: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:27 smithi146 ceph-mon[103278]: pgmap v4965: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:27 smithi078 ceph-mon[91136]: pgmap v4965: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:27 smithi078 ceph-mon[99959]: pgmap v4965: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:29 smithi146 ceph-mon[103278]: pgmap v4966: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:29 smithi078 ceph-mon[99959]: pgmap v4966: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:29 smithi078 ceph-mon[91136]: pgmap v4966: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:05:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:05:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[91136]: pgmap v4967: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-mon[99959]: pgmap v4967: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:30 smithi146 ceph-mon[103278]: pgmap v4967: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:31.085 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:30.815Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:31.085 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:30.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:31.085 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:30.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:33 smithi146 ceph-mon[103278]: pgmap v4968: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:33 smithi078 ceph-mon[91136]: pgmap v4968: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:33 smithi078 ceph-mon[99959]: pgmap v4968: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[91136]: pgmap v4969: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[99959]: pgmap v4969: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:35 smithi146 ceph-mon[103278]: pgmap v4969: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:37 smithi078 ceph-mon[91136]: pgmap v4970: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:37 smithi078 ceph-mon[99959]: pgmap v4970: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:37 smithi146 ceph-mon[103278]: pgmap v4970: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:39 smithi146 ceph-mon[103278]: pgmap v4971: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:39 smithi078 ceph-mon[91136]: pgmap v4971: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:39 smithi078 ceph-mon[99959]: pgmap v4971: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:05:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:05:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:40.815Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:40.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:40.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:41 smithi078 ceph-mon[91136]: pgmap v4972: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:41 smithi078 ceph-mon[99959]: pgmap v4972: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:41.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:41 smithi146 ceph-mon[103278]: pgmap v4972: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:43 smithi078 ceph-mon[91136]: pgmap v4973: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:43 smithi078 ceph-mon[99959]: pgmap v4973: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:43 smithi146 ceph-mon[103278]: pgmap v4973: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:44.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:44.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:44.700 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:05:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:45.225 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 2h - - 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 2h - - 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 15s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 10s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 15s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 10s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 15s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 15s ago 2h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 10s ago 2h 199M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 15s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:05:45.226 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 15s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 10s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 15s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 15s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 15s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 15s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 10s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 10s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 10s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 10s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 10s ago 2h 80.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 15s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 10s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 15s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:05:45.227 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 10s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:05:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:45 smithi078 ceph-mon[99959]: pgmap v4974: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:45 smithi078 ceph-mon[91136]: pgmap v4974: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:45.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:05:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:05:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:05:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:05:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:05:45.538 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:05:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:45 smithi146 ceph-mon[103278]: pgmap v4974: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:05:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:05:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:05:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:05:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:05:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:05:45.806 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:05:46.112 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:05:46.113 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:05:46.113 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[91136]: from='client.55311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[91136]: from='client.75236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[91136]: from='client.75242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/301869034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/480314343' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[99959]: from='client.55311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[99959]: from='client.75236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[99959]: from='client.75242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/301869034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:46.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/480314343' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:46 smithi146 ceph-mon[103278]: from='client.55311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:46 smithi146 ceph-mon[103278]: from='client.75236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:46 smithi146 ceph-mon[103278]: from='client.75242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/301869034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:05:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/480314343' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:05:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:47 smithi146 ceph-mon[103278]: from='client.75254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:47 smithi146 ceph-mon[103278]: pgmap v4975: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:47 smithi078 ceph-mon[91136]: from='client.75254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:47 smithi078 ceph-mon[91136]: pgmap v4975: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:47 smithi078 ceph-mon[99959]: from='client.75254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:05:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:47 smithi078 ceph-mon[99959]: pgmap v4975: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:49.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:49.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:49 smithi078 ceph-mon[91136]: pgmap v4976: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:49.588 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:49.589 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:49 smithi078 ceph-mon[99959]: pgmap v4976: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:49 smithi146 ceph-mon[103278]: pgmap v4976: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:50.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:50.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:50.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:50.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[91136]: pgmap v4977: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:50.927 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:05:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:05:50.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:50.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:50.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:50.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-mon[99959]: pgmap v4977: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:50.928 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:50.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:50.928 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:50.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:05:50.928 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:05:50.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:05:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:50 smithi146 ceph-mon[103278]: pgmap v4977: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:53 smithi146 ceph-mon[103278]: pgmap v4978: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:53 smithi078 ceph-mon[91136]: pgmap v4978: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:53 smithi078 ceph-mon[99959]: pgmap v4978: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:55 smithi146 ceph-mon[103278]: pgmap v4979: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[99959]: pgmap v4979: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[91136]: pgmap v4979: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:05:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:57 smithi146 ceph-mon[103278]: pgmap v4980: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:57 smithi078 ceph-mon[99959]: pgmap v4980: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:57 smithi078 ceph-mon[91136]: pgmap v4980: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:05:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:59 smithi146 ceph-mon[103278]: pgmap v4981: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:05:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:59 smithi078 ceph-mon[91136]: pgmap v4981: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:05:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:05:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:05:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:59 smithi078 ceph-mon[99959]: pgmap v4981: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:05:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:05:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:06:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:06:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:00.817Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:00.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:00.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:01.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:01 smithi146 ceph-mon[103278]: pgmap v4982: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:01 smithi078 ceph-mon[91136]: pgmap v4982: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:01 smithi078 ceph-mon[99959]: pgmap v4982: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:03 smithi146 ceph-mon[103278]: pgmap v4983: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:03 smithi078 ceph-mon[91136]: pgmap v4983: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:03 smithi078 ceph-mon[99959]: pgmap v4983: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:05 smithi146 ceph-mon[103278]: pgmap v4984: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:05 smithi078 ceph-mon[91136]: pgmap v4984: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:05 smithi078 ceph-mon[99959]: pgmap v4984: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:07 smithi078 ceph-mon[91136]: pgmap v4985: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:07 smithi078 ceph-mon[99959]: pgmap v4985: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:07 smithi146 ceph-mon[103278]: pgmap v4985: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:09 smithi078 ceph-mon[91136]: pgmap v4986: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:09 smithi078 ceph-mon[99959]: pgmap v4986: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:09 smithi146 ceph-mon[103278]: pgmap v4986: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:10.875 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:06:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:06:10.875 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:10.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:10.876 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:10.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:10.876 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:10.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[91136]: pgmap v4987: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:10 smithi078 ceph-mon[99959]: pgmap v4987: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:10 smithi146 ceph-mon[103278]: pgmap v4987: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:13 smithi146 ceph-mon[103278]: pgmap v4988: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:13 smithi078 ceph-mon[91136]: pgmap v4988: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:13 smithi078 ceph-mon[99959]: pgmap v4988: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:15 smithi146 ceph-mon[103278]: pgmap v4989: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[99959]: pgmap v4989: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[91136]: pgmap v4989: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:15.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:15.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:16.408 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 2h - - 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 2h - - 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 2s ago 2h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 7s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 2s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:06:16.931 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 7s ago 2h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 2s ago 2h 200M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 2h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 2s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 2s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 2s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 2s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 2s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 2s ago 2h 80.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:06:16.932 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:06:16.933 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:06:16.933 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:06:16.933 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 2s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:06:17.236 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:06:17.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:06:17.236 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:06:17.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:17.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:06:17.236 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:06:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:06:17.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:06:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:06:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:06:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:06:17.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:06:17.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:17 smithi078 ceph-mon[91136]: pgmap v4990: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:17.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1965963236' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:17.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:17 smithi078 ceph-mon[99959]: pgmap v4990: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:17.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1965963236' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:17 smithi146 ceph-mon[103278]: pgmap v4990: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1965963236' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:17.815 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:06:17.815 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:17.816 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:17.816 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:17.816 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:18 smithi146 ceph-mon[103278]: from='client.75266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:18 smithi146 ceph-mon[103278]: from='client.55353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:18 smithi146 ceph-mon[103278]: from='client.75278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3917627177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[91136]: from='client.75266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[91136]: from='client.55353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[91136]: from='client.75278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3917627177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[99959]: from='client.75266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[99959]: from='client.55353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[99959]: from='client.75278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3917627177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:19 smithi146 ceph-mon[103278]: from='client.75290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:19 smithi146 ceph-mon[103278]: pgmap v4991: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:19 smithi078 ceph-mon[91136]: from='client.75290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:19 smithi078 ceph-mon[91136]: pgmap v4991: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:19 smithi078 ceph-mon[99959]: from='client.75290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:19 smithi078 ceph-mon[99959]: pgmap v4991: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:06:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:06:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:20.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:20.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:20.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:21.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:21 smithi146 ceph-mon[103278]: pgmap v4992: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:21 smithi078 ceph-mon[91136]: pgmap v4992: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:21 smithi078 ceph-mon[99959]: pgmap v4992: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:23 smithi146 ceph-mon[103278]: pgmap v4993: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:23 smithi078 ceph-mon[91136]: pgmap v4993: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:23 smithi078 ceph-mon[99959]: pgmap v4993: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:25 smithi078 ceph-mon[99959]: pgmap v4994: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:25 smithi078 ceph-mon[91136]: pgmap v4994: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:25 smithi146 ceph-mon[103278]: pgmap v4994: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:27 smithi078 ceph-mon[99959]: pgmap v4995: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:27 smithi078 ceph-mon[91136]: pgmap v4995: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:27.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:27 smithi146 ceph-mon[103278]: pgmap v4995: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:29 smithi078 ceph-mon[91136]: pgmap v4996: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:29 smithi078 ceph-mon[99959]: pgmap v4996: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:29 smithi146 ceph-mon[103278]: pgmap v4996: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:31.009 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:06:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:06:31.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:30.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:31.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:30.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:31.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:30.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:31 smithi146 ceph-mon[103278]: pgmap v4997: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[91136]: pgmap v4997: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:31 smithi078 ceph-mon[99959]: pgmap v4997: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:33.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:33 smithi078 ceph-mon[91136]: pgmap v4998: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:33.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:33 smithi078 ceph-mon[99959]: pgmap v4998: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:33 smithi146 ceph-mon[103278]: pgmap v4998: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:35 smithi146 ceph-mon[103278]: pgmap v4999: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[91136]: pgmap v4999: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[99959]: pgmap v4999: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:37 smithi146 ceph-mon[103278]: pgmap v5000: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:37.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:37 smithi078 ceph-mon[91136]: pgmap v5000: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:37.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:37 smithi078 ceph-mon[99959]: pgmap v5000: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:39 smithi146 ceph-mon[103278]: pgmap v5001: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:39 smithi078 ceph-mon[91136]: pgmap v5001: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:39 smithi078 ceph-mon[99959]: pgmap v5001: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:06:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:06:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:40.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:40.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:40.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:41.357 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:41 smithi146 ceph-mon[103278]: pgmap v5002: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:41 smithi078 ceph-mon[91136]: pgmap v5002: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:41 smithi078 ceph-mon[99959]: pgmap v5002: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:43 smithi078 ceph-mon[91136]: pgmap v5003: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:43 smithi078 ceph-mon[99959]: pgmap v5003: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:43 smithi146 ceph-mon[103278]: pgmap v5003: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:45 smithi078 ceph-mon[91136]: pgmap v5004: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:45 smithi078 ceph-mon[99959]: pgmap v5004: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:45 smithi146 ceph-mon[103278]: pgmap v5004: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:47 smithi078 ceph-mon[99959]: pgmap v5005: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:47 smithi078 ceph-mon[91136]: pgmap v5005: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:47 smithi146 ceph-mon[103278]: pgmap v5005: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:48.115 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 2h - - 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 2h - - 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 18s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 13s ago 2h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 18s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 13s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 18s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 18s ago 2h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 13s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 18s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 18s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:06:48.645 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 13s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 18s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 18s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 18s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 18s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 13s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 13s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 13s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 13s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 13s ago 2h 80.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 18s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:06:48.646 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 18s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:06:48.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 13s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:06:48.954 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:06:48.954 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:06:48.954 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:06:48.955 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:06:49.223 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:06:49.384 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:06:49.385 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:06:49.385 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:06:49.385 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:06:49.385 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:06:49.385 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:06:49.385 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:06:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[91136]: from='client.55383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[91136]: pgmap v5006: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/668989861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[99959]: from='client.55383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[99959]: pgmap v5006: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/668989861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:49.537 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:06:49.537 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:49.537 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:06:49.537 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:49.537 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:06:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:49 smithi146 ceph-mon[103278]: from='client.55383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:49 smithi146 ceph-mon[103278]: pgmap v5006: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/668989861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='client.75308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='client.55395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='client.55407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2984494505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='client.75308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='client.55395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='client.55407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2984494505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:50.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='client.75308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='client.55395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='client.55407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:06:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2984494505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:06:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:06:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:06:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:50.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:50.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:06:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:06:50.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:06:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:51 smithi146 ceph-mon[103278]: pgmap v5007: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:51 smithi078 ceph-mon[99959]: pgmap v5007: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:51 smithi078 ceph-mon[91136]: pgmap v5007: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:53 smithi078 ceph-mon[91136]: pgmap v5008: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:53 smithi078 ceph-mon[99959]: pgmap v5008: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:53 smithi146 ceph-mon[103278]: pgmap v5008: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[91136]: pgmap v5009: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[99959]: pgmap v5009: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:55 smithi146 ceph-mon[103278]: pgmap v5009: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:06:57.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:57 smithi146 ceph-mon[103278]: pgmap v5010: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:57 smithi078 ceph-mon[91136]: pgmap v5010: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:57 smithi078 ceph-mon[99959]: pgmap v5010: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:06:58.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:06:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:06:59 smithi146 ceph-mon[103278]: pgmap v5011: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:06:59 smithi078 ceph-mon[99959]: pgmap v5011: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:06:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:06:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:06:59 smithi078 ceph-mon[91136]: pgmap v5011: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:07:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:07:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:00.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:00.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:00.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:01 smithi146 ceph-mon[103278]: pgmap v5012: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:01 smithi078 ceph-mon[91136]: pgmap v5012: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:01 smithi078 ceph-mon[99959]: pgmap v5012: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:02 smithi146 ceph-mon[103278]: pgmap v5013: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:02 smithi078 ceph-mon[99959]: pgmap v5013: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:02 smithi078 ceph-mon[91136]: pgmap v5013: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:05 smithi146 ceph-mon[103278]: pgmap v5014: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:05 smithi078 ceph-mon[91136]: pgmap v5014: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:05 smithi078 ceph-mon[99959]: pgmap v5014: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:07 smithi078 ceph-mon[99959]: pgmap v5015: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:07 smithi078 ceph-mon[91136]: pgmap v5015: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:07 smithi146 ceph-mon[103278]: pgmap v5015: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:09 smithi078 ceph-mon[99959]: pgmap v5016: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:09 smithi078 ceph-mon[91136]: pgmap v5016: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:09 smithi146 ceph-mon[103278]: pgmap v5016: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:07:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:07:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:10.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:10.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:10.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:11 smithi146 ceph-mon[103278]: pgmap v5017: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[91136]: pgmap v5017: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[99959]: pgmap v5017: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:13 smithi146 ceph-mon[103278]: pgmap v5018: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:13 smithi078 ceph-mon[91136]: pgmap v5018: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:13 smithi078 ceph-mon[99959]: pgmap v5018: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:07:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[91136]: pgmap v5019: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[99959]: pgmap v5019: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:15 smithi146 ceph-mon[103278]: pgmap v5019: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:17 smithi078 ceph-mon[91136]: pgmap v5020: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:17 smithi078 ceph-mon[99959]: pgmap v5020: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:17 smithi146 ceph-mon[103278]: pgmap v5020: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:19 smithi078 ceph-mon[91136]: pgmap v5021: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:19 smithi078 ceph-mon[99959]: pgmap v5021: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:19 smithi146 ceph-mon[103278]: pgmap v5021: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:19.832 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 2h - - 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 2h - - 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 10s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 5s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 10s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:07:20.362 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 5s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 10s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 10s ago 2h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 5s ago 2h 201M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 10s ago 2h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 10s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 5s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 10s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 10s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 10s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 10s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 5s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 5s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 5s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:07:20.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 5s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:07:20.364 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 5s ago 2h 80.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:07:20.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 10s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:07:20.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:07:20.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 10s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:07:20.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 5s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:07:20.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:07:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:07:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:07:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:07:20.699 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:07:20.979 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:07:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:07:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:07:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:07:20.980 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:07:20.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:07:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T23:07:20.981 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:20.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:20.981 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:20.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:20.981 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:20.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:21.312 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:07:21.312 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:21.312 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[91136]: from='client.75338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[91136]: from='client.75344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[91136]: pgmap v5022: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[91136]: from='client.75350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3365416679' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[99959]: from='client.75338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[99959]: from='client.75344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[99959]: pgmap v5022: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[99959]: from='client.75350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3365416679' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:21.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:21 smithi146 ceph-mon[103278]: from='client.75338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:21 smithi146 ceph-mon[103278]: from='client.75344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:21 smithi146 ceph-mon[103278]: pgmap v5022: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:21.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:21 smithi146 ceph-mon[103278]: from='client.75350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:21.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3365416679' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:22.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:22 smithi146 ceph-mon[103278]: from='client.55443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:22.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/163551155' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:22 smithi078 ceph-mon[91136]: from='client.55443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/163551155' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:22 smithi078 ceph-mon[99959]: from='client.55443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/163551155' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:23 smithi146 ceph-mon[103278]: pgmap v5023: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:23 smithi078 ceph-mon[91136]: pgmap v5023: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:23 smithi078 ceph-mon[99959]: pgmap v5023: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:25 smithi146 ceph-mon[103278]: pgmap v5024: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:25 smithi078 ceph-mon[91136]: pgmap v5024: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:25 smithi078 ceph-mon[99959]: pgmap v5024: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:27.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:27 smithi146 ceph-mon[103278]: pgmap v5025: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:27 smithi078 ceph-mon[91136]: pgmap v5025: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:27 smithi078 ceph-mon[99959]: pgmap v5025: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:29 smithi146 ceph-mon[103278]: pgmap v5026: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:29 smithi078 ceph-mon[91136]: pgmap v5026: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:29 smithi078 ceph-mon[99959]: pgmap v5026: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:07:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:07:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:30.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:30.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:30.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:31.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[91136]: pgmap v5027: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[99959]: pgmap v5027: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.636 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:31 smithi146 ceph-mon[103278]: pgmap v5027: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:33 smithi146 ceph-mon[103278]: pgmap v5028: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:33 smithi078 ceph-mon[91136]: pgmap v5028: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:33 smithi078 ceph-mon[99959]: pgmap v5028: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[91136]: pgmap v5029: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[99959]: pgmap v5029: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:35 smithi146 ceph-mon[103278]: pgmap v5029: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:37.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:37 smithi078 ceph-mon[91136]: pgmap v5030: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:37.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:37 smithi078 ceph-mon[99959]: pgmap v5030: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:37 smithi146 ceph-mon[103278]: pgmap v5030: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:39 smithi146 ceph-mon[103278]: pgmap v5031: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:39 smithi078 ceph-mon[91136]: pgmap v5031: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:39 smithi078 ceph-mon[99959]: pgmap v5031: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:07:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:07:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:40.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:40.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:40.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:41 smithi146 ceph-mon[103278]: pgmap v5032: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:41 smithi078 ceph-mon[91136]: pgmap v5032: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:41 smithi078 ceph-mon[99959]: pgmap v5032: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:43.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:43 smithi146 ceph-mon[103278]: pgmap v5033: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:43.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:43 smithi078 ceph-mon[91136]: pgmap v5033: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:43 smithi078 ceph-mon[99959]: pgmap v5033: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:45 smithi146 ceph-mon[103278]: pgmap v5034: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:45 smithi078 ceph-mon[99959]: pgmap v5034: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:45 smithi078 ceph-mon[91136]: pgmap v5034: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:47.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:47 smithi078 ceph-mon[91136]: pgmap v5035: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:47.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:47 smithi078 ceph-mon[99959]: pgmap v5035: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:47 smithi146 ceph-mon[103278]: pgmap v5035: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:49.588 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:49.589 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:49 smithi078 ceph-mon[99959]: pgmap v5036: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:49.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:49.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:49 smithi078 ceph-mon[91136]: pgmap v5036: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:49 smithi146 ceph-mon[103278]: pgmap v5036: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:50.821 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:07:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:07:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:50.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:50.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:07:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:07:50.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:07:51.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[99959]: pgmap v5037: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[91136]: pgmap v5037: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.605 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.605 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:07:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:51 smithi146 ceph-mon[103278]: pgmap v5037: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:07:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 2h - - 2024-09-20T23:07:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 2h - - 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 1s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 17s ago 2h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 1s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 17s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 1s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 1s ago 2h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 17s ago 2h 202M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 1s ago 2h 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 1s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 17s ago 2h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 1s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:07:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 1s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 1s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 1s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 17s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 17s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 17s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 17s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 17s ago 2h 80.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 1s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 1s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:07:52.156 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:07:52.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:07:52.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:07:52.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:07:52.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:52.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:07:52.462 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:07:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:07:52.729 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:07:53.038 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:07:53.039 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:07:53.039 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:07:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:53 smithi146 ceph-mon[103278]: from='client.75374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:53 smithi146 ceph-mon[103278]: from='client.75380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:53 smithi146 ceph-mon[103278]: from='client.55467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:53 smithi146 ceph-mon[103278]: pgmap v5038: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2197655072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1069075814' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[91136]: from='client.75374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[91136]: from='client.75380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[91136]: from='client.55467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[91136]: pgmap v5038: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2197655072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1069075814' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[99959]: from='client.75374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[99959]: from='client.75380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[99959]: from='client.55467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[99959]: pgmap v5038: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2197655072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:07:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1069075814' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:07:54.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:54 smithi078 ceph-mon[99959]: from='client.55479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:54.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:54 smithi078 ceph-mon[91136]: from='client.55479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:54 smithi146 ceph-mon[103278]: from='client.55479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:07:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:55 smithi146 ceph-mon[103278]: pgmap v5039: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[91136]: pgmap v5039: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[99959]: pgmap v5039: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:07:57.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:57 smithi078 ceph-mon[91136]: pgmap v5040: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:57.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:57 smithi078 ceph-mon[99959]: pgmap v5040: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:57.662 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:57 smithi146 ceph-mon[103278]: pgmap v5040: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:07:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:07:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:07:59 smithi146 ceph-mon[103278]: pgmap v5041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:07:59 smithi078 ceph-mon[91136]: pgmap v5041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:07:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:07:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:07:59 smithi078 ceph-mon[99959]: pgmap v5041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:08:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:08:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:00.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:00.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:00.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:01 smithi078 ceph-mon[91136]: pgmap v5042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:01 smithi078 ceph-mon[99959]: pgmap v5042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:01 smithi146 ceph-mon[103278]: pgmap v5042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:03.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:03 smithi146 ceph-mon[103278]: pgmap v5043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:03 smithi078 ceph-mon[91136]: pgmap v5043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:03 smithi078 ceph-mon[99959]: pgmap v5043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:05 smithi146 ceph-mon[103278]: pgmap v5044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:05 smithi078 ceph-mon[91136]: pgmap v5044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:05 smithi078 ceph-mon[99959]: pgmap v5044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:07 smithi146 ceph-mon[103278]: pgmap v5045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:07 smithi078 ceph-mon[99959]: pgmap v5045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:07 smithi078 ceph-mon[91136]: pgmap v5045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:08 smithi146 ceph-mon[103278]: pgmap v5046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:08 smithi078 ceph-mon[99959]: pgmap v5046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:08 smithi078 ceph-mon[91136]: pgmap v5046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:08:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:08:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:10.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:10.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:10.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:11 smithi146 ceph-mon[103278]: pgmap v5047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[99959]: pgmap v5047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[91136]: pgmap v5047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:13 smithi146 ceph-mon[103278]: pgmap v5048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:13 smithi078 ceph-mon[91136]: pgmap v5048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:13 smithi078 ceph-mon[99959]: pgmap v5048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:15.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:15 smithi078 ceph-mon[99959]: pgmap v5049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:15.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:15.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:15 smithi078 ceph-mon[91136]: pgmap v5049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:15.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:15 smithi146 ceph-mon[103278]: pgmap v5049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:17 smithi078 ceph-mon[91136]: pgmap v5050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:17 smithi078 ceph-mon[99959]: pgmap v5050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:17 smithi146 ceph-mon[103278]: pgmap v5050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:19 smithi146 ceph-mon[103278]: pgmap v5051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:19 smithi078 ceph-mon[91136]: pgmap v5051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:19 smithi078 ceph-mon[99959]: pgmap v5051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:08:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:08:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:20.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:20.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:20.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:21 smithi146 ceph-mon[103278]: pgmap v5052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:21 smithi078 ceph-mon[91136]: pgmap v5052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:21 smithi078 ceph-mon[99959]: pgmap v5052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:23.332 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:08:23.498 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:23 smithi146 ceph-mon[103278]: pgmap v5053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:23 smithi078 ceph-mon[99959]: pgmap v5053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:23 smithi078 ceph-mon[91136]: pgmap v5053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:23.858 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:08:23.858 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 2h - - 2024-09-20T23:08:23.858 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 2h - - 2024-09-20T23:08:23.858 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 13s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:08:23.858 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 8s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 13s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 13s ago 2h 572M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 13s ago 3h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 8s ago 2h 203M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 13s ago 2h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 13s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 8s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 13s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 13s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:08:23.859 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 13s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:08:23.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 13s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:08:23.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:08:23.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:08:23.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:08:23.860 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:08:23.860 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 8s ago 2h 75.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:08:23.861 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 13s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:08:23.861 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:08:23.861 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 13s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:08:23.861 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:08:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:08:24.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:08:24.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:08:24.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:08:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:08:24.442 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:08:24.443 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:08:24.443 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:08:24.443 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:08:24.443 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:08:24.443 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:08:24.443 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:08:24.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:24 smithi078 ceph-mon[91136]: from='client.55491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:24.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/102263857' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:24.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:24 smithi078 ceph-mon[99959]: from='client.55491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:24.444 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/102263857' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:24.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:24 smithi146 ceph-mon[103278]: from='client.55491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:24.469 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/102263857' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:08:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:24.747 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:24.747 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:24.747 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:25 smithi146 ceph-mon[103278]: from='client.75416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:25 smithi146 ceph-mon[103278]: from='client.75422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:25 smithi146 ceph-mon[103278]: pgmap v5054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/690695825' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[99959]: from='client.75416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[99959]: from='client.75422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[99959]: pgmap v5054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/690695825' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[91136]: from='client.75416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[91136]: from='client.75422 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[91136]: pgmap v5054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/690695825' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:26 smithi146 ceph-mon[103278]: from='client.55515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:26 smithi078 ceph-mon[91136]: from='client.55515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:26 smithi078 ceph-mon[99959]: from='client.55515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:27 smithi146 ceph-mon[103278]: pgmap v5055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:27 smithi078 ceph-mon[99959]: pgmap v5055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:27 smithi078 ceph-mon[91136]: pgmap v5055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:29 smithi146 ceph-mon[103278]: pgmap v5056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:29 smithi078 ceph-mon[91136]: pgmap v5056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:29 smithi078 ceph-mon[99959]: pgmap v5056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:08:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:08:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:30.829Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:30.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:30.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:31 smithi146 ceph-mon[103278]: pgmap v5057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:31.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[91136]: pgmap v5057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:31.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.822 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[99959]: pgmap v5057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:31.822 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.822 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:31.822 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:33 smithi146 ceph-mon[103278]: pgmap v5058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:33 smithi078 ceph-mon[99959]: pgmap v5058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:33 smithi078 ceph-mon[91136]: pgmap v5058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:35 smithi146 ceph-mon[103278]: pgmap v5059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:35 smithi078 ceph-mon[91136]: pgmap v5059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:35 smithi078 ceph-mon[99959]: pgmap v5059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:36 smithi146 ceph-mon[103278]: pgmap v5060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[99959]: pgmap v5060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:36.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:36 smithi078 ceph-mon[91136]: pgmap v5060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:39 smithi146 ceph-mon[103278]: pgmap v5061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:39 smithi078 ceph-mon[91136]: pgmap v5061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:39 smithi078 ceph-mon[99959]: pgmap v5061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:08:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:08:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:40.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:40.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:40.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:41 smithi146 ceph-mon[103278]: pgmap v5062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:41 smithi078 ceph-mon[91136]: pgmap v5062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:41 smithi078 ceph-mon[99959]: pgmap v5062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:43.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:43 smithi146 ceph-mon[103278]: pgmap v5063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:43.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:43 smithi078 ceph-mon[91136]: pgmap v5063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:43 smithi078 ceph-mon[99959]: pgmap v5063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:45 smithi146 ceph-mon[103278]: pgmap v5064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:45 smithi078 ceph-mon[91136]: pgmap v5064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:45 smithi078 ceph-mon[99959]: pgmap v5064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:47.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:47 smithi078 ceph-mon[91136]: pgmap v5065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:47.566 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:47 smithi078 ceph-mon[99959]: pgmap v5065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:47 smithi146 ceph-mon[103278]: pgmap v5065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:49 smithi146 ceph-mon[103278]: pgmap v5066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:49 smithi078 ceph-mon[91136]: pgmap v5066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:49 smithi078 ceph-mon[99959]: pgmap v5066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:50.928 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:08:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:08:50.928 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:50.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:50.928 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:50.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:08:50.928 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:08:50.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:08:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:51 smithi146 ceph-mon[103278]: pgmap v5067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[91136]: pgmap v5067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[99959]: pgmap v5067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:51.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:53 smithi146 ceph-mon[103278]: pgmap v5068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:53 smithi078 ceph-mon[99959]: pgmap v5068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:53 smithi078 ceph-mon[91136]: pgmap v5068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:55.039 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:08:55.568 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 2h - - 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 2h - - 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 4s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 0s ago 2h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 4s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 0s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 4s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 4s ago 3h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 0s ago 2h 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 4s ago 2h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 4s ago 2h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 0s ago 2h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 4s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:08:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 4s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 4s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 4s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 0s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 0s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 0s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 0s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 0s ago 2h 77.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 4s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 4s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:08:55.570 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:08:55.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:55 smithi146 ceph-mon[103278]: pgmap v5069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:55 smithi078 ceph-mon[99959]: pgmap v5069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:55 smithi078 ceph-mon[91136]: pgmap v5069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:08:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:08:55.889 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:08:56.155 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:08:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:08:56.156 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:08:56.466 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:08:56.467 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:08:56.467 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:08:56.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[91136]: from='client.55527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:56.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[91136]: from='client.75452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:56.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/111599725' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[99959]: from='client.55527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[99959]: from='client.75452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/111599725' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:56 smithi146 ceph-mon[103278]: from='client.55527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:56 smithi146 ceph-mon[103278]: from='client.75452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:08:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/111599725' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[91136]: from='client.75458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[91136]: from='client.75470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[91136]: pgmap v5070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1203695957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[99959]: from='client.75458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[99959]: from='client.75470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[99959]: pgmap v5070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1203695957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:57.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:57 smithi146 ceph-mon[103278]: from='client.75458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:57.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:57 smithi146 ceph-mon[103278]: from='client.75470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:08:57.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:57 smithi146 ceph-mon[103278]: pgmap v5070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:08:57.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1203695957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:08:58.463 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:58.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:08:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:08:59 smithi078 ceph-mon[91136]: pgmap v5071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:59.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:59.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:08:59 smithi078 ceph-mon[99959]: pgmap v5071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:08:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:08:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:08:59 smithi146 ceph-mon[103278]: pgmap v5071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:00.956 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:09:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:09:00.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:00.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:00.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:00.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:00.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:00.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:01 smithi146 ceph-mon[103278]: pgmap v5072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:01 smithi078 ceph-mon[91136]: pgmap v5072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:01 smithi078 ceph-mon[99959]: pgmap v5072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:03 smithi146 ceph-mon[103278]: pgmap v5073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:03 smithi078 ceph-mon[91136]: pgmap v5073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:03 smithi078 ceph-mon[99959]: pgmap v5073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:05.634 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:05 smithi146 ceph-mon[103278]: pgmap v5074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:05 smithi078 ceph-mon[91136]: pgmap v5074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:05 smithi078 ceph-mon[99959]: pgmap v5074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:07 smithi146 ceph-mon[103278]: pgmap v5075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:07 smithi078 ceph-mon[91136]: pgmap v5075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:07 smithi078 ceph-mon[99959]: pgmap v5075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:09 smithi146 ceph-mon[103278]: pgmap v5076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:09 smithi078 ceph-mon[99959]: pgmap v5076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:09 smithi078 ceph-mon[91136]: pgmap v5076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:09:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:09:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:10.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:10.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:10.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:11 smithi146 ceph-mon[103278]: pgmap v5077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[91136]: pgmap v5077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[99959]: pgmap v5077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:13 smithi146 ceph-mon[103278]: pgmap v5078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:13 smithi078 ceph-mon[91136]: pgmap v5078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:13 smithi078 ceph-mon[99959]: pgmap v5078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:15.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:15 smithi146 ceph-mon[103278]: pgmap v5079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:15 smithi078 ceph-mon[91136]: pgmap v5079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:15 smithi078 ceph-mon[99959]: pgmap v5079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:16 smithi146 ceph-mon[103278]: pgmap v5080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[91136]: pgmap v5080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:16 smithi078 ceph-mon[99959]: pgmap v5080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:19.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:19.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:19 smithi078 ceph-mon[99959]: pgmap v5081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:19.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:19.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:19 smithi078 ceph-mon[91136]: pgmap v5081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:19 smithi146 ceph-mon[103278]: pgmap v5081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:09:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:09:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:20.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:20.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:20.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:21 smithi146 ceph-mon[103278]: pgmap v5082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:21 smithi078 ceph-mon[91136]: pgmap v5082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:21 smithi078 ceph-mon[99959]: pgmap v5082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:23 smithi146 ceph-mon[103278]: pgmap v5083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:23 smithi078 ceph-mon[91136]: pgmap v5083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:23 smithi078 ceph-mon[99959]: pgmap v5083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:25 smithi146 ceph-mon[103278]: pgmap v5084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:25 smithi078 ceph-mon[91136]: pgmap v5084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:25 smithi078 ceph-mon[99959]: pgmap v5084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:26.762 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 3h - - 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 2h - - 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 16s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 11s ago 2h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 16s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 11s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 16s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 16s ago 3h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:09:27.287 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 11s ago 2h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 16s ago 2h 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 16s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 11s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 16s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 16s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 16s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 16s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 11s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 11s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 11s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 11s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:09:27.288 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 11s ago 2h 78.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:09:27.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 16s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:09:27.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:09:27.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 16s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:09:27.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 11s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:09:27.596 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:27 smithi078 ceph-mon[91136]: pgmap v5085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:27.597 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:27 smithi078 ceph-mon[99959]: pgmap v5085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:09:27.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:09:27.598 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:09:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:27 smithi146 ceph-mon[103278]: pgmap v5085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:09:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:09:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:09:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:09:27.866 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:09:28.176 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:09:28.176 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:28.176 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:28.176 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:28.176 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[91136]: from='client.75482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[91136]: from='client.55569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[91136]: from='client.75494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4036798946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2913904827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[99959]: from='client.75482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[99959]: from='client.55569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[99959]: from='client.75494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4036798946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2913904827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:28.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:28 smithi146 ceph-mon[103278]: from='client.75482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:28 smithi146 ceph-mon[103278]: from='client.55569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:28 smithi146 ceph-mon[103278]: from='client.75494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4036798946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:09:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2913904827' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:09:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:29 smithi146 ceph-mon[103278]: from='client.75506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:29 smithi146 ceph-mon[103278]: pgmap v5086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:29 smithi078 ceph-mon[91136]: from='client.75506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:29 smithi078 ceph-mon[91136]: pgmap v5086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:29 smithi078 ceph-mon[99959]: from='client.75506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:29 smithi078 ceph-mon[99959]: pgmap v5086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:09:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:09:31.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:30.834Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:31.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:30.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:31.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:30.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[91136]: pgmap v5087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[99959]: pgmap v5087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:31 smithi146 ceph-mon[103278]: pgmap v5087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:33 smithi146 ceph-mon[103278]: pgmap v5088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:33 smithi078 ceph-mon[91136]: pgmap v5088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:33 smithi078 ceph-mon[99959]: pgmap v5088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:35 smithi146 ceph-mon[103278]: pgmap v5089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:35 smithi078 ceph-mon[91136]: pgmap v5089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:35 smithi078 ceph-mon[99959]: pgmap v5089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:36 smithi146 ceph-mon[103278]: pgmap v5090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[99959]: pgmap v5090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:36 smithi078 ceph-mon[91136]: pgmap v5090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:39.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:39.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:39 smithi078 ceph-mon[91136]: pgmap v5091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:39.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:39.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:39 smithi078 ceph-mon[99959]: pgmap v5091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:39 smithi146 ceph-mon[103278]: pgmap v5091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:40.837 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:09:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:09:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:40.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:40.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:40.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:41 smithi146 ceph-mon[103278]: pgmap v5092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:41 smithi078 ceph-mon[91136]: pgmap v5092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:41 smithi078 ceph-mon[99959]: pgmap v5092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:43 smithi146 ceph-mon[103278]: pgmap v5093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:43 smithi078 ceph-mon[91136]: pgmap v5093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:43 smithi078 ceph-mon[99959]: pgmap v5093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:45.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:45 smithi146 ceph-mon[103278]: pgmap v5094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:45 smithi078 ceph-mon[91136]: pgmap v5094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:45 smithi078 ceph-mon[99959]: pgmap v5094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:47.540 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:47 smithi146 ceph-mon[103278]: pgmap v5095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:47.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:47 smithi078 ceph-mon[91136]: pgmap v5095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:47.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:47 smithi078 ceph-mon[99959]: pgmap v5095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:49 smithi146 ceph-mon[103278]: pgmap v5096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:49 smithi078 ceph-mon[99959]: pgmap v5096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:49 smithi078 ceph-mon[91136]: pgmap v5096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:09:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:09:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:50.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:50.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:09:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:09:50.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:09:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:51 smithi146 ceph-mon[103278]: pgmap v5097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[91136]: pgmap v5097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[99959]: pgmap v5097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:53 smithi146 ceph-mon[103278]: pgmap v5098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:53 smithi078 ceph-mon[91136]: pgmap v5098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:53 smithi078 ceph-mon[99959]: pgmap v5098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:55 smithi146 ceph-mon[103278]: pgmap v5099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:55 smithi078 ceph-mon[91136]: pgmap v5099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:55 smithi078 ceph-mon[99959]: pgmap v5099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:56 smithi146 ceph-mon[103278]: pgmap v5100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[99959]: pgmap v5100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:09:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:56 smithi078 ceph-mon[91136]: pgmap v5100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:09:58.492 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:09:59.020 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 3h - - 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 3h - - 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 7s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 3s ago 2h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 7s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 3s ago 2h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 7s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 7s ago 3h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 3s ago 3h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 7s ago 2h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:09:59.021 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 3s ago 2h 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 3s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 3s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 3s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 3s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 3s ago 2h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:09:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:09:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:59 smithi078 ceph-mon[99959]: pgmap v5101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:09:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:59 smithi078 ceph-mon[91136]: pgmap v5101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:09:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:59.336 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:09:59.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:09:59.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:09:59.338 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:09:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:09:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:59 smithi146 ceph-mon[103278]: pgmap v5101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:09:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:09:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:09:59.611 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:09:59.612 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:09:59.948 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:09:59.948 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:09:59.948 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:09:59.948 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T23:09:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[91136]: from='client.75518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[91136]: from='client.75524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[91136]: from='client.55611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2947275367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:00.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1005299960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:00.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[99959]: from='client.75518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[99959]: from='client.75524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[99959]: from='client.55611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2947275367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:00.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1005299960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:00.507 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:00 smithi146 ceph-mon[103278]: from='client.75518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:00 smithi146 ceph-mon[103278]: from='client.75524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:00 smithi146 ceph-mon[103278]: from='client.55611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2947275367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1005299960' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:10:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:10:01.116 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:00.836Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:01.116 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:00.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:01.116 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:00.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:01.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:01 smithi078 ceph-mon[99959]: from='client.75542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:01.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:01 smithi078 ceph-mon[99959]: pgmap v5102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:01.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:01 smithi078 ceph-mon[91136]: from='client.75542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:01.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:01 smithi078 ceph-mon[91136]: pgmap v5102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:01 smithi146 ceph-mon[103278]: from='client.75542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:01 smithi146 ceph-mon[103278]: pgmap v5102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:03 smithi146 ceph-mon[103278]: pgmap v5103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:03.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:03 smithi078 ceph-mon[91136]: pgmap v5103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:03.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:03 smithi078 ceph-mon[99959]: pgmap v5103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:05 smithi146 ceph-mon[103278]: pgmap v5104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:05 smithi078 ceph-mon[91136]: pgmap v5104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:05 smithi078 ceph-mon[99959]: pgmap v5104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:07.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:07 smithi146 ceph-mon[103278]: pgmap v5105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:07.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:07 smithi078 ceph-mon[91136]: pgmap v5105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:07.629 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:07 smithi078 ceph-mon[99959]: pgmap v5105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:09 smithi146 ceph-mon[103278]: pgmap v5106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:09 smithi078 ceph-mon[91136]: pgmap v5106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:09 smithi078 ceph-mon[99959]: pgmap v5106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:10:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:10:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:10.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:10.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:10.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:11 smithi146 ceph-mon[103278]: pgmap v5107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:11 smithi078 ceph-mon[91136]: pgmap v5107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:11 smithi078 ceph-mon[99959]: pgmap v5107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:12 smithi146 ceph-mon[103278]: pgmap v5108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[91136]: pgmap v5108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:12 smithi078 ceph-mon[99959]: pgmap v5108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:14 smithi146 ceph-mon[103278]: pgmap v5109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:14 smithi078 ceph-mon[91136]: pgmap v5109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:14 smithi078 ceph-mon[99959]: pgmap v5109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:17 smithi146 ceph-mon[103278]: pgmap v5110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[99959]: pgmap v5110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:17 smithi078 ceph-mon[91136]: pgmap v5110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:19 smithi146 ceph-mon[103278]: pgmap v5111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:19 smithi078 ceph-mon[99959]: pgmap v5111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:19 smithi078 ceph-mon[91136]: pgmap v5111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:20.995 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:10:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:10:20.995 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:20.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:20.996 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:20.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:20.996 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:20.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:21.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:21 smithi078 ceph-mon[99959]: pgmap v5112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:21.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:21 smithi078 ceph-mon[91136]: pgmap v5112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:21 smithi146 ceph-mon[103278]: pgmap v5112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:23 smithi146 ceph-mon[103278]: pgmap v5113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:23 smithi078 ceph-mon[91136]: pgmap v5113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:23 smithi078 ceph-mon[99959]: pgmap v5113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:25 smithi146 ceph-mon[103278]: pgmap v5114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:25 smithi078 ceph-mon[91136]: pgmap v5114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:25 smithi078 ceph-mon[99959]: pgmap v5114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:27.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:27 smithi078 ceph-mon[91136]: pgmap v5115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:27.650 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:27 smithi078 ceph-mon[99959]: pgmap v5115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:27 smithi146 ceph-mon[103278]: pgmap v5115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:28.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:29 smithi146 ceph-mon[103278]: pgmap v5116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:29 smithi078 ceph-mon[91136]: pgmap v5116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:29 smithi078 ceph-mon[99959]: pgmap v5116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:30.257 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:10:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 3h - - 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 3h - - 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 19s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 14s ago 2h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 19s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 14s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 19s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 19s ago 3h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 14s ago 3h 204M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 19s ago 3h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 19s ago 2h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:10:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 14s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 19s ago 2h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 19s ago 2h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 19s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 19s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 14s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 14s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 14s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 14s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 14s ago 2h 75.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 19s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 19s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:10:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 14s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:10:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:10:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:10:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:30.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:30.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:30.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:10:31.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:10:31.093 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:10:31.362 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:10:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:10:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:10:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:10:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:10:31.363 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:10:31.683 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:10:31.683 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:10:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:10:31.683 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:10:31.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:31 smithi078 ceph-mon[99959]: from='client.55635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:31.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:31 smithi078 ceph-mon[99959]: pgmap v5117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:31.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2693076331' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:31.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:31 smithi078 ceph-mon[91136]: from='client.55635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:31.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:31 smithi078 ceph-mon[91136]: pgmap v5117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:31.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2693076331' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:31 smithi146 ceph-mon[103278]: from='client.55635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:31 smithi146 ceph-mon[103278]: pgmap v5117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2693076331' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:10:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='client.75560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='client.75566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='client.75578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/160283488' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='client.75560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='client.75566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='client.75578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/160283488' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='client.75560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='client.75566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='client.75578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:10:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:32.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/160283488' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:10:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:33 smithi146 ceph-mon[103278]: pgmap v5118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:33 smithi078 ceph-mon[91136]: pgmap v5118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:33 smithi078 ceph-mon[99959]: pgmap v5118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:35 smithi146 ceph-mon[103278]: pgmap v5119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:35 smithi078 ceph-mon[91136]: pgmap v5119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:35 smithi078 ceph-mon[99959]: pgmap v5119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:37 smithi146 ceph-mon[103278]: pgmap v5120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[91136]: pgmap v5120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:37 smithi078 ceph-mon[99959]: pgmap v5120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:39 smithi146 ceph-mon[103278]: pgmap v5121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:39 smithi078 ceph-mon[91136]: pgmap v5121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:39 smithi078 ceph-mon[99959]: pgmap v5121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:40.919 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:10:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:10:40.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:40.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:40.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:40.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:40.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:40.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:41.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:41 smithi078 ceph-mon[99959]: pgmap v5122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:41.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:41 smithi078 ceph-mon[91136]: pgmap v5122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:41 smithi146 ceph-mon[103278]: pgmap v5122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:43 smithi146 ceph-mon[103278]: pgmap v5123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:43 smithi078 ceph-mon[91136]: pgmap v5123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:43 smithi078 ceph-mon[99959]: pgmap v5123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:45 smithi146 ceph-mon[103278]: pgmap v5124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:45.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:45 smithi078 ceph-mon[91136]: pgmap v5124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:45.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:45 smithi078 ceph-mon[99959]: pgmap v5124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:47.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:47 smithi146 ceph-mon[103278]: pgmap v5125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:47.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:47 smithi078 ceph-mon[91136]: pgmap v5125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:47.664 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:47 smithi078 ceph-mon[99959]: pgmap v5125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:49.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:49.471 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:49 smithi146 ceph-mon[103278]: pgmap v5126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:49 smithi078 ceph-mon[91136]: pgmap v5126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:49 smithi078 ceph-mon[99959]: pgmap v5126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:10:50.977 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:10:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:10:50.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:50.839Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:50.977 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:50.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:10:50.978 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:10:50.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:10:51.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:51 smithi078 ceph-mon[91136]: pgmap v5127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:51.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:51 smithi078 ceph-mon[99959]: pgmap v5127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:51 smithi146 ceph-mon[103278]: pgmap v5127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:52 smithi146 ceph-mon[103278]: pgmap v5128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[91136]: pgmap v5128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:53.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:52 smithi078 ceph-mon[99959]: pgmap v5128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:10:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:55 smithi146 ceph-mon[103278]: pgmap v5129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:10:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:55 smithi078 ceph-mon[91136]: pgmap v5129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:10:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:55 smithi078 ceph-mon[99959]: pgmap v5129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:57 smithi146 ceph-mon[103278]: pgmap v5130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[99959]: pgmap v5130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.670 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[91136]: pgmap v5130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:10:57.670 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.670 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:57.670 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:10:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:10:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:10:59 smithi146 ceph-mon[103278]: pgmap v5131: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:10:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:10:59 smithi078 ceph-mon[99959]: pgmap v5131: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:10:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:10:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:10:59 smithi078 ceph-mon[91136]: pgmap v5131: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:11:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:11:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:11:01.251 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:00.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:01.251 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:00.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:01.251 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:00.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:01 smithi078 ceph-mon[91136]: pgmap v5132: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T23:11:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:01 smithi078 ceph-mon[99959]: pgmap v5132: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T23:11:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:01 smithi146 ceph-mon[103278]: pgmap v5132: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-20T23:11:02.016 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:11:02.577 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:11:02.577 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 3h - - 2024-09-20T23:11:02.577 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 3h - - 2024-09-20T23:11:02.577 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 10s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:11:02.577 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 6s ago 2h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 10s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 6s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 10s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 10s ago 3h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 6s ago 3h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 10s ago 3h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 10s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 6s ago 2h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 10s ago 3h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 10s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 10s ago 2h 69.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 10s ago 2h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:11:02.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 6s ago 2h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 6s ago 2h 70.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 6s ago 2h 75.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 6s ago 2h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 6s ago 2h 75.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 10s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 10s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:11:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:11:02.929 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:11:02.929 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:11:02.929 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:11:02.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:11:02.930 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:11:02.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:11:02.931 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:11:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:11:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:11:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:11:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:11:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:11:03.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:11:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[99959]: from='client.75590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[99959]: from='client.75596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[99959]: pgmap v5133: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2867921112' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[91136]: from='client.75590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[91136]: from='client.75596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[91136]: pgmap v5133: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:11:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2867921112' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:03.576 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:11:03.576 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:11:03.576 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:11:03.577 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:11:03.577 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:11:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:03 smithi146 ceph-mon[103278]: from='client.75590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:03 smithi146 ceph-mon[103278]: from='client.75596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:03 smithi146 ceph-mon[103278]: pgmap v5133: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:11:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2867921112' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:04 smithi146 ceph-mon[103278]: from='client.55683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:04 smithi146 ceph-mon[103278]: from='client.75614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2515174772' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:04 smithi078 ceph-mon[99959]: from='client.55683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:04 smithi078 ceph-mon[99959]: from='client.75614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2515174772' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:04 smithi078 ceph-mon[91136]: from='client.55683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:04 smithi078 ceph-mon[91136]: from='client.75614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2515174772' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:05 smithi146 ceph-mon[103278]: pgmap v5134: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:11:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:05 smithi078 ceph-mon[91136]: pgmap v5134: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:11:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:05 smithi078 ceph-mon[99959]: pgmap v5134: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:11:07.673 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:07 smithi078 ceph-mon[91136]: pgmap v5135: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:11:07.673 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:07 smithi078 ceph-mon[99959]: pgmap v5135: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:11:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:07 smithi146 ceph-mon[103278]: pgmap v5135: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:11:09.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:09.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:09 smithi146 ceph-mon[103278]: pgmap v5136: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:11:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:09 smithi078 ceph-mon[91136]: pgmap v5136: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:09 smithi078 ceph-mon[99959]: pgmap v5136: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:11:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:11:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:11:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:10.841Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:10.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:10.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:11 smithi146 ceph-mon[103278]: pgmap v5137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T23:11:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:11 smithi078 ceph-mon[91136]: pgmap v5137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T23:11:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:11 smithi078 ceph-mon[99959]: pgmap v5137: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T23:11:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:12 smithi146 ceph-mon[103278]: pgmap v5138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-20T23:11:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[91136]: pgmap v5138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:12 smithi078 ceph-mon[99959]: pgmap v5138: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-20T23:11:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:14 smithi146 ceph-mon[103278]: pgmap v5139: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:11:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:14 smithi078 ceph-mon[91136]: pgmap v5139: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:11:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:14 smithi078 ceph-mon[99959]: pgmap v5139: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[91136]: pgmap v5140: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[99959]: pgmap v5140: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:17 smithi146 ceph-mon[103278]: pgmap v5140: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-20T23:11:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:19 smithi146 ceph-mon[103278]: pgmap v5141: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:11:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:19 smithi078 ceph-mon[99959]: pgmap v5141: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:11:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:19 smithi078 ceph-mon[91136]: pgmap v5141: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:11:21.029 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:11:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:11:21.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:20.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:21.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:20.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:21.029 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:20.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:21 smithi146 ceph-mon[103278]: pgmap v5142: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:11:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:21 smithi078 ceph-mon[91136]: pgmap v5142: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:11:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:21 smithi078 ceph-mon[99959]: pgmap v5142: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:11:23.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:23 smithi078 ceph-mon[91136]: pgmap v5143: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-20T23:11:23.419 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:23 smithi078 ceph-mon[99959]: pgmap v5143: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-20T23:11:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:23 smithi146 ceph-mon[103278]: pgmap v5143: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-20T23:11:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:25 smithi146 ceph-mon[103278]: pgmap v5144: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:11:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:25 smithi078 ceph-mon[91136]: pgmap v5144: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:11:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:25 smithi078 ceph-mon[99959]: pgmap v5144: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:11:27.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:27 smithi078 ceph-mon[91136]: pgmap v5145: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:11:27.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:27 smithi078 ceph-mon[99959]: pgmap v5145: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:11:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:27 smithi146 ceph-mon[103278]: pgmap v5145: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:11:28.634 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:29.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:29.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:29 smithi146 ceph-mon[103278]: pgmap v5146: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:11:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:29 smithi078 ceph-mon[91136]: pgmap v5146: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:11:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:29 smithi078 ceph-mon[99959]: pgmap v5146: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:11:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:11:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:11:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:30.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:30.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:30.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:31 smithi146 ceph-mon[103278]: pgmap v5147: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:31 smithi078 ceph-mon[91136]: pgmap v5147: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:31 smithi078 ceph-mon[99959]: pgmap v5147: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:32 smithi146 ceph-mon[103278]: pgmap v5148: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[99959]: pgmap v5148: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:33.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:32 smithi078 ceph-mon[91136]: pgmap v5148: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:33.873 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:11:34.400 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:11:34.400 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 3h - - 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 3h - - 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 2s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 17s ago 2h 87.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 2s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 17s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 2s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 2s ago 3h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 17s ago 3h 205M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 2s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 2s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 17s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:11:34.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 2s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 2s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 2s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 2s ago 2h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 17s ago 2h 86.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 17s ago 2h 71.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 17s ago 2h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 17s ago 2h 71.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 17s ago 2h 78.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 2s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 2s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:11:34.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 17s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:11:34.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:11:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:11:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:11:34.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:11:34.715 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:11:34.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:11:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:11:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:11:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:11:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:11:34.982 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:11:34.982 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:11:35.290 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:11:35.290 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:11:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:11:35.290 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:11:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:11:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:35 smithi146 ceph-mon[103278]: from='client.55707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:35 smithi146 ceph-mon[103278]: from='client.55713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:35 smithi146 ceph-mon[103278]: pgmap v5149: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:35 smithi146 ceph-mon[103278]: from='client.75638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2465275434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/333905867' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[91136]: from='client.55707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[91136]: from='client.55713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[91136]: pgmap v5149: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[91136]: from='client.75638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2465275434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/333905867' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[99959]: from='client.55707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[99959]: from='client.55713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[99959]: pgmap v5149: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[99959]: from='client.75638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2465275434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:11:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/333905867' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:11:36.637 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:36 smithi146 ceph-mon[103278]: from='client.55731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:36 smithi078 ceph-mon[91136]: from='client.55731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:36 smithi078 ceph-mon[99959]: from='client.55731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[99959]: pgmap v5150: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[91136]: pgmap v5150: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:37 smithi146 ceph-mon[103278]: pgmap v5150: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:39 smithi146 ceph-mon[103278]: pgmap v5151: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:39 smithi078 ceph-mon[91136]: pgmap v5151: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:39 smithi078 ceph-mon[99959]: pgmap v5151: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:11:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:11:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:40.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:40.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:40.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:41 smithi146 ceph-mon[103278]: pgmap v5152: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:41 smithi078 ceph-mon[91136]: pgmap v5152: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:41 smithi078 ceph-mon[99959]: pgmap v5152: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:43 smithi078 ceph-mon[91136]: pgmap v5153: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:43 smithi078 ceph-mon[99959]: pgmap v5153: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:43 smithi146 ceph-mon[103278]: pgmap v5153: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:45.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:45 smithi078 ceph-mon[91136]: pgmap v5154: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:45 smithi078 ceph-mon[99959]: pgmap v5154: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:45 smithi146 ceph-mon[103278]: pgmap v5154: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:47.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:47 smithi078 ceph-mon[91136]: pgmap v5155: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:47.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:47 smithi078 ceph-mon[99959]: pgmap v5155: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:47 smithi146 ceph-mon[103278]: pgmap v5155: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:49 smithi146 ceph-mon[103278]: pgmap v5156: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:49 smithi078 ceph-mon[91136]: pgmap v5156: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:49 smithi078 ceph-mon[99959]: pgmap v5156: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:11:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:11:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:50.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:50.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:11:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:11:50.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:11:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:51 smithi146 ceph-mon[103278]: pgmap v5157: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:51 smithi078 ceph-mon[91136]: pgmap v5157: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:51 smithi078 ceph-mon[99959]: pgmap v5157: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:53 smithi146 ceph-mon[103278]: pgmap v5158: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[91136]: pgmap v5158: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:53 smithi078 ceph-mon[99959]: pgmap v5158: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:55 smithi146 ceph-mon[103278]: pgmap v5159: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:55 smithi078 ceph-mon[99959]: pgmap v5159: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:55 smithi078 ceph-mon[91136]: pgmap v5159: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:57.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[91136]: pgmap v5160: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:57.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[99959]: pgmap v5160: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:57.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:57 smithi146 ceph-mon[103278]: pgmap v5160: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:11:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:11:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:11:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:11:59 smithi146 ceph-mon[103278]: pgmap v5161: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:11:59 smithi078 ceph-mon[91136]: pgmap v5161: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:11:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:11:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:11:59 smithi078 ceph-mon[99959]: pgmap v5161: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:12:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:12:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:00.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:00.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:12:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:00.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:01 smithi146 ceph-mon[103278]: pgmap v5162: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:01 smithi078 ceph-mon[91136]: pgmap v5162: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:01 smithi078 ceph-mon[99959]: pgmap v5162: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:03.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:03 smithi078 ceph-mon[99959]: pgmap v5163: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:03.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:03 smithi078 ceph-mon[91136]: pgmap v5163: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:03 smithi146 ceph-mon[103278]: pgmap v5163: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:05.583 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:12:05.712 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:05 smithi078 ceph-mon[99959]: pgmap v5164: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:05.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:05 smithi078 ceph-mon[91136]: pgmap v5164: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:05 smithi146 ceph-mon[103278]: pgmap v5164: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:06.111 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 3h - - 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 3h - - 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 14s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 9s ago 2h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 9s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 3h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 9s ago 3h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:12:06.112 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 14s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 9s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 14s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 9s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 9s ago 2h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 9s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 9s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 9s ago 2h 78.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:12:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 14s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:12:06.114 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:12:06.422 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:12:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:12:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:12:06.424 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:12:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:12:06.694 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:12:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:12:07.000 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:12:07.000 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:07.000 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:07.000 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:07.000 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:07.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[91136]: from='client.55743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[91136]: from='client.55749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[91136]: from='client.55755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[91136]: pgmap v5165: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3430783348' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/745023476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[99959]: from='client.55743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[99959]: from='client.55749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[99959]: from='client.55755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[99959]: pgmap v5165: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3430783348' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:07.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/745023476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:07 smithi146 ceph-mon[103278]: from='client.55743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:07 smithi146 ceph-mon[103278]: from='client.55749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:07 smithi146 ceph-mon[103278]: from='client.55755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:07 smithi146 ceph-mon[103278]: pgmap v5165: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3430783348' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/745023476' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:08 smithi146 ceph-mon[103278]: from='client.55767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:08 smithi078 ceph-mon[91136]: from='client.55767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:08 smithi078 ceph-mon[99959]: from='client.55767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:09.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:09.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:09 smithi146 ceph-mon[103278]: pgmap v5166: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:09 smithi078 ceph-mon[91136]: pgmap v5166: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:09 smithi078 ceph-mon[99959]: pgmap v5166: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:12:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:12:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:10.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:10.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:12:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:10.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:11.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:11 smithi146 ceph-mon[103278]: pgmap v5167: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:11 smithi078 ceph-mon[91136]: pgmap v5167: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:11 smithi078 ceph-mon[99959]: pgmap v5167: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:13 smithi146 ceph-mon[103278]: pgmap v5168: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[99959]: pgmap v5168: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:13 smithi078 ceph-mon[91136]: pgmap v5168: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:15 smithi146 ceph-mon[103278]: pgmap v5169: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:15 smithi078 ceph-mon[91136]: pgmap v5169: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:15 smithi078 ceph-mon[99959]: pgmap v5169: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:17.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[91136]: pgmap v5170: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:17.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[99959]: pgmap v5170: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:17.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:17 smithi146 ceph-mon[103278]: pgmap v5170: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:19 smithi146 ceph-mon[103278]: pgmap v5171: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:19 smithi078 ceph-mon[91136]: pgmap v5171: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:19 smithi078 ceph-mon[99959]: pgmap v5171: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:20.970 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:12:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:12:20.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:20.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:20.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:20.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:12:20.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:20.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:21 smithi146 ceph-mon[103278]: pgmap v5172: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:21 smithi078 ceph-mon[91136]: pgmap v5172: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:21 smithi078 ceph-mon[99959]: pgmap v5172: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:23 smithi146 ceph-mon[103278]: pgmap v5173: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:23 smithi078 ceph-mon[91136]: pgmap v5173: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:23 smithi078 ceph-mon[99959]: pgmap v5173: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:25 smithi078 ceph-mon[91136]: pgmap v5174: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:25 smithi078 ceph-mon[99959]: pgmap v5174: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:25 smithi146 ceph-mon[103278]: pgmap v5174: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:27.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:27 smithi078 ceph-mon[91136]: pgmap v5175: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:27.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:27 smithi078 ceph-mon[99959]: pgmap v5175: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:27 smithi146 ceph-mon[103278]: pgmap v5175: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:29 smithi146 ceph-mon[103278]: pgmap v5176: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:29 smithi078 ceph-mon[99959]: pgmap v5176: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:29 smithi078 ceph-mon[91136]: pgmap v5176: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:12:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:12:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:30.846Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:30.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:12:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:30.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:31.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:31 smithi146 ceph-mon[103278]: pgmap v5177: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:31 smithi078 ceph-mon[91136]: pgmap v5177: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:31 smithi078 ceph-mon[99959]: pgmap v5177: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:33 smithi146 ceph-mon[103278]: pgmap v5178: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[91136]: pgmap v5178: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[99959]: pgmap v5178: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:35 smithi146 ceph-mon[103278]: pgmap v5179: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:35 smithi078 ceph-mon[99959]: pgmap v5179: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:35 smithi078 ceph-mon[91136]: pgmap v5179: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:37.294 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:12:37.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[99959]: pgmap v5180: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:37.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[91136]: pgmap v5180: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:37 smithi146 ceph-mon[103278]: pgmap v5180: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:37.821 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:12:37.821 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 3h - - 2024-09-20T23:12:37.821 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 3h - - 2024-09-20T23:12:37.821 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 5s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:12:37.821 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 0s ago 2h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:12:37.821 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 5s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 0s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 5s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 5s ago 3h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 0s ago 3h 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 5s ago 3h 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 5s ago 2h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 0s ago 2h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 5s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 5s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 5s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 5s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 0s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:12:37.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 0s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 0s ago 2h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 0s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 0s ago 2h 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 5s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 5s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:12:37.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 0s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:12:38.128 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:12:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:12:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:12:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:12:38.129 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:12:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:12:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:12:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:12:38.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:12:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:38 smithi146 ceph-mon[103278]: from='client.75698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1594305823' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:38.709 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:12:38.709 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:12:38.709 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:12:38.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:38 smithi078 ceph-mon[99959]: from='client.75698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:38.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1594305823' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:38.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:38 smithi078 ceph-mon[91136]: from='client.75698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:38.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1594305823' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:39 smithi146 ceph-mon[103278]: from='client.55785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:39 smithi146 ceph-mon[103278]: from='client.55791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:39 smithi146 ceph-mon[103278]: pgmap v5181: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:39 smithi146 ceph-mon[103278]: from='client.55803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4220784222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[91136]: from='client.55785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[91136]: from='client.55791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[91136]: pgmap v5181: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[91136]: from='client.55803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4220784222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[99959]: from='client.55785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[99959]: from='client.55791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[99959]: pgmap v5181: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[99959]: from='client.55803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:12:39.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4220784222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:12:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:12:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:12:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:40.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:40.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:12:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:40.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:41 smithi146 ceph-mon[103278]: pgmap v5182: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:41 smithi078 ceph-mon[91136]: pgmap v5182: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:41 smithi078 ceph-mon[99959]: pgmap v5182: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:43 smithi146 ceph-mon[103278]: pgmap v5183: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:43 smithi078 ceph-mon[91136]: pgmap v5183: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:43 smithi078 ceph-mon[99959]: pgmap v5183: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:45 smithi146 ceph-mon[103278]: pgmap v5184: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:45 smithi078 ceph-mon[99959]: pgmap v5184: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:45 smithi078 ceph-mon[91136]: pgmap v5184: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:47.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:47 smithi078 ceph-mon[91136]: pgmap v5185: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:47.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:47 smithi078 ceph-mon[99959]: pgmap v5185: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:47 smithi146 ceph-mon[103278]: pgmap v5185: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:49 smithi146 ceph-mon[103278]: pgmap v5186: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:49 smithi078 ceph-mon[91136]: pgmap v5186: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:49 smithi078 ceph-mon[99959]: pgmap v5186: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:51.033 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:12:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:12:51.033 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:50.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:51.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:50.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:12:51.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:12:50.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:12:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:51 smithi146 ceph-mon[103278]: pgmap v5187: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:51 smithi078 ceph-mon[91136]: pgmap v5187: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:51 smithi078 ceph-mon[99959]: pgmap v5187: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:53 smithi146 ceph-mon[103278]: pgmap v5188: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[91136]: pgmap v5188: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[99959]: pgmap v5188: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:55 smithi146 ceph-mon[103278]: pgmap v5189: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:55 smithi078 ceph-mon[91136]: pgmap v5189: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:55 smithi078 ceph-mon[99959]: pgmap v5189: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:57 smithi146 ceph-mon[103278]: pgmap v5190: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[91136]: pgmap v5190: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[99959]: pgmap v5190: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:57.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:12:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:12:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:12:59 smithi146 ceph-mon[103278]: pgmap v5191: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:12:59 smithi078 ceph-mon[91136]: pgmap v5191: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:12:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:12:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:12:59 smithi078 ceph-mon[99959]: pgmap v5191: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:00 smithi146 ceph-mon[103278]: pgmap v5192: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:00 smithi078 ceph-mon[91136]: pgmap v5192: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:13:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:13:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:00 smithi078 ceph-mon[99959]: pgmap v5192: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:00.848Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:00.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:00.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:03 smithi146 ceph-mon[103278]: pgmap v5193: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:03 smithi078 ceph-mon[99959]: pgmap v5193: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:03 smithi078 ceph-mon[91136]: pgmap v5193: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:05.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:05 smithi078 ceph-mon[99959]: pgmap v5194: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:05.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:05 smithi078 ceph-mon[91136]: pgmap v5194: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:05 smithi146 ceph-mon[103278]: pgmap v5194: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:07 smithi146 ceph-mon[103278]: pgmap v5195: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:07.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:07 smithi078 ceph-mon[91136]: pgmap v5195: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:07.780 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:07 smithi078 ceph-mon[99959]: pgmap v5195: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:09.042 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 3h - - 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 3h - - 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 17s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 12s ago 2h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:13:09.577 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 12s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 3h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 12s ago 3h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 3h 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 17s ago 2h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 12s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 12s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 12s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 12s ago 3h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:13:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 12s ago 2h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:13:09.579 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 12s ago 2h 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:13:09.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:13:09.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:13:09.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:13:09.579 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 12s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:13:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:09 smithi146 ceph-mon[103278]: pgmap v5196: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:09 smithi078 ceph-mon[91136]: pgmap v5196: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:09 smithi078 ceph-mon[99959]: pgmap v5196: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:09.887 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:13:09.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:09.889 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:13:09.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:13:09.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:13:09.889 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:13:10.156 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:13:10.157 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:13:10.157 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:13:10.157 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:13:10.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:13:10.465 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:13:10.466 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:10.466 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:10.466 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:10.466 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:10.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-mon[91136]: from='client.55815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:10.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-mon[91136]: from='client.55821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:10.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4156680426' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:10.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-mon[99959]: from='client.55815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:10.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-mon[99959]: from='client.55821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:10.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4156680426' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:10 smithi146 ceph-mon[103278]: from='client.55815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:10 smithi146 ceph-mon[103278]: from='client.55821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4156680426' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:13:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:13:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:10.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:10.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:10.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:11 smithi146 ceph-mon[103278]: from='client.75746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:11 smithi146 ceph-mon[103278]: from='client.75758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:11 smithi146 ceph-mon[103278]: pgmap v5197: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4152979030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[91136]: from='client.75746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[91136]: from='client.75758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[91136]: pgmap v5197: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4152979030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[99959]: from='client.75746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[99959]: from='client.75758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[99959]: pgmap v5197: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4152979030' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:13 smithi146 ceph-mon[103278]: pgmap v5198: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[91136]: pgmap v5198: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[99959]: pgmap v5198: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:15 smithi146 ceph-mon[103278]: pgmap v5199: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:15 smithi078 ceph-mon[91136]: pgmap v5199: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:15 smithi078 ceph-mon[99959]: pgmap v5199: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:17 smithi146 ceph-mon[103278]: pgmap v5200: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[91136]: pgmap v5200: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:17.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[99959]: pgmap v5200: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:17.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:17.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:19 smithi078 ceph-mon[91136]: pgmap v5201: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:19 smithi078 ceph-mon[99959]: pgmap v5201: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:19 smithi146 ceph-mon[103278]: pgmap v5201: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:13:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:13:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:20.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:20.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:20.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:21 smithi146 ceph-mon[103278]: pgmap v5202: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:21 smithi078 ceph-mon[91136]: pgmap v5202: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:21 smithi078 ceph-mon[99959]: pgmap v5202: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:23 smithi146 ceph-mon[103278]: pgmap v5203: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:23 smithi078 ceph-mon[91136]: pgmap v5203: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:23 smithi078 ceph-mon[99959]: pgmap v5203: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:25.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:25 smithi078 ceph-mon[99959]: pgmap v5204: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:25.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:25 smithi078 ceph-mon[91136]: pgmap v5204: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:25 smithi146 ceph-mon[103278]: pgmap v5204: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:27.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:27 smithi078 ceph-mon[91136]: pgmap v5205: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:27.583 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:27 smithi078 ceph-mon[99959]: pgmap v5205: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:27 smithi146 ceph-mon[103278]: pgmap v5205: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:28.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:29 smithi146 ceph-mon[103278]: pgmap v5206: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:29 smithi078 ceph-mon[91136]: pgmap v5206: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:29 smithi078 ceph-mon[99959]: pgmap v5206: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:30 smithi078 ceph-mon[91136]: pgmap v5207: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:30 smithi078 ceph-mon[99959]: pgmap v5207: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:30 smithi146 ceph-mon[103278]: pgmap v5207: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:13:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:13:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:30.850Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:30.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:30.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:33 smithi146 ceph-mon[103278]: pgmap v5208: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[91136]: pgmap v5208: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[99959]: pgmap v5208: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:35 smithi146 ceph-mon[103278]: pgmap v5209: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:35 smithi078 ceph-mon[91136]: pgmap v5209: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:35 smithi078 ceph-mon[99959]: pgmap v5209: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:37 smithi146 ceph-mon[103278]: pgmap v5210: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:37 smithi078 ceph-mon[91136]: pgmap v5210: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:37 smithi078 ceph-mon[99959]: pgmap v5210: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:39 smithi146 ceph-mon[103278]: pgmap v5211: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:39 smithi078 ceph-mon[91136]: pgmap v5211: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:39 smithi078 ceph-mon[99959]: pgmap v5211: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:40.763 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:13:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:13:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:13:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:40.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:41.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:40.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:41.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:40.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 3h - - 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 3h - - 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 8s ago 2h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 3s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 8s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:13:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 3s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 8s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 8s ago 3h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 3s ago 3h 206M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 8s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 8s ago 2h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 3s ago 2h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 8s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 8s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 8s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 8s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 3s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 3s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 3s ago 3h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 3s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:13:41.288 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 3s ago 2h 78.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:13:41.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 8s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:13:41.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:13:41.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 8s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:13:41.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 3s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:13:41.604 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:13:41.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:13:41.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:13:41.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:13:41.605 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:13:41.606 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:41 smithi078 ceph-mon[99959]: pgmap v5212: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:41.606 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:41 smithi078 ceph-mon[91136]: pgmap v5212: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:41 smithi146 ceph-mon[103278]: pgmap v5212: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:41.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:13:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:13:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:13:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:13:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:13:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:13:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:13:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:13:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:13:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:13:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:13:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:13:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:13:41.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:13:42.184 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:13:42.185 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:13:42.185 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[91136]: from='client.75770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[91136]: from='client.55857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[91136]: from='client.75782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2163839078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4171261266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[99959]: from='client.75770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[99959]: from='client.55857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[99959]: from='client.75782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2163839078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:42.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4171261266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:42 smithi146 ceph-mon[103278]: from='client.75770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:42 smithi146 ceph-mon[103278]: from='client.55857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:42 smithi146 ceph-mon[103278]: from='client.75782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2163839078' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:13:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4171261266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:13:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:43 smithi146 ceph-mon[103278]: from='client.55875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:43 smithi146 ceph-mon[103278]: pgmap v5213: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:43 smithi078 ceph-mon[91136]: from='client.55875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:43 smithi078 ceph-mon[91136]: pgmap v5213: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:43 smithi078 ceph-mon[99959]: from='client.55875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:13:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:43 smithi078 ceph-mon[99959]: pgmap v5213: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:44 smithi146 ceph-mon[103278]: pgmap v5214: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:44 smithi078 ceph-mon[91136]: pgmap v5214: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:44 smithi078 ceph-mon[99959]: pgmap v5214: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:47.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:47 smithi078 ceph-mon[91136]: pgmap v5215: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:47.664 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:47 smithi078 ceph-mon[99959]: pgmap v5215: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:47 smithi146 ceph-mon[103278]: pgmap v5215: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:49 smithi146 ceph-mon[103278]: pgmap v5216: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:49 smithi078 ceph-mon[91136]: pgmap v5216: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:49 smithi078 ceph-mon[99959]: pgmap v5216: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:13:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:13:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:50.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:50.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:13:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:13:50.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:13:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:51 smithi146 ceph-mon[103278]: pgmap v5217: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:51 smithi078 ceph-mon[91136]: pgmap v5217: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:51 smithi078 ceph-mon[99959]: pgmap v5217: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:53.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:53 smithi146 ceph-mon[103278]: pgmap v5218: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:53.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[91136]: pgmap v5218: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[99959]: pgmap v5218: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:53.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:55 smithi146 ceph-mon[103278]: pgmap v5219: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:55 smithi078 ceph-mon[91136]: pgmap v5219: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:55 smithi078 ceph-mon[99959]: pgmap v5219: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:57 smithi146 ceph-mon[103278]: pgmap v5220: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:57 smithi078 ceph-mon[91136]: pgmap v5220: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:57 smithi078 ceph-mon[99959]: pgmap v5220: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:13:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:13:58 smithi146 ceph-mon[103278]: pgmap v5221: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[91136]: pgmap v5221: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:13:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:13:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:13:58 smithi078 ceph-mon[99959]: pgmap v5221: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:14:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:14:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:00.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:00.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:01.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:00.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:01.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:01 smithi146 ceph-mon[103278]: pgmap v5222: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:01 smithi078 ceph-mon[91136]: pgmap v5222: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:01 smithi078 ceph-mon[99959]: pgmap v5222: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:02.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 23:14:01 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T23:14:01.720Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726863240607 maxt=1726869600000 ulid=01J88VDFAPC17SHSTVEZAEVGWR duration=33.138196ms 2024-09-20T23:14:02.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 23:14:01 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T23:14:01.723Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=2.551075ms 2024-09-20T23:14:02.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 23:14:01 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T23:14:01.723Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=1 mint=1726869600000 2024-09-20T23:14:02.017 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 20 23:14:01 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-20T23:14:01.727Z caller=head.go:1241 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=1 duration=4.79875ms 2024-09-20T23:14:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:03 smithi146 ceph-mon[103278]: pgmap v5223: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:03 smithi078 ceph-mon[91136]: pgmap v5223: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:03 smithi078 ceph-mon[99959]: pgmap v5223: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:05 smithi146 ceph-mon[103278]: pgmap v5224: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:05 smithi078 ceph-mon[91136]: pgmap v5224: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:05 smithi078 ceph-mon[99959]: pgmap v5224: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:07 smithi146 ceph-mon[103278]: pgmap v5225: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:07 smithi078 ceph-mon[91136]: pgmap v5225: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:07 smithi078 ceph-mon[99959]: pgmap v5225: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:09 smithi146 ceph-mon[103278]: pgmap v5226: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:09 smithi078 ceph-mon[91136]: pgmap v5226: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:09 smithi078 ceph-mon[99959]: pgmap v5226: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:14:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:14:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:10.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:10.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:10.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:11 smithi146 ceph-mon[103278]: pgmap v5227: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:11 smithi078 ceph-mon[91136]: pgmap v5227: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:11 smithi078 ceph-mon[99959]: pgmap v5227: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:12.487 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:14:13.017 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:14:13.017 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 20s ago 3h - - 2024-09-20T23:14:13.017 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 3h - - 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 20s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (2h) 15s ago 2h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 20s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 15s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 20s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 20s ago 3h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 15s ago 3h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 20s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (2h) 20s ago 2h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (2h) 15s ago 2h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 20s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 20s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 20s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:14:13.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 20s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 15s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 15s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 15s ago 3h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 15s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 15s ago 3h 79.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 20s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 20s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:14:13.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 15s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:13.337 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:14:13.338 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:14:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:14:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:14:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:14:13.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:14:13.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[91136]: pgmap v5228: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:13.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:13.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4146630233' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:13.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[99959]: pgmap v5228: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:13.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:13.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4146630233' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:13.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:13 smithi146 ceph-mon[103278]: pgmap v5228: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:13.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:13.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:13.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4146630233' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:13.920 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:14:13.920 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:13.920 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:14 smithi146 ceph-mon[103278]: from='client.75806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:14 smithi146 ceph-mon[103278]: from='client.55893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:14 smithi146 ceph-mon[103278]: from='client.55899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4232823779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[91136]: from='client.75806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[91136]: from='client.55893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[91136]: from='client.55899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4232823779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[99959]: from='client.75806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[99959]: from='client.55893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[99959]: from='client.55899 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:14.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4232823779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:15 smithi146 ceph-mon[103278]: from='client.75830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:15 smithi146 ceph-mon[103278]: pgmap v5229: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:15 smithi078 ceph-mon[99959]: from='client.75830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:15 smithi078 ceph-mon[99959]: pgmap v5229: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:15 smithi078 ceph-mon[91136]: from='client.75830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:15 smithi078 ceph-mon[91136]: pgmap v5229: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:17 smithi146 ceph-mon[103278]: pgmap v5230: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:17 smithi078 ceph-mon[91136]: pgmap v5230: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:17 smithi078 ceph-mon[99959]: pgmap v5230: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:18 smithi146 ceph-mon[103278]: pgmap v5231: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[99959]: pgmap v5231: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:19.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:18 smithi078 ceph-mon[91136]: pgmap v5231: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:14:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:14:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:20.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:20.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:20.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:21 smithi146 ceph-mon[103278]: pgmap v5232: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:21 smithi078 ceph-mon[91136]: pgmap v5232: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:21 smithi078 ceph-mon[99959]: pgmap v5232: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:23 smithi146 ceph-mon[103278]: pgmap v5233: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:23 smithi078 ceph-mon[91136]: pgmap v5233: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:23 smithi078 ceph-mon[99959]: pgmap v5233: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:25 smithi146 ceph-mon[103278]: pgmap v5234: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:25 smithi078 ceph-mon[91136]: pgmap v5234: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:25 smithi078 ceph-mon[99959]: pgmap v5234: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 23:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T23:14:25.847702413Z level=info msg="Completed cleanup jobs" duration=11.871863ms 2024-09-20T23:14:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:27 smithi146 ceph-mon[103278]: pgmap v5235: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:27 smithi078 ceph-mon[91136]: pgmap v5235: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:27 smithi078 ceph-mon[99959]: pgmap v5235: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:29 smithi078 ceph-mon[91136]: pgmap v5236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:29 smithi078 ceph-mon[99959]: pgmap v5236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:29 smithi146 ceph-mon[103278]: pgmap v5236: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:14:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:14:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:30.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:30.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:30.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:31 smithi146 ceph-mon[103278]: pgmap v5237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:31 smithi078 ceph-mon[91136]: pgmap v5237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:31 smithi078 ceph-mon[99959]: pgmap v5237: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:33.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:33 smithi146 ceph-mon[103278]: pgmap v5238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:33.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[91136]: pgmap v5238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[99959]: pgmap v5238: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:33.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:35.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:35 smithi146 ceph-mon[103278]: pgmap v5239: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:35 smithi078 ceph-mon[91136]: pgmap v5239: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:35 smithi078 ceph-mon[99959]: pgmap v5239: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:36 smithi146 ceph-mon[103278]: pgmap v5240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:36 smithi078 ceph-mon[91136]: pgmap v5240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:36 smithi078 ceph-mon[99959]: pgmap v5240: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:38 smithi146 ceph-mon[103278]: pgmap v5241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[99959]: pgmap v5241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:39.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:38 smithi078 ceph-mon[91136]: pgmap v5241: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:40.956 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:14:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:14:40.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:40.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:40.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:40.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:40.957 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:40.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:41 smithi146 ceph-mon[103278]: pgmap v5242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:41 smithi078 ceph-mon[91136]: pgmap v5242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:41 smithi078 ceph-mon[99959]: pgmap v5242: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:43 smithi146 ceph-mon[103278]: pgmap v5243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:43 smithi078 ceph-mon[91136]: pgmap v5243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:43 smithi078 ceph-mon[99959]: pgmap v5243: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 3h - - 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 3h - - 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (2h) 11s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 6s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 11s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 6s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 11s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 11s ago 3h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 6s ago 3h 208M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:14:44.744 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 11s ago 3h 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 11s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 6s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 11s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 11s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 11s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 11s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 6s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 6s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 6s ago 3h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 6s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 6s ago 3h 97.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 11s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 11s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:14:44.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 6s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:14:45.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:14:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:14:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:14:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:14:45.057 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:14:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:14:45.323 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:14:45.633 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:14:45.633 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:14:45.633 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:14:45.634 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:45 smithi078 ceph-mon[99959]: from='client.75842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:45.634 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:45 smithi078 ceph-mon[99959]: pgmap v5244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:45.634 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3403151135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:45.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:45 smithi078 ceph-mon[91136]: from='client.75842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:45.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:45 smithi078 ceph-mon[91136]: pgmap v5244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:45.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3403151135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:45 smithi146 ceph-mon[103278]: from='client.75842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:45 smithi146 ceph-mon[103278]: pgmap v5244: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3403151135' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:14:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:46 smithi146 ceph-mon[103278]: from='client.75848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:46 smithi146 ceph-mon[103278]: from='client.55935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:46 smithi146 ceph-mon[103278]: from='client.75866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/628152382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[91136]: from='client.75848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[91136]: from='client.55935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[91136]: from='client.75866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/628152382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[99959]: from='client.75848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[99959]: from='client.55935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[99959]: from='client.75866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:14:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/628152382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:14:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:47 smithi146 ceph-mon[103278]: pgmap v5245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:47 smithi078 ceph-mon[91136]: pgmap v5245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:47 smithi078 ceph-mon[99959]: pgmap v5245: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:49.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:49.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:49 smithi078 ceph-mon[91136]: pgmap v5246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:49.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:49.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:49 smithi078 ceph-mon[99959]: pgmap v5246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:49 smithi146 ceph-mon[103278]: pgmap v5246: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:50.857 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:14:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:14:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:50.855Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:50.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:14:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:14:50.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:51 smithi146 ceph-mon[103278]: pgmap v5247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:51 smithi078 ceph-mon[91136]: pgmap v5247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:51 smithi078 ceph-mon[99959]: pgmap v5247: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:53 smithi146 ceph-mon[103278]: pgmap v5248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[91136]: pgmap v5248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[99959]: pgmap v5248: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:54 smithi146 ceph-mon[103278]: pgmap v5249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:54 smithi078 ceph-mon[91136]: pgmap v5249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:54 smithi078 ceph-mon[99959]: pgmap v5249: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:57 smithi146 ceph-mon[103278]: pgmap v5250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:57 smithi078 ceph-mon[91136]: pgmap v5250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:57 smithi078 ceph-mon[99959]: pgmap v5250: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:14:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:14:58 smithi146 ceph-mon[103278]: pgmap v5251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[91136]: pgmap v5251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:14:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:14:58 smithi078 ceph-mon[99959]: pgmap v5251: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:01.009 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:15:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:15:01.009 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:00.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:01.010 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:00.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:01.010 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:00.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:01 smithi146 ceph-mon[103278]: pgmap v5252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:01 smithi078 ceph-mon[91136]: pgmap v5252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:01 smithi078 ceph-mon[99959]: pgmap v5252: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:03 smithi146 ceph-mon[103278]: pgmap v5253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:15:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:03 smithi078 ceph-mon[99959]: pgmap v5253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:15:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:03 smithi078 ceph-mon[91136]: pgmap v5253: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:15:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:05 smithi146 ceph-mon[103278]: pgmap v5254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:05 smithi078 ceph-mon[99959]: pgmap v5254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:05 smithi078 ceph-mon[91136]: pgmap v5254: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:07 smithi146 ceph-mon[103278]: pgmap v5255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:15:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:07 smithi078 ceph-mon[91136]: pgmap v5255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:15:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:07 smithi078 ceph-mon[99959]: pgmap v5255: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:15:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:09 smithi146 ceph-mon[103278]: pgmap v5256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:09.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:09.767 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:09 smithi078 ceph-mon[99959]: pgmap v5256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:09.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:09.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:09 smithi078 ceph-mon[91136]: pgmap v5256: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:15:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:15:11.238 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:10.857Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:11.238 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:10.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:11.238 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:10.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:11 smithi078 ceph-mon[99959]: pgmap v5257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:11 smithi078 ceph-mon[91136]: pgmap v5257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:11 smithi146 ceph-mon[103278]: pgmap v5257: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:12 smithi146 ceph-mon[103278]: pgmap v5258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:12 smithi078 ceph-mon[91136]: pgmap v5258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:12 smithi078 ceph-mon[99959]: pgmap v5258: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:14.694 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:14 smithi146 ceph-mon[103278]: pgmap v5259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:14 smithi078 ceph-mon[91136]: pgmap v5259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:14 smithi078 ceph-mon[99959]: pgmap v5259: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:15.933 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:15:16.461 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:15:16.461 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 3h - - 2024-09-20T23:15:16.461 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 3h - - 2024-09-20T23:15:16.461 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 3s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:15:16.461 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 18s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 3s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 18s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 3s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 3s ago 3h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 18s ago 3h 209M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 3s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 3s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 18s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 3s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 3s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:15:16.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 3s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 3s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 18s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 18s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 18s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 18s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 18s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 3s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 3s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:15:16.463 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 18s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:15:16.770 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:15:17.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:15:17.036 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:15:17.347 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:15:17.347 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:17.347 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:17.347 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:17.347 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:17 smithi146 ceph-mon[103278]: from='client.55959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:17 smithi146 ceph-mon[103278]: from='client.75884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:17 smithi146 ceph-mon[103278]: pgmap v5260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4016827586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3416734560' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[91136]: from='client.55959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[91136]: from='client.75884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[91136]: pgmap v5260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4016827586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3416734560' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[99959]: from='client.55959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[99959]: from='client.75884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[99959]: pgmap v5260: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4016827586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3416734560' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:18 smithi146 ceph-mon[103278]: from='client.75890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:18 smithi146 ceph-mon[103278]: from='client.55983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[91136]: from='client.75890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[91136]: from='client.55983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[99959]: from='client.75890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[99959]: from='client.55983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:19.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:19.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:19 smithi078 ceph-mon[91136]: pgmap v5261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:19.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:19.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:19 smithi078 ceph-mon[99959]: pgmap v5261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:19 smithi146 ceph-mon[103278]: pgmap v5261: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:20.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:15:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:15:20.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:20.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:20.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:20.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:20.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:20.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:21 smithi146 ceph-mon[103278]: pgmap v5262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:21 smithi078 ceph-mon[99959]: pgmap v5262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:21 smithi078 ceph-mon[91136]: pgmap v5262: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:23 smithi146 ceph-mon[103278]: pgmap v5263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:23 smithi078 ceph-mon[99959]: pgmap v5263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:23 smithi078 ceph-mon[91136]: pgmap v5263: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:25 smithi146 ceph-mon[103278]: pgmap v5264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:25 smithi078 ceph-mon[91136]: pgmap v5264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:25 smithi078 ceph-mon[99959]: pgmap v5264: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:27 smithi146 ceph-mon[103278]: pgmap v5265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:27 smithi078 ceph-mon[91136]: pgmap v5265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:27 smithi078 ceph-mon[99959]: pgmap v5265: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:29.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:29.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:29 smithi078 ceph-mon[99959]: pgmap v5266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:29.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:29.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:29 smithi078 ceph-mon[91136]: pgmap v5266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:29 smithi146 ceph-mon[103278]: pgmap v5266: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:30 smithi146 ceph-mon[103278]: pgmap v5267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:30 smithi078 ceph-mon[91136]: pgmap v5267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:15:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:15:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:30 smithi078 ceph-mon[99959]: pgmap v5267: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:31.141 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:30.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:31.141 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:30.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:31.141 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:30.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:33 smithi146 ceph-mon[103278]: pgmap v5268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:33 smithi078 ceph-mon[91136]: pgmap v5268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:33 smithi078 ceph-mon[99959]: pgmap v5268: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:34.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:34.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:34.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:34.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:34 smithi146 ceph-mon[103278]: pgmap v5269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[91136]: pgmap v5269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:34 smithi078 ceph-mon[99959]: pgmap v5269: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:37 smithi146 ceph-mon[103278]: pgmap v5270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:37 smithi078 ceph-mon[91136]: pgmap v5270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:37 smithi078 ceph-mon[99959]: pgmap v5270: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:39 smithi146 ceph-mon[103278]: pgmap v5271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[91136]: pgmap v5271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:39 smithi078 ceph-mon[99959]: pgmap v5271: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:40.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:15:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:15:40.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:40.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:40.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:40.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:40.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:41 smithi146 ceph-mon[103278]: pgmap v5272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:41.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:41 smithi078 ceph-mon[91136]: pgmap v5272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:41.835 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:41 smithi078 ceph-mon[99959]: pgmap v5272: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:43 smithi146 ceph-mon[103278]: pgmap v5273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:43 smithi078 ceph-mon[99959]: pgmap v5273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:43 smithi078 ceph-mon[91136]: pgmap v5273: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:45 smithi146 ceph-mon[103278]: pgmap v5274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:45 smithi078 ceph-mon[99959]: pgmap v5274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:45 smithi078 ceph-mon[91136]: pgmap v5274: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:46 smithi146 ceph-mon[103278]: pgmap v5275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:46 smithi078 ceph-mon[91136]: pgmap v5275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:46 smithi078 ceph-mon[99959]: pgmap v5275: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:47.639 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 3h - - 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 3h - - 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 14s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 9s ago 3h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 14s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 9s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 14s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:15:48.166 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 14s ago 3h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 9s ago 3h 207M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 14s ago 3h 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 14s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 9s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 14s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 14s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 14s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 14s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 9s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 9s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 9s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 9s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 9s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:15:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 2h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:15:48.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:15:48.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 14s ago 2h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:15:48.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 9s ago 2h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:15:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:15:48.483 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:15:48.483 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:15:48.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:15:48.483 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:15:48.749 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:15:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:15:48.751 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:15:49.059 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:15:49.060 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:15:49.060 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:49.060 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: from='client.55995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: from='client.75920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: from='client.56007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: pgmap v5276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3055973105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1265238581' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: from='client.55995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: from='client.75920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: from='client.56007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: pgmap v5276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3055973105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1265238581' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: from='client.55995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: from='client.75920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: from='client.56007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: pgmap v5276: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3055973105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:15:49.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1265238581' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:15:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:50 smithi146 ceph-mon[103278]: from='client.56019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:50.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:50 smithi078 ceph-mon[91136]: from='client.56019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:50.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:50 smithi078 ceph-mon[99959]: from='client.56019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:15:50.777 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:15:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:15:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:50.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:50.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:15:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:15:50.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:15:51.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:51 smithi078 ceph-mon[91136]: pgmap v5277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:51.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:51 smithi078 ceph-mon[99959]: pgmap v5277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:51 smithi146 ceph-mon[103278]: pgmap v5277: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:52.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:52 smithi078 ceph-mon[91136]: pgmap v5278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:52.766 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:52 smithi078 ceph-mon[99959]: pgmap v5278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:52 smithi146 ceph-mon[103278]: pgmap v5278: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:54 smithi146 ceph-mon[103278]: pgmap v5279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[91136]: pgmap v5279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:54 smithi078 ceph-mon[99959]: pgmap v5279: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:57.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:57 smithi146 ceph-mon[103278]: pgmap v5280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:57 smithi078 ceph-mon[91136]: pgmap v5280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:57 smithi078 ceph-mon[99959]: pgmap v5280: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:15:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:15:59 smithi146 ceph-mon[103278]: pgmap v5281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[99959]: pgmap v5281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:15:59.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:15:59.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:15:59 smithi078 ceph-mon[91136]: pgmap v5281: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:00.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:16:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:16:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:00.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:00.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:00.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:01 smithi146 ceph-mon[103278]: pgmap v5282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:01 smithi078 ceph-mon[91136]: pgmap v5282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:01 smithi078 ceph-mon[99959]: pgmap v5282: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:03 smithi146 ceph-mon[103278]: pgmap v5283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:03 smithi078 ceph-mon[91136]: pgmap v5283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:03 smithi078 ceph-mon[99959]: pgmap v5283: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:04 smithi146 ceph-mon[103278]: pgmap v5284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:04 smithi078 ceph-mon[91136]: pgmap v5284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:04 smithi078 ceph-mon[99959]: pgmap v5284: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:07 smithi146 ceph-mon[103278]: pgmap v5285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:07 smithi078 ceph-mon[99959]: pgmap v5285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:07 smithi078 ceph-mon[91136]: pgmap v5285: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:08 smithi146 ceph-mon[103278]: pgmap v5286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:08 smithi078 ceph-mon[91136]: pgmap v5286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:08 smithi078 ceph-mon[99959]: pgmap v5286: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:10.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:16:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:16:11.162 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:10.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:11.162 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:10.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:11.162 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:10.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:11 smithi146 ceph-mon[103278]: pgmap v5287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:11 smithi078 ceph-mon[91136]: pgmap v5287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:11 smithi078 ceph-mon[99959]: pgmap v5287: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:12 smithi146 ceph-mon[103278]: pgmap v5288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:12 smithi078 ceph-mon[99959]: pgmap v5288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:12 smithi078 ceph-mon[91136]: pgmap v5288: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:14 smithi146 ceph-mon[103278]: pgmap v5289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[99959]: pgmap v5289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:14 smithi078 ceph-mon[91136]: pgmap v5289: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:17 smithi146 ceph-mon[103278]: pgmap v5290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:17 smithi078 ceph-mon[91136]: pgmap v5290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:17 smithi078 ceph-mon[99959]: pgmap v5290: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:19.356 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:16:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:19 smithi146 ceph-mon[103278]: pgmap v5291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[91136]: pgmap v5291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[99959]: pgmap v5291: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:19.885 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:16:19.885 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 3h - - 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 3h - - 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 6s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 1s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 6s ago 2h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 1s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 6s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 6s ago 3h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 1s ago 3h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 6s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 6s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 1s ago 3h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 6s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 6s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:16:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 6s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 6s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 1s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 1s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 1s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 1s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 1s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 6s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 6s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:16:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:16:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:16:20.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:16:20.212 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:16:20.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:16:20.494 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:16:20.495 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:16:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:20 smithi146 ceph-mon[103278]: from='client.75950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1246481592' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-mon[99959]: from='client.75950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1246481592' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:16:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:16:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-mon[91136]: from='client.75950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1246481592' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:20.814 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:16:20.815 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:20.815 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:20.815 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:20.815 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:20.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:20.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:20.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:21 smithi146 ceph-mon[103278]: from='client.56037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:21 smithi146 ceph-mon[103278]: from='client.56043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:21 smithi146 ceph-mon[103278]: pgmap v5292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1662682792' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[91136]: from='client.56037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[91136]: from='client.56043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[91136]: pgmap v5292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1662682792' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[99959]: from='client.56037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[99959]: from='client.56043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[99959]: pgmap v5292: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1662682792' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:22 smithi146 ceph-mon[103278]: from='client.75974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:22 smithi078 ceph-mon[99959]: from='client.75974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:22 smithi078 ceph-mon[91136]: from='client.75974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:23 smithi146 ceph-mon[103278]: pgmap v5293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:23 smithi078 ceph-mon[99959]: pgmap v5293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:23 smithi078 ceph-mon[91136]: pgmap v5293: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:25 smithi146 ceph-mon[103278]: pgmap v5294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:25 smithi078 ceph-mon[91136]: pgmap v5294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:25 smithi078 ceph-mon[99959]: pgmap v5294: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:27 smithi146 ceph-mon[103278]: pgmap v5295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:27 smithi078 ceph-mon[91136]: pgmap v5295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:27 smithi078 ceph-mon[99959]: pgmap v5295: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:29 smithi146 ceph-mon[103278]: pgmap v5296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:29 smithi078 ceph-mon[99959]: pgmap v5296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:29 smithi078 ceph-mon[91136]: pgmap v5296: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:30 smithi146 ceph-mon[103278]: pgmap v5297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:30 smithi078 ceph-mon[91136]: pgmap v5297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:30 smithi078 ceph-mon[99959]: pgmap v5297: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:16:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:16:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:30.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:30.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:30.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:32 smithi146 ceph-mon[103278]: pgmap v5298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:32 smithi078 ceph-mon[91136]: pgmap v5298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:32 smithi078 ceph-mon[99959]: pgmap v5298: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:35.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.263 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[91136]: pgmap v5299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:35.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.263 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:34 smithi078 ceph-mon[99959]: pgmap v5299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:34 smithi146 ceph-mon[103278]: pgmap v5299: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:36 smithi146 ceph-mon[103278]: pgmap v5300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:36 smithi078 ceph-mon[91136]: pgmap v5300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:36 smithi078 ceph-mon[99959]: pgmap v5300: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:16:38.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:38.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:38 smithi146 ceph-mon[103278]: pgmap v5301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:38 smithi078 ceph-mon[91136]: pgmap v5301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:38 smithi078 ceph-mon[99959]: pgmap v5301: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:16:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:40.864 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:40 smithi078 ceph-mon[99959]: pgmap v5302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:16:40.864 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:16:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:16:40.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:40 smithi078 ceph-mon[91136]: pgmap v5302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:16:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:40 smithi146 ceph-mon[103278]: pgmap v5302: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:16:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:40.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:40.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:40.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:42 smithi146 ceph-mon[103278]: pgmap v5303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:16:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:42 smithi078 ceph-mon[91136]: pgmap v5303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:16:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:42 smithi078 ceph-mon[99959]: pgmap v5303: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:16:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:44 smithi078 ceph-mon[91136]: pgmap v5304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:16:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:44 smithi078 ceph-mon[99959]: pgmap v5304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:16:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:44 smithi146 ceph-mon[103278]: pgmap v5304: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:16:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:46 smithi146 ceph-mon[103278]: pgmap v5305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:16:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:46 smithi078 ceph-mon[99959]: pgmap v5305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:16:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:46 smithi078 ceph-mon[91136]: pgmap v5305: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:16:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:48 smithi146 ceph-mon[103278]: pgmap v5306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:48 smithi078 ceph-mon[91136]: pgmap v5306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:48 smithi078 ceph-mon[99959]: pgmap v5306: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:50 smithi146 ceph-mon[103278]: pgmap v5307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T23:16:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:50 smithi078 ceph-mon[91136]: pgmap v5307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T23:16:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:50 smithi078 ceph-mon[99959]: pgmap v5307: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T23:16:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:16:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:16:51.124 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:50.864Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:51.124 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:50.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:16:51.124 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:16:50.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:16:51.124 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:16:51.662 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 3h - - 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 3h - - 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 17s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 13s ago 3h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 17s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 13s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 17s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 17s ago 3h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 13s ago 3h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 17s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 17s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 13s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:16:51.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 17s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 17s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 17s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 17s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 13s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 13s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 13s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 13s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 13s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:16:51.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:16:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:51 smithi146 ceph-mon[103278]: from='client.75986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:51 smithi146 ceph-mon[103278]: from='client.56073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:51 smithi078 ceph-mon[99959]: from='client.75986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:51 smithi078 ceph-mon[99959]: from='client.56073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:51 smithi078 ceph-mon[91136]: from='client.75986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:51 smithi078 ceph-mon[91136]: from='client.56073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:51.991 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:16:51.992 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:16:51.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:16:51.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:16:51.993 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:16:52.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:16:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:16:52.262 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:16:52.579 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:16:52.579 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:52.580 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:16:52.580 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:52.580 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[99959]: from='client.75998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1191017216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[99959]: from='client.56091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[99959]: pgmap v5308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[91136]: from='client.75998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1191017216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[91136]: from='client.56091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:52 smithi078 ceph-mon[91136]: pgmap v5308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:16:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:52 smithi146 ceph-mon[103278]: from='client.75998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1191017216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:16:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:52 smithi146 ceph-mon[103278]: from='client.56091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:16:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:52 smithi146 ceph-mon[103278]: pgmap v5308: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:16:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4134074095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4134074095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4134074095' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:16:55.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.054 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[99959]: pgmap v5309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:55.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:55 smithi078 ceph-mon[91136]: pgmap v5309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:55 smithi146 ceph-mon[103278]: pgmap v5309: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:56 smithi146 ceph-mon[103278]: pgmap v5310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T23:16:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:56 smithi078 ceph-mon[91136]: pgmap v5310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T23:16:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:56 smithi078 ceph-mon[99959]: pgmap v5310: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-20T23:16:58.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:16:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:59 smithi146 ceph-mon[103278]: pgmap v5311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[91136]: pgmap v5311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:16:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[99959]: pgmap v5311: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T23:16:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:16:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:00 smithi146 ceph-mon[103278]: pgmap v5312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:17:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:00 smithi078 ceph-mon[91136]: pgmap v5312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:17:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:00 smithi078 ceph-mon[99959]: pgmap v5312: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T23:17:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:17:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:17:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:00.864Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:00.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:00.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:02 smithi146 ceph-mon[103278]: pgmap v5313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:17:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:02 smithi078 ceph-mon[99959]: pgmap v5313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:17:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:02 smithi078 ceph-mon[91136]: pgmap v5313: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:17:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:04 smithi146 ceph-mon[103278]: pgmap v5314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:17:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:04 smithi078 ceph-mon[91136]: pgmap v5314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:17:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:04 smithi078 ceph-mon[99959]: pgmap v5314: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T23:17:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:06 smithi146 ceph-mon[103278]: pgmap v5315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:17:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:06 smithi078 ceph-mon[91136]: pgmap v5315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:17:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:06 smithi078 ceph-mon[99959]: pgmap v5315: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T23:17:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:08 smithi078 ceph-mon[99959]: pgmap v5316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:08 smithi078 ceph-mon[91136]: pgmap v5316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:08 smithi146 ceph-mon[103278]: pgmap v5316: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:10 smithi146 ceph-mon[103278]: pgmap v5317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:10 smithi078 ceph-mon[99959]: pgmap v5317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:10 smithi078 ceph-mon[91136]: pgmap v5317: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:17:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:17:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:10.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:10.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:10.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:12 smithi146 ceph-mon[103278]: pgmap v5318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:12 smithi078 ceph-mon[91136]: pgmap v5318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:12 smithi078 ceph-mon[99959]: pgmap v5318: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:15.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[91136]: pgmap v5319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:15.315 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.316 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:15 smithi078 ceph-mon[99959]: pgmap v5319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:15 smithi146 ceph-mon[103278]: pgmap v5319: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:16 smithi078 ceph-mon[91136]: pgmap v5320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:16 smithi078 ceph-mon[99959]: pgmap v5320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:16 smithi146 ceph-mon[103278]: pgmap v5320: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:18.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:18.610 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:18 smithi146 ceph-mon[103278]: pgmap v5321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:18 smithi078 ceph-mon[91136]: pgmap v5321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:18 smithi078 ceph-mon[99959]: pgmap v5321: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:20.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:20 smithi078 ceph-mon[91136]: pgmap v5322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:20.867 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:20 smithi078 ceph-mon[99959]: pgmap v5322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:20.867 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:17:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:17:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:20 smithi146 ceph-mon[103278]: pgmap v5322: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:20.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:20.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:20.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:22 smithi146 ceph-mon[103278]: pgmap v5323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:22 smithi078 ceph-mon[91136]: pgmap v5323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:22 smithi078 ceph-mon[99959]: pgmap v5323: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:22.873 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 3h - - 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 3h - - 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 9s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 4s ago 3h 88.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 9s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 4s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 9s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:17:23.400 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 9s ago 3h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 4s ago 3h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 9s ago 3h 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 9s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 4s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 9s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 9s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 9s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 9s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 4s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 4s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 4s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:17:23.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 4s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:17:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 4s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:17:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 9s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:17:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:17:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 9s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:17:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:23.712 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:17:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:17:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:17:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:17:23.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:17:23.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:17:23.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:23 smithi078 ceph-mon[99959]: from='client.56103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:23.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:23 smithi078 ceph-mon[99959]: from='client.56109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:23.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:23 smithi078 ceph-mon[99959]: from='client.76034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:23.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:23 smithi078 ceph-mon[91136]: from='client.56103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:23.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:23 smithi078 ceph-mon[91136]: from='client.56109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:23.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:23 smithi078 ceph-mon[91136]: from='client.76034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:23.985 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:17:23.985 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:17:23.985 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:17:23.985 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:17:23.985 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:17:23.986 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:17:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:23 smithi146 ceph-mon[103278]: from='client.56103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:23 smithi146 ceph-mon[103278]: from='client.56109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:23 smithi146 ceph-mon[103278]: from='client.76034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:24.292 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:17:24.292 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:24.292 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:24.292 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/179829616' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[91136]: from='client.76046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/444269501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[91136]: pgmap v5324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/179829616' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[99959]: from='client.76046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/444269501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:24.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:24 smithi078 ceph-mon[99959]: pgmap v5324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/179829616' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:24 smithi146 ceph-mon[103278]: from='client.76046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/444269501' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:24 smithi146 ceph-mon[103278]: pgmap v5324: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:26 smithi078 ceph-mon[91136]: pgmap v5325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:26 smithi078 ceph-mon[99959]: pgmap v5325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:26 smithi146 ceph-mon[103278]: pgmap v5325: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:29 smithi146 ceph-mon[103278]: pgmap v5326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:29 smithi078 ceph-mon[91136]: pgmap v5326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:29 smithi078 ceph-mon[99959]: pgmap v5326: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:30 smithi078 ceph-mon[91136]: pgmap v5327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:30 smithi078 ceph-mon[99959]: pgmap v5327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:30.867 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:17:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:17:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:30 smithi146 ceph-mon[103278]: pgmap v5327: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:30.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:30.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:30.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:32 smithi078 ceph-mon[91136]: pgmap v5328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:32 smithi078 ceph-mon[99959]: pgmap v5328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:32 smithi146 ceph-mon[103278]: pgmap v5328: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[99959]: pgmap v5329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:35 smithi078 ceph-mon[91136]: pgmap v5329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:35 smithi146 ceph-mon[103278]: pgmap v5329: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:36 smithi078 ceph-mon[91136]: pgmap v5330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:36 smithi078 ceph-mon[99959]: pgmap v5330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:36 smithi146 ceph-mon[103278]: pgmap v5330: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:38.602 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:38.602 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:38 smithi146 ceph-mon[103278]: pgmap v5331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:38 smithi078 ceph-mon[99959]: pgmap v5331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:38 smithi078 ceph-mon[91136]: pgmap v5331: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:40.868 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:17:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:17:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:40 smithi146 ceph-mon[103278]: pgmap v5332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:40 smithi078 ceph-mon[99959]: pgmap v5332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:40 smithi078 ceph-mon[91136]: pgmap v5332: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:40.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:40.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:40.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:42 smithi078 ceph-mon[99959]: pgmap v5333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:42 smithi078 ceph-mon[91136]: pgmap v5333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:42 smithi146 ceph-mon[103278]: pgmap v5333: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:44 smithi078 ceph-mon[91136]: pgmap v5334: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:44 smithi078 ceph-mon[99959]: pgmap v5334: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:44 smithi146 ceph-mon[103278]: pgmap v5334: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:46 smithi078 ceph-mon[91136]: pgmap v5335: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:46 smithi078 ceph-mon[99959]: pgmap v5335: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:46 smithi146 ceph-mon[103278]: pgmap v5335: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:17:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:48 smithi078 ceph-mon[91136]: pgmap v5336: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:48 smithi078 ceph-mon[99959]: pgmap v5336: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:48 smithi146 ceph-mon[103278]: pgmap v5336: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:50 smithi078 ceph-mon[99959]: pgmap v5337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:50 smithi078 ceph-mon[91136]: pgmap v5337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:17:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:17:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:50 smithi146 ceph-mon[103278]: pgmap v5337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:50.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:50.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:17:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:17:50.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:52 smithi078 ceph-mon[91136]: pgmap v5338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:52 smithi078 ceph-mon[99959]: pgmap v5338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:52 smithi146 ceph-mon[103278]: pgmap v5338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:54.605 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 3h - - 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 3h - - 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 0s ago 3h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 16s ago 3h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 0s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 16s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 0s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:17:55.158 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 0s ago 3h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 16s ago 3h 210M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 0s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 0s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 16s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 0s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 0s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 0s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 0s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 16s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 16s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 16s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 16s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 16s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 0s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:17:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:17:55.160 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 0s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:17:55.160 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:17:55.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:17:55.486 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[99959]: pgmap v5339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:55 smithi078 ceph-mon[91136]: pgmap v5339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:55.763 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:17:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:17:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:17:55.763 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:17:55.764 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:17:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:55 smithi146 ceph-mon[103278]: pgmap v5339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:56.084 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:17:56.084 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:56.084 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:17:56.084 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:56.084 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:17:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:56 smithi146 ceph-mon[103278]: from='client.76058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:56 smithi146 ceph-mon[103278]: from='client.56145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:56 smithi146 ceph-mon[103278]: from='client.56151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/126617237' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3603889354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[91136]: from='client.76058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[91136]: from='client.56145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[91136]: from='client.56151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/126617237' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3603889354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[99959]: from='client.76058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[99959]: from='client.56145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[99959]: from='client.56151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:56.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/126617237' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:17:56.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3603889354' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:17:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:57 smithi078 ceph-mon[91136]: from='client.56163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:57 smithi078 ceph-mon[91136]: pgmap v5340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:57 smithi078 ceph-mon[99959]: from='client.56163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:57 smithi078 ceph-mon[99959]: pgmap v5340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:57 smithi146 ceph-mon[103278]: from='client.56163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:17:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:57 smithi146 ceph-mon[103278]: pgmap v5340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:17:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:17:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:59 smithi146 ceph-mon[103278]: pgmap v5341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:17:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[91136]: pgmap v5341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[99959]: pgmap v5341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:17:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:17:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:00.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:18:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:18:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:00.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:00.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:00.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:01 smithi146 ceph-mon[103278]: pgmap v5342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:01 smithi078 ceph-mon[91136]: pgmap v5342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:01 smithi078 ceph-mon[99959]: pgmap v5342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:03 smithi146 ceph-mon[103278]: pgmap v5343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:03 smithi078 ceph-mon[91136]: pgmap v5343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:03 smithi078 ceph-mon[99959]: pgmap v5343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:05 smithi146 ceph-mon[103278]: pgmap v5344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:05 smithi078 ceph-mon[91136]: pgmap v5344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:05 smithi078 ceph-mon[99959]: pgmap v5344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:07 smithi146 ceph-mon[103278]: pgmap v5345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:07 smithi078 ceph-mon[99959]: pgmap v5345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:07 smithi078 ceph-mon[91136]: pgmap v5345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:09 smithi146 ceph-mon[103278]: pgmap v5346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:09 smithi078 ceph-mon[91136]: pgmap v5346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:09 smithi078 ceph-mon[99959]: pgmap v5346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:10 smithi146 ceph-mon[103278]: pgmap v5347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:10 smithi078 ceph-mon[91136]: pgmap v5347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:10 smithi078 ceph-mon[99959]: pgmap v5347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:18:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:18:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:10.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:10.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:10.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:12 smithi146 ceph-mon[103278]: pgmap v5348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:18:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:12 smithi078 ceph-mon[91136]: pgmap v5348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:18:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:12 smithi078 ceph-mon[99959]: pgmap v5348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:18:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:14 smithi078 ceph-mon[91136]: pgmap v5349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:14 smithi078 ceph-mon[99959]: pgmap v5349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:14 smithi146 ceph-mon[103278]: pgmap v5349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:15.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:15.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:15.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:15.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:15.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:15.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:16.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:16 smithi078 ceph-mon[91136]: pgmap v5350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:16.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:16 smithi078 ceph-mon[99959]: pgmap v5350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:16 smithi146 ceph-mon[103278]: pgmap v5350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:18 smithi078 ceph-mon[91136]: pgmap v5351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:18 smithi078 ceph-mon[99959]: pgmap v5351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:18 smithi146 ceph-mon[103278]: pgmap v5351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:20.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:20.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:18:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:18:21.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:20.870Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:21.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:20.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:21.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:20.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:21 smithi146 ceph-mon[103278]: pgmap v5352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:21 smithi078 ceph-mon[99959]: pgmap v5352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:21 smithi078 ceph-mon[91136]: pgmap v5352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:22 smithi078 ceph-mon[91136]: pgmap v5353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:22 smithi078 ceph-mon[99959]: pgmap v5353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:22 smithi146 ceph-mon[103278]: pgmap v5353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:24 smithi078 ceph-mon[91136]: pgmap v5354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:24 smithi078 ceph-mon[99959]: pgmap v5354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:24 smithi146 ceph-mon[103278]: pgmap v5354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:26.380 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:18:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:26 smithi078 ceph-mon[91136]: from='client.56175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:26 smithi078 ceph-mon[91136]: pgmap v5355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:26 smithi078 ceph-mon[99959]: from='client.56175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:26 smithi078 ceph-mon[99959]: pgmap v5355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 3h - - 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 3h - - 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 12s ago 3h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 7s ago 3h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 12s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 7s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 12s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 12s ago 3h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:18:26.912 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 7s ago 3h 211M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 12s ago 3h 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 12s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 7s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 12s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 12s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 12s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 12s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 7s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 7s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 7s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 7s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 7s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 12s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 12s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:18:26.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:18:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:26 smithi146 ceph-mon[103278]: from='client.56175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:26 smithi146 ceph-mon[103278]: pgmap v5355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:27.221 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:18:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:18:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:18:27.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:18:27.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:18:27.223 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:18:27.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:18:27.492 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:18:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:27 smithi078 ceph-mon[99959]: from='client.76100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:27 smithi078 ceph-mon[99959]: from='client.76106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3294627441' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:27 smithi078 ceph-mon[91136]: from='client.76100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:27 smithi078 ceph-mon[91136]: from='client.76106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3294627441' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:27.800 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:18:27.800 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:27.801 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:27.801 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:27.801 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:27 smithi146 ceph-mon[103278]: from='client.76100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:27 smithi146 ceph-mon[103278]: from='client.76106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3294627441' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[91136]: from='client.56199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3478625408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[91136]: pgmap v5356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[99959]: from='client.56199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3478625408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:28.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:28 smithi078 ceph-mon[99959]: pgmap v5356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:28 smithi146 ceph-mon[103278]: from='client.56199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3478625408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:18:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:28 smithi146 ceph-mon[103278]: pgmap v5356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:30 smithi078 ceph-mon[91136]: pgmap v5357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:30 smithi078 ceph-mon[99959]: pgmap v5357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:18:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:18:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:30 smithi146 ceph-mon[103278]: pgmap v5357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:31.224 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:30.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:31.224 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:30.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:31.224 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:30.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:32 smithi078 ceph-mon[99959]: pgmap v5358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:32 smithi078 ceph-mon[91136]: pgmap v5358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:32 smithi146 ceph-mon[103278]: pgmap v5358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:34 smithi078 ceph-mon[91136]: pgmap v5359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:34 smithi078 ceph-mon[99959]: pgmap v5359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:34 smithi146 ceph-mon[103278]: pgmap v5359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:36.907 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:36 smithi078 ceph-mon[99959]: pgmap v5360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:36.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:36 smithi078 ceph-mon[91136]: pgmap v5360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:36 smithi146 ceph-mon[103278]: pgmap v5360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:38.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:38.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:38 smithi078 ceph-mon[99959]: pgmap v5361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:38.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:38.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:38 smithi078 ceph-mon[91136]: pgmap v5361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:38 smithi146 ceph-mon[103278]: pgmap v5361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:40.873 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:18:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:18:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:40.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:40.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:40.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:41 smithi146 ceph-mon[103278]: pgmap v5362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:41 smithi078 ceph-mon[91136]: pgmap v5362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:41 smithi078 ceph-mon[99959]: pgmap v5362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:42 smithi078 ceph-mon[91136]: pgmap v5363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:42 smithi078 ceph-mon[99959]: pgmap v5363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:42.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:42 smithi146 ceph-mon[103278]: pgmap v5363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:43.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:44 smithi078 ceph-mon[91136]: pgmap v5364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:44 smithi078 ceph-mon[99959]: pgmap v5364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:44 smithi146 ceph-mon[103278]: pgmap v5364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:46 smithi078 ceph-mon[91136]: pgmap v5365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:46 smithi078 ceph-mon[99959]: pgmap v5365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:46 smithi146 ceph-mon[103278]: pgmap v5365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:48 smithi078 ceph-mon[91136]: pgmap v5366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:48 smithi078 ceph-mon[99959]: pgmap v5366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:48 smithi146 ceph-mon[103278]: pgmap v5366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:50.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:50 smithi078 ceph-mon[91136]: pgmap v5367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:50 smithi078 ceph-mon[99959]: pgmap v5367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:50.874 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:18:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:18:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:50 smithi146 ceph-mon[103278]: pgmap v5367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:50.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:50.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:18:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:18:50.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:18:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:52 smithi078 ceph-mon[91136]: pgmap v5368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:52 smithi078 ceph-mon[99959]: pgmap v5368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:52 smithi146 ceph-mon[103278]: pgmap v5368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:54 smithi078 ceph-mon[91136]: pgmap v5369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:54 smithi078 ceph-mon[99959]: pgmap v5369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:54 smithi146 ceph-mon[103278]: pgmap v5369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:18:57.239 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:56 smithi078 ceph-mon[99959]: pgmap v5370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:57.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:56 smithi078 ceph-mon[91136]: pgmap v5370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:56 smithi146 ceph-mon[103278]: pgmap v5370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:18:58.100 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:18:58.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:58.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 3h - - 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 3h - - 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 3s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 19s ago 3h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 3s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 19s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:18:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 3s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 3s ago 3h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 19s ago 3h 212M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 3s ago 3h 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 3s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 19s ago 3h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 3s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 3s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 3s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 3s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 19s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 19s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 19s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 19s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:18:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 19s ago 3h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:18:58.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 3s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:18:58.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:18:58.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 3s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:18:58.669 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:18:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:18:58.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:18:58.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:18:59.253 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:18:59.254 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:18:59.255 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:18:59.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:59 smithi146 ceph-mon[103278]: from='client.56211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:59.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:59.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:59 smithi146 ceph-mon[103278]: from='client.76136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:59.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:59 smithi146 ceph-mon[103278]: pgmap v5371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:59.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:18:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3364602141' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[91136]: from='client.56211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[91136]: from='client.76136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[91136]: pgmap v5371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3364602141' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[99959]: from='client.56211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[99959]: from='client.76136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:18:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[99959]: pgmap v5371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:18:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:18:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3364602141' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:18:59.570 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:18:59.571 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:18:59.571 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:18:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:00.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:00 smithi146 ceph-mon[103278]: from='client.76142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:00.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:00 smithi146 ceph-mon[103278]: from='client.76154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:00.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2074322420' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:19:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[91136]: from='client.76142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[91136]: from='client.76154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2074322420' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[99959]: from='client.76142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[99959]: from='client.76154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2074322420' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:00.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:00.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:00.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:19:01.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:01 smithi146 ceph-mon[103278]: pgmap v5372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:01 smithi078 ceph-mon[91136]: pgmap v5372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:01 smithi078 ceph-mon[99959]: pgmap v5372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:02 smithi078 ceph-mon[91136]: pgmap v5373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:02 smithi078 ceph-mon[99959]: pgmap v5373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:02.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:02 smithi146 ceph-mon[103278]: pgmap v5373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:04 smithi078 ceph-mon[91136]: pgmap v5374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:04 smithi078 ceph-mon[99959]: pgmap v5374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:04 smithi146 ceph-mon[103278]: pgmap v5374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:06 smithi078 ceph-mon[99959]: pgmap v5375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:07.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:06 smithi078 ceph-mon[91136]: pgmap v5375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:07.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:06 smithi146 ceph-mon[103278]: pgmap v5375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:08 smithi078 ceph-mon[91136]: pgmap v5376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:08 smithi078 ceph-mon[99959]: pgmap v5376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:08 smithi146 ceph-mon[103278]: pgmap v5376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:10 smithi078 ceph-mon[91136]: pgmap v5377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:10 smithi078 ceph-mon[99959]: pgmap v5377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:19:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:19:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:10 smithi146 ceph-mon[103278]: pgmap v5377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:10.874Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:10.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:10.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:19:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:12 smithi078 ceph-mon[91136]: pgmap v5378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:12 smithi078 ceph-mon[99959]: pgmap v5378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:12 smithi146 ceph-mon[103278]: pgmap v5378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:14.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:14 smithi078 ceph-mon[99959]: pgmap v5379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:14.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:14 smithi078 ceph-mon[91136]: pgmap v5379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:14 smithi146 ceph-mon[103278]: pgmap v5379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:16 smithi146 ceph-mon[103278]: pgmap v5380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:16 smithi078 ceph-mon[91136]: pgmap v5380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:16 smithi078 ceph-mon[99959]: pgmap v5380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:18.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:18.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:18 smithi078 ceph-mon[91136]: pgmap v5381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:18.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:18.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:18 smithi078 ceph-mon[99959]: pgmap v5381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:18 smithi146 ceph-mon[103278]: pgmap v5381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:20.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[91136]: pgmap v5382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:19:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:20.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-mon[99959]: pgmap v5382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:20 smithi146 ceph-mon[103278]: pgmap v5382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:20.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:20.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:20.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:19:22.592 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:22 smithi146 ceph-mon[103278]: pgmap v5383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:22 smithi078 ceph-mon[91136]: pgmap v5383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:22 smithi078 ceph-mon[99959]: pgmap v5383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:24 smithi146 ceph-mon[103278]: pgmap v5384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:24 smithi078 ceph-mon[91136]: pgmap v5384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:24 smithi078 ceph-mon[99959]: pgmap v5384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:26 smithi078 ceph-mon[99959]: pgmap v5385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:26 smithi078 ceph-mon[91136]: pgmap v5385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:26 smithi146 ceph-mon[103278]: pgmap v5385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:29.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:29.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:29 smithi078 ceph-mon[99959]: pgmap v5386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:29.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:29.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:29 smithi078 ceph-mon[91136]: pgmap v5386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:29 smithi146 ceph-mon[103278]: pgmap v5386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:29.872 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 3h - - 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 15s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 15s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 15s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 15s ago 3h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:19:30.399 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 10s ago 3h 213M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 15s ago 3h 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 15s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 15s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 15s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 15s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 15s ago 3h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 10s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 10s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 10s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 10s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 10s ago 3h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 15s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:19:30.400 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 15s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:19:30.401 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:19:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:19:30.717 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:19:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[99959]: from='client.76166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[99959]: from='client.76172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[99959]: from='client.76178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[99959]: pgmap v5387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[91136]: from='client.76166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:30.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[91136]: from='client.76172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:30.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[91136]: from='client.76178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:30.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-mon[91136]: pgmap v5387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:19:30.985 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:19:30.986 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:19:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:30 smithi146 ceph-mon[103278]: from='client.76166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:30 smithi146 ceph-mon[103278]: from='client.76172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:30 smithi146 ceph-mon[103278]: from='client.76178 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:30 smithi146 ceph-mon[103278]: pgmap v5387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:30.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:30.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:30.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:19:31.298 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:19:31.298 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:19:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:19:31.298 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:31.298 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:19:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3898167855' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:31 smithi078 ceph-mon[91136]: from='client.56271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2313243953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3898167855' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:31 smithi078 ceph-mon[99959]: from='client.56271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2313243953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3898167855' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:19:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:31 smithi146 ceph-mon[103278]: from='client.56271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:19:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2313243953' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:19:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:32 smithi146 ceph-mon[103278]: pgmap v5388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:32 smithi078 ceph-mon[99959]: pgmap v5388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:32 smithi078 ceph-mon[91136]: pgmap v5388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:34 smithi146 ceph-mon[103278]: pgmap v5389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:34 smithi078 ceph-mon[91136]: pgmap v5389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:34 smithi078 ceph-mon[99959]: pgmap v5389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:37 smithi146 ceph-mon[103278]: pgmap v5390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:37.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:37 smithi078 ceph-mon[99959]: pgmap v5390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:37.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:37 smithi078 ceph-mon[91136]: pgmap v5390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:38 smithi078 ceph-mon[99959]: pgmap v5391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:38 smithi078 ceph-mon[91136]: pgmap v5391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:38 smithi146 ceph-mon[103278]: pgmap v5391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:40.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[91136]: pgmap v5392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:40.879 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:19:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:19:40.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:40.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:40.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:40.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-mon[99959]: pgmap v5392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:40 smithi146 ceph-mon[103278]: pgmap v5392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:40.876Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:40.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:40.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:19:42.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:42 smithi146 ceph-mon[103278]: pgmap v5393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:42 smithi078 ceph-mon[91136]: pgmap v5393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:42 smithi078 ceph-mon[99959]: pgmap v5393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:44 smithi146 ceph-mon[103278]: pgmap v5394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:44 smithi078 ceph-mon[91136]: pgmap v5394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:44 smithi078 ceph-mon[99959]: pgmap v5394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:46 smithi146 ceph-mon[103278]: pgmap v5395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:46 smithi078 ceph-mon[91136]: pgmap v5395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:46 smithi078 ceph-mon[99959]: pgmap v5395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:48 smithi146 ceph-mon[103278]: pgmap v5396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:48 smithi078 ceph-mon[91136]: pgmap v5396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:48 smithi078 ceph-mon[99959]: pgmap v5396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:50.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:50 smithi078 ceph-mon[99959]: pgmap v5397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:50.839 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:19:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:19:50.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:50 smithi078 ceph-mon[91136]: pgmap v5397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:50 smithi146 ceph-mon[103278]: pgmap v5397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:50.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:50.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:19:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:19:50.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:19:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:52 smithi146 ceph-mon[103278]: pgmap v5398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:52 smithi078 ceph-mon[99959]: pgmap v5398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:52 smithi078 ceph-mon[91136]: pgmap v5398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:54 smithi146 ceph-mon[103278]: pgmap v5399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:54 smithi078 ceph-mon[91136]: pgmap v5399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:54 smithi078 ceph-mon[99959]: pgmap v5399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:19:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:57 smithi146 ceph-mon[103278]: pgmap v5400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:57 smithi078 ceph-mon[91136]: pgmap v5400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:57 smithi078 ceph-mon[99959]: pgmap v5400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:19:58.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:58.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:58.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:19:59.615 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:59.615 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:19:59 smithi078 ceph-mon[99959]: pgmap v5401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:59.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:59.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:19:59 smithi078 ceph-mon[91136]: pgmap v5401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:19:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:19:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:19:59 smithi146 ceph-mon[103278]: pgmap v5401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:00.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T23:19:59.998+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:00.837 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:20:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:20:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:00 smithi146 ceph-mon[103278]: pgmap v5402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[91136]: pgmap v5402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-mon[99959]: pgmap v5402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:01.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:00.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:01.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:00.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:01.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:00.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:01.596 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:20:02.116 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:20:02.116 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 3h - - 2024-09-20T23:20:02.116 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 3h - - 2024-09-20T23:20:02.116 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 6s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:20:02.116 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 2s ago 3h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:20:02.116 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 6s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 2s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 6s ago 3h 573M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 6s ago 3h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 2s ago 3h 214M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 6s ago 3h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 6s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 2s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 6s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 6s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 6s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 6s ago 3h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 2s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:20:02.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 2s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 2s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 2s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 2s ago 3h 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 6s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 6s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:20:02.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:02.426 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:20:02.427 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:20:02.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:02.427 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:20:02.427 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:20:02.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:20:02.427 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:20:02.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1750098509' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:20:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:20:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:20:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:20:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:20:02.701 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:20:02.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1750098509' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:02.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1750098509' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:03.006 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:03.006 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:20:03.007 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:20:03.007 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:20:03.007 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:20:03.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:03 smithi146 ceph-mon[103278]: from='client.76202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:03 smithi146 ceph-mon[103278]: from='client.56289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:03 smithi146 ceph-mon[103278]: from='client.56295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:03 smithi146 ceph-mon[103278]: pgmap v5403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:03.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2807932001' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[91136]: from='client.76202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[91136]: from='client.56289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[91136]: from='client.56295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[91136]: pgmap v5403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2807932001' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[99959]: from='client.76202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[99959]: from='client.56289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[99959]: from='client.56295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[99959]: pgmap v5403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2807932001' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:04.749 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:04 smithi146 ceph-mon[103278]: from='client.56307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:04 smithi078 ceph-mon[91136]: from='client.56307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:04 smithi078 ceph-mon[99959]: from='client.56307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:05 smithi146 ceph-mon[103278]: pgmap v5404: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:05 smithi078 ceph-mon[91136]: pgmap v5404: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:05 smithi078 ceph-mon[99959]: pgmap v5404: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:07 smithi146 ceph-mon[103278]: pgmap v5405: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:07 smithi078 ceph-mon[91136]: pgmap v5405: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:07 smithi078 ceph-mon[99959]: pgmap v5405: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:09.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:08 smithi146 ceph-mon[103278]: pgmap v5406: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:08 smithi078 ceph-mon[91136]: pgmap v5406: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:08 smithi078 ceph-mon[99959]: pgmap v5406: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:10.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:10 smithi078 ceph-mon[91136]: pgmap v5407: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:10.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:10 smithi078 ceph-mon[99959]: pgmap v5407: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:10.881 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:20:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:20:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:10 smithi146 ceph-mon[103278]: pgmap v5407: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:10.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:10.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:10.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:12 smithi146 ceph-mon[103278]: pgmap v5408: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:12 smithi078 ceph-mon[91136]: pgmap v5408: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:12 smithi078 ceph-mon[99959]: pgmap v5408: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:14 smithi146 ceph-mon[103278]: pgmap v5409: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:14 smithi078 ceph-mon[91136]: pgmap v5409: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:14 smithi078 ceph-mon[99959]: pgmap v5409: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:16.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:17 smithi146 ceph-mon[103278]: pgmap v5410: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:17 smithi078 ceph-mon[91136]: pgmap v5410: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:17 smithi078 ceph-mon[99959]: pgmap v5410: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:18.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:18.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:18 smithi078 ceph-mon[99959]: pgmap v5411: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:18.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:18.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:18 smithi078 ceph-mon[91136]: pgmap v5411: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:18 smithi146 ceph-mon[103278]: pgmap v5411: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:20.672 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:20:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:20:21.014 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:20.880Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:21.014 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:20.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:21.015 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:20.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:21 smithi146 ceph-mon[103278]: pgmap v5412: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[91136]: pgmap v5412: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:21 smithi078 ceph-mon[99959]: pgmap v5412: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:22 smithi078 ceph-mon[99959]: pgmap v5413: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:22 smithi078 ceph-mon[91136]: pgmap v5413: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:22.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:22 smithi146 ceph-mon[103278]: pgmap v5413: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:24.788 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:24 smithi146 ceph-mon[103278]: pgmap v5414: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:24 smithi078 ceph-mon[91136]: pgmap v5414: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:24 smithi078 ceph-mon[99959]: pgmap v5414: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:26 smithi146 ceph-mon[103278]: pgmap v5415: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:26 smithi078 ceph-mon[91136]: pgmap v5415: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:26 smithi078 ceph-mon[99959]: pgmap v5415: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:29 smithi146 ceph-mon[103278]: pgmap v5416: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:29 smithi078 ceph-mon[99959]: pgmap v5416: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:29 smithi078 ceph-mon[91136]: pgmap v5416: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:30.882 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:20:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:20:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:30.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:30.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:30.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:31 smithi146 ceph-mon[103278]: pgmap v5417: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:31 smithi078 ceph-mon[99959]: pgmap v5417: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:31 smithi078 ceph-mon[91136]: pgmap v5417: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:33.304 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:20:33.831 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 3h - - 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 3h - - 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 18s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 13s ago 3h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 18s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 13s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 18s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 18s ago 3h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 13s ago 3h 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 18s ago 3h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:20:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 18s ago 3h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 13s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 18s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 18s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 18s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 18s ago 3h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 13s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 13s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 13s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 13s ago 3h 72.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 13s ago 3h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 18s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 18s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:20:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:20:33.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:33 smithi078 ceph-mon[91136]: pgmap v5418: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:33.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:33 smithi078 ceph-mon[91136]: from='client.76238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:33.834 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:33 smithi078 ceph-mon[99959]: pgmap v5418: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:33.834 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:33 smithi078 ceph-mon[99959]: from='client.76238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:33 smithi146 ceph-mon[103278]: pgmap v5418: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:33 smithi146 ceph-mon[103278]: from='client.76238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.142 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:20:34.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:20:34.142 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:20:34.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:34.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:20:34.143 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:20:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:20:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:20:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:20:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:20:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:20:34.413 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:20:34.724 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:20:34.724 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:20:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:20:34.724 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:20:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[99959]: from='client.56325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[99959]: from='client.56331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1177829424' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[99959]: from='client.76262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[91136]: from='client.56325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[91136]: from='client.56331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1177829424' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:34.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:34 smithi078 ceph-mon[91136]: from='client.76262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:34 smithi146 ceph-mon[103278]: from='client.56325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:34 smithi146 ceph-mon[103278]: from='client.56331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1177829424' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:20:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:34 smithi146 ceph-mon[103278]: from='client.76262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:20:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[91136]: pgmap v5419: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/244596496' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[99959]: pgmap v5419: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/244596496' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:35.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:35 smithi146 ceph-mon[103278]: pgmap v5419: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/244596496' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:37 smithi146 ceph-mon[103278]: pgmap v5420: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:37 smithi078 ceph-mon[91136]: pgmap v5420: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:37 smithi078 ceph-mon[99959]: pgmap v5420: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:39.820 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:39 smithi078 ceph-mon[99959]: pgmap v5421: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:39.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:39 smithi078 ceph-mon[91136]: pgmap v5421: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:39 smithi146 ceph-mon[103278]: pgmap v5421: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:20:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:20:41.143 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:40.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:41.143 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:40.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:41.143 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:40.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:42.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:42 smithi078 ceph-mon[99959]: pgmap v5422: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:42.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:42 smithi078 ceph-mon[91136]: pgmap v5422: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:42 smithi146 ceph-mon[103278]: pgmap v5422: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:44 smithi146 ceph-mon[103278]: pgmap v5423: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:44 smithi078 ceph-mon[91136]: pgmap v5423: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:44 smithi078 ceph-mon[99959]: pgmap v5423: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:46.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:46 smithi146 ceph-mon[103278]: pgmap v5424: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:46 smithi078 ceph-mon[91136]: pgmap v5424: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:46 smithi078 ceph-mon[99959]: pgmap v5424: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:48 smithi146 ceph-mon[103278]: pgmap v5425: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:48 smithi078 ceph-mon[91136]: pgmap v5425: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:48 smithi078 ceph-mon[99959]: pgmap v5425: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:50.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:50 smithi078 ceph-mon[91136]: pgmap v5426: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:50.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:50 smithi078 ceph-mon[99959]: pgmap v5426: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:50 smithi146 ceph-mon[103278]: pgmap v5426: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:20:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:20:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:20:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:50.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:50.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:20:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:20:50.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:20:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:52 smithi146 ceph-mon[103278]: pgmap v5427: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:52 smithi078 ceph-mon[91136]: pgmap v5427: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:52 smithi078 ceph-mon[99959]: pgmap v5427: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:54 smithi146 ceph-mon[103278]: pgmap v5428: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:54 smithi078 ceph-mon[91136]: pgmap v5428: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:54 smithi078 ceph-mon[99959]: pgmap v5428: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:20:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:56 smithi146 ceph-mon[103278]: pgmap v5429: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:20:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[99959]: pgmap v5429: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:20:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[91136]: pgmap v5429: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:20:56.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:56.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:20:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:58 smithi146 ceph-mon[103278]: pgmap v5430: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:20:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:58 smithi078 ceph-mon[91136]: pgmap v5430: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:20:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:58 smithi078 ceph-mon[99959]: pgmap v5430: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:20:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:59.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:59.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:59.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:20:59.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:20:59.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:00.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:00 smithi078 ceph-mon[99959]: pgmap v5431: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:21:00.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:00 smithi078 ceph-mon[91136]: pgmap v5431: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:21:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:00 smithi146 ceph-mon[103278]: pgmap v5431: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:21:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:21:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:21:01.276 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:00.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:01.276 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:00.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:01.276 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:00.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:02 smithi078 ceph-mon[91136]: pgmap v5432: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T23:21:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:02 smithi078 ceph-mon[99959]: pgmap v5432: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T23:21:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:02 smithi146 ceph-mon[103278]: pgmap v5432: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-20T23:21:04.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:04 smithi146 ceph-mon[103278]: pgmap v5433: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:21:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:04 smithi078 ceph-mon[99959]: pgmap v5433: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:21:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:04 smithi078 ceph-mon[91136]: pgmap v5433: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:21:05.030 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:21:05.558 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 3h - - 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 3h - - 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 9s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 5s ago 3h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 9s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 5s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (2h) 9s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 9s ago 3h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 5s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 9s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 9s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 5s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 9s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:21:05.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 9s ago 3h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 9s ago 3h 70.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 9s ago 3h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 5s ago 3h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 5s ago 3h 71.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 5s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 5s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 5s ago 3h 77.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 9s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 9s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:21:05.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:05.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:21:05.867 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:21:06.134 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:21:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:21:06.135 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:21:06.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:21:06.444 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:21:06.444 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:06.445 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:06.445 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:06.445 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[91136]: pgmap v5434: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[91136]: from='client.76274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[91136]: from='client.56361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/644026430' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[99959]: pgmap v5434: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[99959]: from='client.76274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:06.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[99959]: from='client.56361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:06.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/644026430' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:06 smithi146 ceph-mon[103278]: pgmap v5434: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-20T23:21:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:06 smithi146 ceph-mon[103278]: from='client.76274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:06 smithi146 ceph-mon[103278]: from='client.56361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/644026430' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:07 smithi146 ceph-mon[103278]: from='client.56367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:07 smithi146 ceph-mon[103278]: from='client.56379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2153469578' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:07 smithi078 ceph-mon[91136]: from='client.56367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:07 smithi078 ceph-mon[91136]: from='client.56379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2153469578' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:07 smithi078 ceph-mon[99959]: from='client.56367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:07 smithi078 ceph-mon[99959]: from='client.56379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2153469578' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:08 smithi078 ceph-mon[91136]: pgmap v5435: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:21:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:08 smithi078 ceph-mon[99959]: pgmap v5435: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:21:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:08 smithi146 ceph-mon[103278]: pgmap v5435: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:21:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:10 smithi078 ceph-mon[99959]: pgmap v5436: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:21:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:10 smithi078 ceph-mon[91136]: pgmap v5436: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:21:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:10 smithi146 ceph-mon[103278]: pgmap v5436: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:21:10.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:21:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:21:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:10.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:10.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:10.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:12 smithi146 ceph-mon[103278]: pgmap v5437: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T23:21:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:12 smithi078 ceph-mon[91136]: pgmap v5437: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T23:21:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:12 smithi078 ceph-mon[99959]: pgmap v5437: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T23:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:14 smithi146 ceph-mon[103278]: pgmap v5438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:14 smithi078 ceph-mon[99959]: pgmap v5438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:21:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:14 smithi078 ceph-mon[91136]: pgmap v5438: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:21:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:16 smithi146 ceph-mon[103278]: pgmap v5439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:21:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[91136]: pgmap v5439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[99959]: pgmap v5439: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:18 smithi078 ceph-mon[91136]: pgmap v5440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-20T23:21:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:18 smithi078 ceph-mon[99959]: pgmap v5440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-20T23:21:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:18 smithi146 ceph-mon[103278]: pgmap v5440: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-20T23:21:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:20 smithi078 ceph-mon[91136]: pgmap v5441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:21:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:20 smithi078 ceph-mon[99959]: pgmap v5441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:21:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:20 smithi146 ceph-mon[103278]: pgmap v5441: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:21:20.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:21:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:21:21.218 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:20.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:21.219 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:20.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:21.219 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:20.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:22.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:22 smithi078 ceph-mon[91136]: pgmap v5442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:21:22.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:22 smithi078 ceph-mon[99959]: pgmap v5442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:21:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:22 smithi146 ceph-mon[103278]: pgmap v5442: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-20T23:21:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:24 smithi146 ceph-mon[103278]: pgmap v5443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:21:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:24 smithi078 ceph-mon[91136]: pgmap v5443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:21:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:24 smithi078 ceph-mon[99959]: pgmap v5443: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-20T23:21:26.648 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:26 smithi146 ceph-mon[103278]: pgmap v5444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:21:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:26 smithi078 ceph-mon[91136]: pgmap v5444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:21:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:26 smithi078 ceph-mon[99959]: pgmap v5444: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-20T23:21:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:28 smithi078 ceph-mon[99959]: pgmap v5445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:21:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:28 smithi078 ceph-mon[91136]: pgmap v5445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:21:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:28 smithi146 ceph-mon[103278]: pgmap v5445: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:21:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:30 smithi146 ceph-mon[103278]: pgmap v5446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:21:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:30 smithi078 ceph-mon[91136]: pgmap v5446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:21:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:30 smithi078 ceph-mon[99959]: pgmap v5446: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:21:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:21:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:21:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:30.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:30.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:30.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:32 smithi146 ceph-mon[103278]: pgmap v5447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:32 smithi078 ceph-mon[91136]: pgmap v5447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:32 smithi078 ceph-mon[99959]: pgmap v5447: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:34 smithi146 ceph-mon[103278]: pgmap v5448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:21:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:34 smithi078 ceph-mon[91136]: pgmap v5448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:21:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:34 smithi078 ceph-mon[99959]: pgmap v5448: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T23:21:36.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[99959]: pgmap v5449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[91136]: pgmap v5449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:21:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:36 smithi146 ceph-mon[103278]: pgmap v5449: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 3h - - 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 3h - - 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 1s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 16s ago 3h 87.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (2h) 1s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 16s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:21:37.275 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 1s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 1s ago 3h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 16s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 1s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 1s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 16s ago 3h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 1s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 1s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 1s ago 3h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 1s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 16s ago 3h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 16s ago 3h 72.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 16s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 16s ago 3h 72.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (2h) 16s ago 3h 70.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 1s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:21:37.276 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:21:37.277 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 1s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:21:37.277 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:21:37.587 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:21:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:21:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:21:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:21:37.589 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:21:37.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:21:37.857 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:21:37.857 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:21:37.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:21:38.165 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:21:38.166 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:38.166 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:21:38.166 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:38.166 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:21:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:38 smithi146 ceph-mon[103278]: pgmap v5450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:38 smithi146 ceph-mon[103278]: from='client.56391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:38 smithi146 ceph-mon[103278]: from='client.76316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:38 smithi146 ceph-mon[103278]: from='client.76322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1045540568' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2882036478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[91136]: pgmap v5450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[91136]: from='client.56391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[91136]: from='client.76316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[91136]: from='client.76322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1045540568' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2882036478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[99959]: pgmap v5450: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[99959]: from='client.56391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[99959]: from='client.76316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[99959]: from='client.76322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1045540568' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:21:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2882036478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:21:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:39 smithi146 ceph-mon[103278]: from='client.56415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:39 smithi078 ceph-mon[99959]: from='client.56415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:39 smithi078 ceph-mon[91136]: from='client.56415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:21:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:40 smithi146 ceph-mon[103278]: pgmap v5451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:40 smithi078 ceph-mon[91136]: pgmap v5451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:40 smithi078 ceph-mon[99959]: pgmap v5451: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:21:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:21:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:40.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:40.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:40.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:41.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[91136]: pgmap v5452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[99959]: pgmap v5452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:41.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:41 smithi146 ceph-mon[103278]: pgmap v5452: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:43.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:43 smithi078 ceph-mon[99959]: pgmap v5453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:43.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:43.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:43 smithi078 ceph-mon[91136]: pgmap v5453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:43.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:43 smithi146 ceph-mon[103278]: pgmap v5453: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:45.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:45 smithi146 ceph-mon[103278]: pgmap v5454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:45 smithi078 ceph-mon[91136]: pgmap v5454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:45 smithi078 ceph-mon[99959]: pgmap v5454: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:47.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:47 smithi146 ceph-mon[103278]: pgmap v5455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:47 smithi078 ceph-mon[91136]: pgmap v5455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:47 smithi078 ceph-mon[99959]: pgmap v5455: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:49 smithi146 ceph-mon[103278]: pgmap v5456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:49 smithi078 ceph-mon[99959]: pgmap v5456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:49 smithi078 ceph-mon[91136]: pgmap v5456: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:50.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:21:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:21:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:50.886Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:50.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:21:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:21:50.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:21:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:51 smithi078 ceph-mon[99959]: pgmap v5457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:51 smithi078 ceph-mon[91136]: pgmap v5457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:51 smithi146 ceph-mon[103278]: pgmap v5457: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:53 smithi146 ceph-mon[103278]: pgmap v5458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:53 smithi078 ceph-mon[91136]: pgmap v5458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:53 smithi078 ceph-mon[99959]: pgmap v5458: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:55.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:55 smithi078 ceph-mon[91136]: pgmap v5459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:55.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:55 smithi078 ceph-mon[99959]: pgmap v5459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:55 smithi146 ceph-mon[103278]: pgmap v5459: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:21:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:21:58.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:57 smithi078 ceph-mon[91136]: pgmap v5460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:58.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:57 smithi078 ceph-mon[99959]: pgmap v5460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:57 smithi146 ceph-mon[103278]: pgmap v5460: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:21:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:21:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:21:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:21:59 smithi146 ceph-mon[103278]: pgmap v5461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:21:59 smithi078 ceph-mon[91136]: pgmap v5461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:21:59 smithi078 ceph-mon[99959]: pgmap v5461: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:00.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:22:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:22:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:00.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:00.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:00.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:01 smithi146 ceph-mon[103278]: pgmap v5462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[99959]: pgmap v5462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[91136]: pgmap v5462: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:03.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:03 smithi078 ceph-mon[91136]: pgmap v5463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:03.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:03 smithi078 ceph-mon[99959]: pgmap v5463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:03 smithi146 ceph-mon[103278]: pgmap v5463: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:05 smithi146 ceph-mon[103278]: pgmap v5464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:05 smithi078 ceph-mon[91136]: pgmap v5464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:05 smithi078 ceph-mon[99959]: pgmap v5464: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:07.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:07 smithi146 ceph-mon[103278]: pgmap v5465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:07 smithi078 ceph-mon[99959]: pgmap v5465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:07 smithi078 ceph-mon[91136]: pgmap v5465: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:08.461 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:22:08.673 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:08.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:08 smithi078 ceph-mon[99959]: from='client.76346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:08.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:08.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:08 smithi078 ceph-mon[91136]: from='client.76346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 3h - - 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 3h - - 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 8s ago 3h 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:22:08.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 8s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 13s ago 3h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 8s ago 3h 215M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 13s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 8s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 13s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 13s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 13s ago 3h 71.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 13s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:22:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:22:08.996 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 8s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:22:08.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 13s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:22:08.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:22:08.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 13s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:22:08.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:22:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:08 smithi146 ceph-mon[103278]: from='client.76346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:09.304 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:22:09.304 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:22:09.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:22:09.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:22:09.305 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:22:09.575 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:22:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:22:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:22:09.576 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:22:09.576 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:22:09.885 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:22:09.885 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:09.885 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:09.885 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:09.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:09.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[99959]: pgmap v5466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:09.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[99959]: from='client.56433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:09.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[99959]: from='client.56439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:09.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3432447122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:09.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[91136]: pgmap v5466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:09.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[91136]: from='client.56433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:09.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[91136]: from='client.56439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:09.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3432447122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:09 smithi146 ceph-mon[103278]: pgmap v5466: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:09 smithi146 ceph-mon[103278]: from='client.56433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:09 smithi146 ceph-mon[103278]: from='client.56439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3432447122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:10.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-mon[99959]: from='client.56451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:10.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/461764910' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:10.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-mon[91136]: from='client.56451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:10.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/461764910' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:10.890 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:22:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:22:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:10 smithi146 ceph-mon[103278]: from='client.56451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/461764910' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:10.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:10.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:10.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:11 smithi146 ceph-mon[103278]: pgmap v5467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:11 smithi078 ceph-mon[99959]: pgmap v5467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:11 smithi078 ceph-mon[91136]: pgmap v5467: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:13 smithi146 ceph-mon[103278]: pgmap v5468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:13 smithi078 ceph-mon[99959]: pgmap v5468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:13 smithi078 ceph-mon[91136]: pgmap v5468: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:15 smithi146 ceph-mon[103278]: pgmap v5469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:15 smithi078 ceph-mon[91136]: pgmap v5469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:15 smithi078 ceph-mon[99959]: pgmap v5469: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:18 smithi146 ceph-mon[103278]: pgmap v5470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:18 smithi078 ceph-mon[91136]: pgmap v5470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:18 smithi078 ceph-mon[99959]: pgmap v5470: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:20.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:20 smithi078 ceph-mon[91136]: pgmap v5471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:20.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:20 smithi078 ceph-mon[99959]: pgmap v5471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:20 smithi146 ceph-mon[103278]: pgmap v5471: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:22:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:22:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:20.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:20.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:20.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:21 smithi146 ceph-mon[103278]: pgmap v5472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[99959]: pgmap v5472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[91136]: pgmap v5472: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:24.121 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:23 smithi078 ceph-mon[91136]: pgmap v5473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:24.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:23 smithi078 ceph-mon[99959]: pgmap v5473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:23 smithi146 ceph-mon[103278]: pgmap v5473: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:25 smithi146 ceph-mon[103278]: pgmap v5474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:25 smithi078 ceph-mon[91136]: pgmap v5474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:25 smithi078 ceph-mon[99959]: pgmap v5474: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:28.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:27 smithi146 ceph-mon[103278]: pgmap v5475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:27 smithi078 ceph-mon[91136]: pgmap v5475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:27 smithi078 ceph-mon[99959]: pgmap v5475: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:28.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:28.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:29 smithi146 ceph-mon[103278]: pgmap v5476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:29 smithi078 ceph-mon[91136]: pgmap v5476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:29 smithi078 ceph-mon[99959]: pgmap v5476: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:30.891 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:22:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:22:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:30.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:30.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:30.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:32.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:31 smithi078 ceph-mon[91136]: pgmap v5477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:32.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:31 smithi078 ceph-mon[99959]: pgmap v5477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:31 smithi146 ceph-mon[103278]: pgmap v5477: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:33 smithi146 ceph-mon[103278]: pgmap v5478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:33 smithi078 ceph-mon[91136]: pgmap v5478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:33 smithi078 ceph-mon[99959]: pgmap v5478: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:36.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:35 smithi078 ceph-mon[99959]: pgmap v5479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:36.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:35 smithi078 ceph-mon[91136]: pgmap v5479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:35 smithi146 ceph-mon[103278]: pgmap v5479: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:38 smithi146 ceph-mon[103278]: pgmap v5480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:38 smithi078 ceph-mon[91136]: pgmap v5480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:38 smithi078 ceph-mon[99959]: pgmap v5480: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:40.179 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:22:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:40 smithi146 ceph-mon[103278]: pgmap v5481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:40 smithi078 ceph-mon[99959]: pgmap v5481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:40 smithi078 ceph-mon[91136]: pgmap v5481: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 3h - - 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 3h - - 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 4s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 19s ago 3h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 4s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (2h) 19s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 4s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (2h) 4s ago 3h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:22:40.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 19s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 4s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 4s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 19s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 4s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 4s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 4s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 4s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 19s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 19s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 19s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 19s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 19s ago 3h 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 4s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:22:40.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 4s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:22:40.714 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:22:40.892 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:22:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:22:41.029 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:22:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:22:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:22:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:22:41.252 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:40.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:41.253 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:40.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:41.253 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:40.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:41.302 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:22:41.302 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:22:41.302 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:22:41.302 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:22:41.302 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:22:41.302 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:22:41.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:22:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:41 smithi146 ceph-mon[103278]: from='client.76382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:41 smithi146 ceph-mon[103278]: from='client.76388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2333188741' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[91136]: from='client.76382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[91136]: from='client.76388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2333188741' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[99959]: from='client.76382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[99959]: from='client.76388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2333188741' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:22:41.615 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:22:41.615 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:41.615 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:22:41.615 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:41.615 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:22:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:42 smithi146 ceph-mon[103278]: pgmap v5482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:42 smithi146 ceph-mon[103278]: from='client.56475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:42 smithi146 ceph-mon[103278]: from='client.56487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:42.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/230873672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[99959]: pgmap v5482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[99959]: from='client.56475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[99959]: from='client.56487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/230873672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[91136]: pgmap v5482: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[91136]: from='client.56475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[91136]: from='client.56487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:22:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/230873672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:22:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:44 smithi078 ceph-mon[99959]: pgmap v5483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:44 smithi078 ceph-mon[91136]: pgmap v5483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:44 smithi146 ceph-mon[103278]: pgmap v5483: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:46 smithi146 ceph-mon[103278]: pgmap v5484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:46 smithi078 ceph-mon[91136]: pgmap v5484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:46 smithi078 ceph-mon[99959]: pgmap v5484: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:48 smithi146 ceph-mon[103278]: pgmap v5485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:48 smithi078 ceph-mon[91136]: pgmap v5485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:48 smithi078 ceph-mon[99959]: pgmap v5485: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:49.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:50 smithi078 ceph-mon[91136]: pgmap v5486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:50 smithi078 ceph-mon[99959]: pgmap v5486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:50 smithi146 ceph-mon[103278]: pgmap v5486: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:50.892 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:22:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:22:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:50.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:50.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:22:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:22:50.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:22:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:52 smithi078 ceph-mon[99959]: pgmap v5487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:52 smithi078 ceph-mon[91136]: pgmap v5487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:52 smithi146 ceph-mon[103278]: pgmap v5487: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:54 smithi078 ceph-mon[99959]: pgmap v5488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:54 smithi078 ceph-mon[91136]: pgmap v5488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:54 smithi146 ceph-mon[103278]: pgmap v5488: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:56 smithi078 ceph-mon[91136]: pgmap v5489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:56 smithi078 ceph-mon[99959]: pgmap v5489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:56 smithi146 ceph-mon[103278]: pgmap v5489: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:22:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:22:58.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:58 smithi146 ceph-mon[103278]: pgmap v5490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:58.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:58 smithi078 ceph-mon[91136]: pgmap v5490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:58 smithi078 ceph-mon[99959]: pgmap v5490: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:22:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:22:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:22:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:22:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:22:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:22:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:00 smithi146 ceph-mon[103278]: pgmap v5491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:00 smithi078 ceph-mon[91136]: pgmap v5491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:00 smithi078 ceph-mon[99959]: pgmap v5491: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:23:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:23:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:00.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:00.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:00.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:02 smithi146 ceph-mon[103278]: pgmap v5492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[91136]: pgmap v5492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[99959]: pgmap v5492: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:02.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:04.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:04 smithi078 ceph-mon[99959]: pgmap v5493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:04.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:04 smithi078 ceph-mon[91136]: pgmap v5493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:04 smithi146 ceph-mon[103278]: pgmap v5493: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:06 smithi078 ceph-mon[91136]: pgmap v5494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:06 smithi078 ceph-mon[99959]: pgmap v5494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:06 smithi146 ceph-mon[103278]: pgmap v5494: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:08.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:08 smithi146 ceph-mon[103278]: pgmap v5495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:08.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:08 smithi078 ceph-mon[91136]: pgmap v5495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:08.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:08 smithi078 ceph-mon[99959]: pgmap v5495: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:10.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:10 smithi146 ceph-mon[103278]: pgmap v5496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:10 smithi078 ceph-mon[99959]: pgmap v5496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:10 smithi078 ceph-mon[91136]: pgmap v5496: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:10.894 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:23:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:23:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:10.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:10.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:10.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:11.909 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:23:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:12 smithi078 ceph-mon[99959]: pgmap v5497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:12 smithi078 ceph-mon[91136]: pgmap v5497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:12.435 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:23:12.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 3h - - 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 3h - - 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 16s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 11s ago 3h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 11s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 3h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 11s ago 3h 217M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 16s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 16s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 11s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 16s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 16s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:23:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 16s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 16s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 11s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 11s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 11s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 11s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 11s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 16s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 11s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 16s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:23:12.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 11s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:23:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:12 smithi146 ceph-mon[103278]: pgmap v5497: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:12.744 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:23:12.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:23:12.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:23:12.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:12.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:23:12.745 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:23:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:23:13.013 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:23:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[99959]: from='client.56499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[99959]: from='client.76424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[99959]: from='client.56511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/265803946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[91136]: from='client.56499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[91136]: from='client.76424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[91136]: from='client.56511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/265803946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:13.323 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:23:13.323 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:13.323 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:13 smithi146 ceph-mon[103278]: from='client.56499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:13 smithi146 ceph-mon[103278]: from='client.76424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:13 smithi146 ceph-mon[103278]: from='client.56511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/265803946' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:14 smithi146 ceph-mon[103278]: pgmap v5498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:14 smithi146 ceph-mon[103278]: from='client.56523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2118745935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[91136]: pgmap v5498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[91136]: from='client.56523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2118745935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[99959]: pgmap v5498: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[99959]: from='client.56523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2118745935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:16.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:16 smithi078 ceph-mon[99959]: pgmap v5499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:16.501 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:16 smithi078 ceph-mon[91136]: pgmap v5499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:16 smithi146 ceph-mon[103278]: pgmap v5499: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:18 smithi146 ceph-mon[103278]: pgmap v5500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:18 smithi078 ceph-mon[91136]: pgmap v5500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:18 smithi078 ceph-mon[99959]: pgmap v5500: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:20 smithi078 ceph-mon[91136]: pgmap v5501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:20 smithi078 ceph-mon[99959]: pgmap v5501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:23:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:23:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:20 smithi146 ceph-mon[103278]: pgmap v5501: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:20.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:20.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:20.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:22 smithi146 ceph-mon[103278]: pgmap v5502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[99959]: pgmap v5502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[91136]: pgmap v5502: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:24 smithi146 ceph-mon[103278]: pgmap v5503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:24 smithi078 ceph-mon[91136]: pgmap v5503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:24 smithi078 ceph-mon[99959]: pgmap v5503: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:26 smithi078 ceph-mon[99959]: pgmap v5504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:26 smithi078 ceph-mon[91136]: pgmap v5504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:26 smithi146 ceph-mon[103278]: pgmap v5504: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:28 smithi146 ceph-mon[103278]: pgmap v5505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:28 smithi078 ceph-mon[91136]: pgmap v5505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:28 smithi078 ceph-mon[99959]: pgmap v5505: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:30 smithi146 ceph-mon[103278]: pgmap v5506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:30 smithi078 ceph-mon[91136]: pgmap v5506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:30 smithi078 ceph-mon[99959]: pgmap v5506: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:30.895 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:23:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:23:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:30.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:30.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:30.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:32 smithi146 ceph-mon[103278]: pgmap v5507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:32 smithi078 ceph-mon[91136]: pgmap v5507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:32 smithi078 ceph-mon[99959]: pgmap v5507: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:34 smithi146 ceph-mon[103278]: pgmap v5508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:34 smithi078 ceph-mon[91136]: pgmap v5508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:34 smithi078 ceph-mon[99959]: pgmap v5508: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:36 smithi146 ceph-mon[103278]: pgmap v5509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:36 smithi078 ceph-mon[91136]: pgmap v5509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:36 smithi078 ceph-mon[99959]: pgmap v5509: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:37 smithi146 ceph-mon[103278]: pgmap v5510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[91136]: pgmap v5510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[99959]: pgmap v5510: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:39 smithi146 ceph-mon[103278]: pgmap v5511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:39 smithi078 ceph-mon[91136]: pgmap v5511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:39 smithi078 ceph-mon[99959]: pgmap v5511: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:40.896 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:23:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:23:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:40.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:40.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:40.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:42 smithi146 ceph-mon[103278]: pgmap v5512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[91136]: pgmap v5512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[99959]: pgmap v5512: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:43.619 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:23:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:23:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 3h - - 2024-09-20T23:23:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 3h - - 2024-09-20T23:23:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 7s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:23:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 2s ago 3h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:23:44.142 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 7s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 2s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 7s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 7s ago 3h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (2h) 2s ago 3h 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (2h) 7s ago 3h 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 7s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 2s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 7s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 7s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 7s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 7s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 2s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:23:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 2s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 2s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 2s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 2s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 7s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 7s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:23:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:23:44.453 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:23:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:23:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:23:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:23:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:23:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:23:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:23:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:23:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:23:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:23:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:23:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:23:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:23:44.455 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:23:44.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:44 smithi078 ceph-mon[99959]: pgmap v5513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:44.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:44.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:44 smithi078 ceph-mon[91136]: pgmap v5513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:44.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:44.722 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:23:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:23:44.723 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:23:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:44 smithi146 ceph-mon[103278]: pgmap v5513: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:45.029 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:23:45.029 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:23:45.030 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:23:45.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[91136]: from='client.76454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[91136]: from='client.56541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[91136]: from='client.56547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2544623867' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:45.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2871474803' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:45.660 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[99959]: from='client.76454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.660 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[99959]: from='client.56541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[99959]: from='client.56547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2544623867' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:45.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2871474803' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:45 smithi146 ceph-mon[103278]: from='client.76454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:45 smithi146 ceph-mon[103278]: from='client.56541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:45 smithi146 ceph-mon[103278]: from='client.56547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2544623867' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:23:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2871474803' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:23:46.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:46 smithi078 ceph-mon[99959]: pgmap v5514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:46.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:46 smithi078 ceph-mon[99959]: from='client.76478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:46.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:46 smithi078 ceph-mon[91136]: pgmap v5514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:46.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:46 smithi078 ceph-mon[91136]: from='client.76478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:46 smithi146 ceph-mon[103278]: pgmap v5514: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:46 smithi146 ceph-mon[103278]: from='client.76478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:23:48.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:48 smithi146 ceph-mon[103278]: pgmap v5515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:48 smithi078 ceph-mon[91136]: pgmap v5515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:48 smithi078 ceph-mon[99959]: pgmap v5515: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:49.404 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:50 smithi078 ceph-mon[91136]: pgmap v5516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:50 smithi078 ceph-mon[99959]: pgmap v5516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:50 smithi146 ceph-mon[103278]: pgmap v5516: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:50.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:23:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:23:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:50.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:50.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:23:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:23:50.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:23:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:52 smithi146 ceph-mon[103278]: pgmap v5517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:52 smithi078 ceph-mon[91136]: pgmap v5517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:52 smithi078 ceph-mon[99959]: pgmap v5517: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:54 smithi146 ceph-mon[103278]: pgmap v5518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:54 smithi078 ceph-mon[91136]: pgmap v5518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:54 smithi078 ceph-mon[99959]: pgmap v5518: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:56.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:56 smithi078 ceph-mon[99959]: pgmap v5519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:56.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:56 smithi078 ceph-mon[91136]: pgmap v5519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:56 smithi146 ceph-mon[103278]: pgmap v5519: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:23:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:57 smithi146 ceph-mon[103278]: pgmap v5520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[99959]: pgmap v5520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[91136]: pgmap v5520: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:23:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:23:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:23:59 smithi146 ceph-mon[103278]: pgmap v5521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:23:59 smithi078 ceph-mon[91136]: pgmap v5521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:23:59 smithi078 ceph-mon[99959]: pgmap v5521: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:00.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:24:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:24:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:00.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:00.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:00.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:02 smithi146 ceph-mon[103278]: pgmap v5522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[99959]: pgmap v5522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[91136]: pgmap v5522: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:04 smithi146 ceph-mon[103278]: pgmap v5523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:04 smithi078 ceph-mon[91136]: pgmap v5523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:04 smithi078 ceph-mon[99959]: pgmap v5523: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:06 smithi146 ceph-mon[103278]: pgmap v5524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:06 smithi078 ceph-mon[91136]: pgmap v5524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:06 smithi078 ceph-mon[99959]: pgmap v5524: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:08.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:08 smithi078 ceph-mon[91136]: pgmap v5525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:08.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:08 smithi078 ceph-mon[99959]: pgmap v5525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:08 smithi146 ceph-mon[103278]: pgmap v5525: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:10.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:10 smithi146 ceph-mon[103278]: pgmap v5526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:10 smithi078 ceph-mon[99959]: pgmap v5526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:10 smithi078 ceph-mon[91136]: pgmap v5526: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:24:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:24:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:10.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:10.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:10.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:12 smithi146 ceph-mon[103278]: pgmap v5527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:12 smithi078 ceph-mon[91136]: pgmap v5527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:12 smithi078 ceph-mon[99959]: pgmap v5527: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:14 smithi146 ceph-mon[103278]: pgmap v5528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:14 smithi078 ceph-mon[91136]: pgmap v5528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:14 smithi078 ceph-mon[99959]: pgmap v5528: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:15.326 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 3h - - 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 3h - - 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 19s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 14s ago 3h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 19s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 14s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:24:15.854 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 19s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 19s ago 3h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 14s ago 3h 216M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 19s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 19s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 14s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 19s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 19s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 19s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 19s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 14s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 14s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 14s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 14s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:24:15.855 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 14s ago 3h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:24:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 19s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:24:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 14s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:24:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 19s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:24:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 14s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:24:16.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:24:16.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:16.163 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:24:16.163 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:24:16.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:24:16.163 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:24:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:24:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:24:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:24:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:24:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:24:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:24:16.429 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:24:16.738 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:24:16.738 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:16.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:16.739 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:16.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:16.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:16 smithi078 ceph-mon[91136]: pgmap v5529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:16.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:16 smithi078 ceph-mon[91136]: from='client.56571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:16.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4042165729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:16.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:16 smithi078 ceph-mon[99959]: pgmap v5529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:16.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:16 smithi078 ceph-mon[99959]: from='client.56571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:16.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4042165729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:16 smithi146 ceph-mon[103278]: pgmap v5529: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:16 smithi146 ceph-mon[103278]: from='client.56571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4042165729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='client.76496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='client.76502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='client.76514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/548435495' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='client.76496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='client.76502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='client.76514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/548435495' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='client.76496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='client.76502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='client.76514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/548435495' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:17.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:18.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:18 smithi078 ceph-mon[99959]: pgmap v5530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:18.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:18 smithi078 ceph-mon[91136]: pgmap v5530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:18 smithi146 ceph-mon[103278]: pgmap v5530: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:20 smithi146 ceph-mon[103278]: pgmap v5531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:20 smithi078 ceph-mon[99959]: pgmap v5531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:20 smithi078 ceph-mon[91136]: pgmap v5531: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:24:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:24:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:20.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:20.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:20.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:22 smithi146 ceph-mon[103278]: pgmap v5532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[91136]: pgmap v5532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[99959]: pgmap v5532: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:24 smithi146 ceph-mon[103278]: pgmap v5533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:24 smithi078 ceph-mon[91136]: pgmap v5533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:24 smithi078 ceph-mon[99959]: pgmap v5533: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 23:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T23:24:25.84775464Z level=info msg="Completed cleanup jobs" duration=12.083876ms 2024-09-20T23:24:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:26 smithi146 ceph-mon[103278]: pgmap v5534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:26 smithi078 ceph-mon[99959]: pgmap v5534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:26 smithi078 ceph-mon[91136]: pgmap v5534: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:28.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:28 smithi078 ceph-mon[91136]: pgmap v5535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:28.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:28.763 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:28 smithi078 ceph-mon[99959]: pgmap v5535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:28.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:28 smithi146 ceph-mon[103278]: pgmap v5535: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:29.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:30 smithi146 ceph-mon[103278]: pgmap v5536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:30 smithi078 ceph-mon[91136]: pgmap v5536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:24:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:24:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:30 smithi078 ceph-mon[99959]: pgmap v5536: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:30.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:30.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:30.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:32 smithi078 ceph-mon[99959]: pgmap v5537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:32 smithi078 ceph-mon[91136]: pgmap v5537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:32 smithi146 ceph-mon[103278]: pgmap v5537: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:34 smithi078 ceph-mon[99959]: pgmap v5538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:34 smithi078 ceph-mon[91136]: pgmap v5538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:34 smithi146 ceph-mon[103278]: pgmap v5538: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:36 smithi078 ceph-mon[91136]: pgmap v5539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:36 smithi078 ceph-mon[99959]: pgmap v5539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:36 smithi146 ceph-mon[103278]: pgmap v5539: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:37 smithi146 ceph-mon[103278]: pgmap v5540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[91136]: pgmap v5540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[99959]: pgmap v5540: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:40.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:39 smithi146 ceph-mon[103278]: pgmap v5541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:39 smithi078 ceph-mon[91136]: pgmap v5541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:39 smithi078 ceph-mon[99959]: pgmap v5541: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:40.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:24:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:24:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:40.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:40.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:40.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:42 smithi146 ceph-mon[103278]: pgmap v5542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[91136]: pgmap v5542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[99959]: pgmap v5542: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:44 smithi146 ceph-mon[103278]: pgmap v5543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:44 smithi078 ceph-mon[91136]: pgmap v5543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:44 smithi078 ceph-mon[99959]: pgmap v5543: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:46 smithi146 ceph-mon[103278]: pgmap v5544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:46 smithi078 ceph-mon[99959]: pgmap v5544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:46 smithi078 ceph-mon[91136]: pgmap v5544: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:47.037 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 3h - - 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 3h - - 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 10s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 5s ago 3h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 10s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 5s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 10s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 10s ago 3h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 5s ago 3h 218M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 10s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:24:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 10s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 5s ago 3h 16.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (2h) 10s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 10s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 10s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 10s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 5s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 5s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 5s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 5s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 5s ago 3h 72.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 10s ago 3h 88.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 10s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:24:47.576 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:24:47.892 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:24:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:24:47.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:24:47.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:24:48.167 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:24:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:24:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:24:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:24:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:24:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:24:48.168 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:24:48.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[99959]: pgmap v5545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:48.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[99959]: from='client.76526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:48.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[99959]: from='client.76532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:48.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3028931266' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:48.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[91136]: pgmap v5545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:48.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[91136]: from='client.76526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:48.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[91136]: from='client.76532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:48.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3028931266' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:48 smithi146 ceph-mon[103278]: pgmap v5545: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:48 smithi146 ceph-mon[103278]: from='client.76526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:48 smithi146 ceph-mon[103278]: from='client.76532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3028931266' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:24:48.484 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:24:48.484 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:48.484 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:24:48.484 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:48.484 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:24:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[91136]: from='client.56619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[91136]: from='client.76550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:49.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4161119574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:49.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[99959]: from='client.56619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:49.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[99959]: from='client.76550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:49.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:49.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4161119574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:49 smithi146 ceph-mon[103278]: from='client.56619 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:49 smithi146 ceph-mon[103278]: from='client.76550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:24:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4161119574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:24:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:50 smithi078 ceph-mon[91136]: pgmap v5546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:50 smithi078 ceph-mon[99959]: pgmap v5546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:50.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:50 smithi146 ceph-mon[103278]: pgmap v5546: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:24:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:24:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:50.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:50.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:24:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:24:50.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:24:52.283 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:52 smithi146 ceph-mon[103278]: pgmap v5547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:52 smithi078 ceph-mon[91136]: pgmap v5547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:52 smithi078 ceph-mon[99959]: pgmap v5547: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:54 smithi078 ceph-mon[91136]: pgmap v5548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:54 smithi078 ceph-mon[99959]: pgmap v5548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:54 smithi146 ceph-mon[103278]: pgmap v5548: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:56 smithi078 ceph-mon[91136]: pgmap v5549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:56 smithi078 ceph-mon[99959]: pgmap v5549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:56 smithi146 ceph-mon[103278]: pgmap v5549: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:24:58.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[99959]: pgmap v5550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:58.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[91136]: pgmap v5550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:58.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:58 smithi146 ceph-mon[103278]: pgmap v5550: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:24:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:24:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:24:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:24:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:24:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:24:59.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:24:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:00 smithi146 ceph-mon[103278]: pgmap v5551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:00 smithi078 ceph-mon[99959]: pgmap v5551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:00 smithi078 ceph-mon[91136]: pgmap v5551: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:00.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:25:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:25:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:00.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:00.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:00.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:02 smithi146 ceph-mon[103278]: pgmap v5552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[91136]: pgmap v5552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[99959]: pgmap v5552: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:04 smithi146 ceph-mon[103278]: pgmap v5553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:04 smithi078 ceph-mon[91136]: pgmap v5553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:04 smithi078 ceph-mon[99959]: pgmap v5553: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:06 smithi146 ceph-mon[103278]: pgmap v5554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:06 smithi078 ceph-mon[91136]: pgmap v5554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:06 smithi078 ceph-mon[99959]: pgmap v5554: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:08.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:08 smithi078 ceph-mon[91136]: pgmap v5555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:08.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:08 smithi078 ceph-mon[99959]: pgmap v5555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:08 smithi146 ceph-mon[103278]: pgmap v5555: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:09.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:10.509 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:10 smithi078 ceph-mon[99959]: pgmap v5556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:10.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:10 smithi078 ceph-mon[91136]: pgmap v5556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:10 smithi146 ceph-mon[103278]: pgmap v5556: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:25:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:25:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:10.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:10.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:10.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:12.338 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:12 smithi146 ceph-mon[103278]: pgmap v5557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:12 smithi078 ceph-mon[91136]: pgmap v5557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:12 smithi078 ceph-mon[99959]: pgmap v5557: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:14 smithi146 ceph-mon[103278]: pgmap v5558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:14 smithi078 ceph-mon[91136]: pgmap v5558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:14 smithi078 ceph-mon[99959]: pgmap v5558: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:16 smithi146 ceph-mon[103278]: pgmap v5559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:16 smithi078 ceph-mon[91136]: pgmap v5559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:16 smithi078 ceph-mon[99959]: pgmap v5559: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:18 smithi146 ceph-mon[103278]: pgmap v5560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[91136]: pgmap v5560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[99959]: pgmap v5560: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:18.780 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 3h - - 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 3h - - 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 2s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 17s ago 3h 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 17s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 3h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 17s ago 3h 219M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 2s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:25:19.308 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 17s ago 3h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 2s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 2s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 2s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 17s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 17s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 17s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 17s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 17s ago 3h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 2s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 17s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 2s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:25:19.309 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 17s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:25:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:25:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:25:19.619 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:25:19.886 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:25:19.887 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:25:19.887 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:25:19.887 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:25:20.196 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:25:20.196 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:20.196 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:20.197 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:20.197 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:20 smithi146 ceph-mon[103278]: pgmap v5561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:20 smithi146 ceph-mon[103278]: from='client.56643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:20 smithi146 ceph-mon[103278]: from='client.76568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:20 smithi146 ceph-mon[103278]: from='client.76574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2118942221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3374202241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[99959]: pgmap v5561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[99959]: from='client.56643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[99959]: from='client.76568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[99959]: from='client.76574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2118942221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3374202241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[91136]: pgmap v5561: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[91136]: from='client.56643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[91136]: from='client.76568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[91136]: from='client.76574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2118942221' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:20.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3374202241' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:20.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:25:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:25:21.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:20.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:21.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:20.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:21.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:20.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:21 smithi078 ceph-mon[91136]: from='client.76586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:21 smithi078 ceph-mon[99959]: from='client.76586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:21 smithi146 ceph-mon[103278]: from='client.76586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[91136]: pgmap v5562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[99959]: pgmap v5562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:22 smithi146 ceph-mon[103278]: pgmap v5562: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:24 smithi078 ceph-mon[91136]: pgmap v5563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:24 smithi078 ceph-mon[99959]: pgmap v5563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:24 smithi146 ceph-mon[103278]: pgmap v5563: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:26 smithi078 ceph-mon[99959]: pgmap v5564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:26 smithi078 ceph-mon[91136]: pgmap v5564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:26 smithi146 ceph-mon[103278]: pgmap v5564: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:28 smithi078 ceph-mon[99959]: pgmap v5565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:28 smithi078 ceph-mon[91136]: pgmap v5565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:28.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:28 smithi146 ceph-mon[103278]: pgmap v5565: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:29.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:29.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:30.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:30 smithi078 ceph-mon[99959]: pgmap v5566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:30.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:30 smithi078 ceph-mon[91136]: pgmap v5566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:30.762 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:25:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:25:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:30 smithi146 ceph-mon[103278]: pgmap v5566: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:30.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:30.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:31.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:30.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:32.377 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:32 smithi146 ceph-mon[103278]: pgmap v5567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:32 smithi078 ceph-mon[91136]: pgmap v5567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:32 smithi078 ceph-mon[99959]: pgmap v5567: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:34 smithi146 ceph-mon[103278]: pgmap v5568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:34 smithi078 ceph-mon[91136]: pgmap v5568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:34 smithi078 ceph-mon[99959]: pgmap v5568: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:36 smithi146 ceph-mon[103278]: pgmap v5569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:36 smithi078 ceph-mon[91136]: pgmap v5569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:36 smithi078 ceph-mon[99959]: pgmap v5569: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:38 smithi146 ceph-mon[103278]: pgmap v5570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[91136]: pgmap v5570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[99959]: pgmap v5570: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:40 smithi146 ceph-mon[103278]: pgmap v5571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:40 smithi078 ceph-mon[99959]: pgmap v5571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:40 smithi078 ceph-mon[91136]: pgmap v5571: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:25:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:25:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:40.903Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:40.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:40.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:42 smithi146 ceph-mon[103278]: pgmap v5572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[91136]: pgmap v5572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[99959]: pgmap v5572: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:44 smithi146 ceph-mon[103278]: pgmap v5573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:44 smithi078 ceph-mon[91136]: pgmap v5573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:44 smithi078 ceph-mon[99959]: pgmap v5573: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:46 smithi146 ceph-mon[103278]: pgmap v5574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:46 smithi078 ceph-mon[91136]: pgmap v5574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:46 smithi078 ceph-mon[99959]: pgmap v5574: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:48 smithi146 ceph-mon[103278]: pgmap v5575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:48 smithi078 ceph-mon[91136]: pgmap v5575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:48 smithi078 ceph-mon[99959]: pgmap v5575: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:49.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:49.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:50.492 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:25:50.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:50 smithi078 ceph-mon[99959]: pgmap v5576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:50.760 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:25:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:25:50.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:50 smithi078 ceph-mon[91136]: pgmap v5576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:50 smithi146 ceph-mon[103278]: pgmap v5576: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:50.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:50.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:25:51.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:25:50.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:25:51.056 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:25:51.056 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 3h - - 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 3h - - 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 8s ago 3h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 8s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 3h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:25:51.057 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 8s ago 3h 220M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 8s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (2h) 13s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 13s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 13s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:25:51.058 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 8s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:25:51.059 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 8s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:25:51.059 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 8s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:25:51.059 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 8s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:25:51.059 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 8s ago 3h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:25:51.059 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 13s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:25:51.059 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 8s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:25:51.060 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 13s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:25:51.060 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 8s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:25:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:25:51.379 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:25:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:25:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:25:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:25:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:25:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:25:51.678 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:25:51.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:51 smithi078 ceph-mon[99959]: from='client.76598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:51.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1642449989' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:51.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:51 smithi078 ceph-mon[91136]: from='client.76598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:51.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1642449989' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:51 smithi146 ceph-mon[103278]: from='client.76598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1642449989' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:25:51.995 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:25:51.996 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:25:51.996 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:51.996 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:25:52.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:52 smithi146 ceph-mon[103278]: pgmap v5577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:52.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:52 smithi146 ceph-mon[103278]: from='client.56685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:52.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:52 smithi146 ceph-mon[103278]: from='client.56691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:52.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1273753142' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[91136]: pgmap v5577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[91136]: from='client.56685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[91136]: from='client.56691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1273753142' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[99959]: pgmap v5577: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[99959]: from='client.56685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[99959]: from='client.56691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1273753142' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:25:53.597 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:53 smithi146 ceph-mon[103278]: from='client.76622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:53 smithi078 ceph-mon[91136]: from='client.76622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:53 smithi078 ceph-mon[99959]: from='client.76622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:25:54.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:54 smithi146 ceph-mon[103278]: pgmap v5578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:54 smithi078 ceph-mon[91136]: pgmap v5578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:54 smithi078 ceph-mon[99959]: pgmap v5578: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:56 smithi146 ceph-mon[103278]: pgmap v5579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:56 smithi078 ceph-mon[91136]: pgmap v5579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:56 smithi078 ceph-mon[99959]: pgmap v5579: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:25:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[91136]: pgmap v5580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[99959]: pgmap v5580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:58.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:58 smithi146 ceph-mon[103278]: pgmap v5580: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:25:58.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:25:58.815 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:25:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:25:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:25:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:25:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:00.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:25:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:00 smithi078 ceph-mon[91136]: pgmap v5581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:00 smithi078 ceph-mon[99959]: pgmap v5581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:26:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:26:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:00 smithi146 ceph-mon[103278]: pgmap v5581: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:00.905Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:00.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:00.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:02 smithi146 ceph-mon[103278]: pgmap v5582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[91136]: pgmap v5582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[99959]: pgmap v5582: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:04 smithi146 ceph-mon[103278]: pgmap v5583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:04 smithi078 ceph-mon[91136]: pgmap v5583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:04 smithi078 ceph-mon[99959]: pgmap v5583: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:06 smithi146 ceph-mon[103278]: pgmap v5584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:07.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:06 smithi078 ceph-mon[91136]: pgmap v5584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:06 smithi078 ceph-mon[99959]: pgmap v5584: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:07 smithi146 ceph-mon[103278]: pgmap v5585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:07 smithi078 ceph-mon[91136]: pgmap v5585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:07 smithi078 ceph-mon[99959]: pgmap v5585: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:09 smithi146 ceph-mon[103278]: pgmap v5586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:09 smithi078 ceph-mon[91136]: pgmap v5586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:09 smithi078 ceph-mon[99959]: pgmap v5586: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:10.907 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:26:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:26:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:10.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:10.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:10.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:11 smithi078 ceph-mon[91136]: pgmap v5587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:11 smithi078 ceph-mon[99959]: pgmap v5587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:11 smithi146 ceph-mon[103278]: pgmap v5587: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:13 smithi078 ceph-mon[91136]: pgmap v5588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:13 smithi078 ceph-mon[99959]: pgmap v5588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:13.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:13 smithi146 ceph-mon[103278]: pgmap v5588: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:13.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:15.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:15 smithi146 ceph-mon[103278]: pgmap v5589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:15 smithi078 ceph-mon[91136]: pgmap v5589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:15 smithi078 ceph-mon[99959]: pgmap v5589: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:17 smithi146 ceph-mon[103278]: pgmap v5590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[91136]: pgmap v5590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[99959]: pgmap v5590: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:19 smithi146 ceph-mon[103278]: pgmap v5591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:19 smithi078 ceph-mon[99959]: pgmap v5591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:19 smithi078 ceph-mon[91136]: pgmap v5591: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:20.908 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:26:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:26:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:20.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:20.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:20.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:21 smithi146 ceph-mon[103278]: pgmap v5592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:21 smithi078 ceph-mon[91136]: pgmap v5592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:21 smithi078 ceph-mon[99959]: pgmap v5592: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:22.297 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:26:22.826 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:26:22.826 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 3h - - 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 3h - - 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 5s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 0s ago 3h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 0s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 5s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 5s ago 3h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 0s ago 3h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 5s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 5s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 0s ago 3h 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 5s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 5s ago 3h 81.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:26:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (2h) 5s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 5s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 0s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 0s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 0s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 0s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 0s ago 3h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 5s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 0s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 5s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:26:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 0s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:26:23.137 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:26:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:26:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:26:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:26:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:26:23.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:26:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:26:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:26:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:26:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:23.140 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:26:23.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:26:23.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:26:23.140 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:26:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:26:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:26:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:26:23.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:26:23.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[99959]: from='client.76634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:23.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4293722254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:23.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[91136]: from='client.76634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:23.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4293722254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:23.725 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:26:23.726 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:23.726 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:23.726 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:23 smithi146 ceph-mon[103278]: from='client.76634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4293722254' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:24 smithi146 ceph-mon[103278]: from='client.76640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:24 smithi146 ceph-mon[103278]: pgmap v5593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:24 smithi146 ceph-mon[103278]: from='client.76646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:24 smithi146 ceph-mon[103278]: from='client.76658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3131327260' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[91136]: from='client.76640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[91136]: pgmap v5593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[91136]: from='client.76646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[91136]: from='client.76658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3131327260' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[99959]: from='client.76640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[99959]: pgmap v5593: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[99959]: from='client.76646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[99959]: from='client.76658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3131327260' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:26 smithi146 ceph-mon[103278]: pgmap v5594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:26 smithi078 ceph-mon[91136]: pgmap v5594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:26 smithi078 ceph-mon[99959]: pgmap v5594: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:28 smithi146 ceph-mon[103278]: pgmap v5595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:28 smithi078 ceph-mon[91136]: pgmap v5595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:28 smithi078 ceph-mon[99959]: pgmap v5595: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:30 smithi146 ceph-mon[103278]: pgmap v5596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:30 smithi078 ceph-mon[91136]: pgmap v5596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:30 smithi078 ceph-mon[99959]: pgmap v5596: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:26:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:26:31.218 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:30.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:31.218 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:30.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:31.218 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:30.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:32.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:32 smithi078 ceph-mon[99959]: pgmap v5597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:32.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:32 smithi078 ceph-mon[91136]: pgmap v5597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:32 smithi146 ceph-mon[103278]: pgmap v5597: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:34.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:34 smithi146 ceph-mon[103278]: pgmap v5598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:34 smithi078 ceph-mon[91136]: pgmap v5598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:34 smithi078 ceph-mon[99959]: pgmap v5598: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:36 smithi146 ceph-mon[103278]: pgmap v5599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:36 smithi078 ceph-mon[91136]: pgmap v5599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:36 smithi078 ceph-mon[99959]: pgmap v5599: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:38 smithi146 ceph-mon[103278]: pgmap v5600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[91136]: pgmap v5600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[99959]: pgmap v5600: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:40 smithi146 ceph-mon[103278]: pgmap v5601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:40 smithi078 ceph-mon[91136]: pgmap v5601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:40 smithi078 ceph-mon[99959]: pgmap v5601: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:26:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:26:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:40.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:40.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:40.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:42 smithi146 ceph-mon[103278]: pgmap v5602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:42 smithi078 ceph-mon[99959]: pgmap v5602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:42 smithi078 ceph-mon[91136]: pgmap v5602: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:44 smithi146 ceph-mon[103278]: pgmap v5603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:44 smithi078 ceph-mon[99959]: pgmap v5603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:44 smithi078 ceph-mon[91136]: pgmap v5603: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:46 smithi146 ceph-mon[103278]: pgmap v5604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:46 smithi078 ceph-mon[91136]: pgmap v5604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:46 smithi078 ceph-mon[99959]: pgmap v5604: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:48 smithi146 ceph-mon[103278]: pgmap v5605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:48 smithi078 ceph-mon[91136]: pgmap v5605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:48 smithi078 ceph-mon[99959]: pgmap v5605: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:50 smithi146 ceph-mon[103278]: pgmap v5606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:50 smithi078 ceph-mon[91136]: pgmap v5606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:50 smithi078 ceph-mon[99959]: pgmap v5606: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:26:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:26:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:50.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:50.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:26:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:26:50.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:26:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:52 smithi146 ceph-mon[103278]: pgmap v5607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:52 smithi078 ceph-mon[91136]: pgmap v5607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:52 smithi078 ceph-mon[99959]: pgmap v5607: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:54.016 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:26:54.549 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 3h - - 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 3h - - 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 16s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 12s ago 3h 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 3h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 16s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 16s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 12s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:26:54.550 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 16s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 16s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 16s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (2h) 16s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 12s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 12s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 12s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 12s ago 3h 73.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 3h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 16s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 12s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 16s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:26:54.551 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 12s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:26:54.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:54 smithi146 ceph-mon[103278]: pgmap v5608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:54 smithi078 ceph-mon[91136]: pgmap v5608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:54 smithi078 ceph-mon[99959]: pgmap v5608: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:54.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:26:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:26:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:26:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:26:54.862 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:26:55.130 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:26:55.131 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:26:55.437 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:26:55.438 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:26:55.438 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:55.438 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:26:55.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:55 smithi146 ceph-mon[103278]: from='client.56751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.698 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:55 smithi146 ceph-mon[103278]: from='client.56757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.698 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:55 smithi146 ceph-mon[103278]: from='client.76682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.698 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1067161952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:55.698 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1546835468' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[91136]: from='client.56751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[91136]: from='client.56757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[91136]: from='client.76682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1067161952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1546835468' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[99959]: from='client.56751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[99959]: from='client.56757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[99959]: from='client.76682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1067161952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:26:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1546835468' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:26:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:56 smithi146 ceph-mon[103278]: pgmap v5609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:56.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:56 smithi146 ceph-mon[103278]: from='client.56775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:56 smithi078 ceph-mon[99959]: pgmap v5609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:56 smithi078 ceph-mon[99959]: from='client.56775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:56 smithi078 ceph-mon[91136]: pgmap v5609: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:26:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:56 smithi078 ceph-mon[91136]: from='client.56775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[99959]: pgmap v5610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[91136]: pgmap v5610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:58.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:58 smithi146 ceph-mon[103278]: pgmap v5610: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:26:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:26:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:26:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:26:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:26:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:26:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:26:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:00 smithi078 ceph-mon[91136]: pgmap v5611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:27:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:27:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:00 smithi078 ceph-mon[99959]: pgmap v5611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:00 smithi146 ceph-mon[103278]: pgmap v5611: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:00.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:00.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:00.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:02 smithi146 ceph-mon[103278]: pgmap v5612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:02 smithi078 ceph-mon[99959]: pgmap v5612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:02 smithi078 ceph-mon[91136]: pgmap v5612: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:03 smithi146 ceph-mon[103278]: pgmap v5613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[99959]: pgmap v5613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[91136]: pgmap v5613: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:05 smithi146 ceph-mon[103278]: pgmap v5614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:05 smithi078 ceph-mon[91136]: pgmap v5614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:05 smithi078 ceph-mon[99959]: pgmap v5614: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:07 smithi146 ceph-mon[103278]: pgmap v5615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:07 smithi078 ceph-mon[91136]: pgmap v5615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:08.041 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:07 smithi078 ceph-mon[99959]: pgmap v5615: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:09 smithi146 ceph-mon[103278]: pgmap v5616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:09 smithi078 ceph-mon[99959]: pgmap v5616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:09 smithi078 ceph-mon[91136]: pgmap v5616: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:10.911 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:27:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:27:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:10.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:10.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:10.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:11 smithi146 ceph-mon[103278]: pgmap v5617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:11 smithi078 ceph-mon[91136]: pgmap v5617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:11 smithi078 ceph-mon[99959]: pgmap v5617: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:13.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:13 smithi078 ceph-mon[91136]: pgmap v5618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:13.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:13.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:13 smithi078 ceph-mon[99959]: pgmap v5618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:13.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:13 smithi146 ceph-mon[103278]: pgmap v5618: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:15 smithi146 ceph-mon[103278]: pgmap v5619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:15 smithi078 ceph-mon[99959]: pgmap v5619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:15 smithi078 ceph-mon[91136]: pgmap v5619: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:17 smithi146 ceph-mon[103278]: pgmap v5620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:17 smithi078 ceph-mon[91136]: pgmap v5620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:17 smithi078 ceph-mon[99959]: pgmap v5620: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:20.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:20.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-mon[99959]: pgmap v5621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:20.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:20.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-mon[91136]: pgmap v5621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:20 smithi146 ceph-mon[103278]: pgmap v5621: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:27:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:27:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:20.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:20.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:20.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:22 smithi078 ceph-mon[91136]: pgmap v5622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:22 smithi078 ceph-mon[99959]: pgmap v5622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:22 smithi146 ceph-mon[103278]: pgmap v5622: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:23 smithi146 ceph-mon[103278]: pgmap v5623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[91136]: pgmap v5623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[99959]: pgmap v5623: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:25.733 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:27:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:25 smithi146 ceph-mon[103278]: pgmap v5624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:25 smithi078 ceph-mon[99959]: pgmap v5624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:25 smithi078 ceph-mon[91136]: pgmap v5624: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:26.256 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 3h - - 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 3h - - 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 8s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 3s ago 3h 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 8s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 3s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 8s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 8s ago 3h 350M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 3s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 8s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 8s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 3s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 8s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:27:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 8s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 8s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 8s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (2h) 3s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 3s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 3s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 3s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 3s ago 3h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 8s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 3s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 8s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:27:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 3s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:27:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:27:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:27:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:27:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:27:26.567 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:27:26.837 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:27:26.837 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:27:26.837 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:27:26.837 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:27:26.838 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:27:26.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[99959]: from='client.56787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[99959]: from='client.56793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[99959]: from='client.76718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2278436493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[91136]: from='client.56787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[91136]: from='client.56793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[91136]: from='client.76718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:26.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2278436493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:26 smithi146 ceph-mon[103278]: from='client.56787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:26 smithi146 ceph-mon[103278]: from='client.56793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:26 smithi146 ceph-mon[103278]: from='client.76718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2278436493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:27:27.145 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:27.146 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:27 smithi146 ceph-mon[103278]: pgmap v5625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:27 smithi146 ceph-mon[103278]: from='client.56811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1965155717' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:27 smithi078 ceph-mon[99959]: pgmap v5625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:27 smithi078 ceph-mon[99959]: from='client.56811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1965155717' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:27 smithi078 ceph-mon[91136]: pgmap v5625: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:28.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:27 smithi078 ceph-mon[91136]: from='client.56811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:28.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1965155717' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:29 smithi146 ceph-mon[103278]: pgmap v5626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:29 smithi078 ceph-mon[99959]: pgmap v5626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:29 smithi078 ceph-mon[91136]: pgmap v5626: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:30.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:27:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:27:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:30.911Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:30.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:30.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:31 smithi078 ceph-mon[91136]: pgmap v5627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:31 smithi078 ceph-mon[99959]: pgmap v5627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:31 smithi146 ceph-mon[103278]: pgmap v5627: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:34.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:33 smithi078 ceph-mon[99959]: pgmap v5628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:34.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:33 smithi078 ceph-mon[91136]: pgmap v5628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:33 smithi146 ceph-mon[103278]: pgmap v5628: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:35.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:35 smithi146 ceph-mon[103278]: pgmap v5629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:36.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:35 smithi078 ceph-mon[91136]: pgmap v5629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:36.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:35 smithi078 ceph-mon[99959]: pgmap v5629: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:37 smithi078 ceph-mon[91136]: pgmap v5630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:37 smithi078 ceph-mon[99959]: pgmap v5630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:37 smithi146 ceph-mon[103278]: pgmap v5630: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:40 smithi146 ceph-mon[103278]: pgmap v5631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-mon[99959]: pgmap v5631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-mon[91136]: pgmap v5631: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:40.913 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:27:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:27:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:40.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:40.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:40.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:42 smithi146 ceph-mon[103278]: pgmap v5632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:42 smithi078 ceph-mon[99959]: pgmap v5632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:42 smithi078 ceph-mon[91136]: pgmap v5632: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:43 smithi146 ceph-mon[103278]: pgmap v5633: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[99959]: pgmap v5633: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[91136]: pgmap v5633: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:45 smithi146 ceph-mon[103278]: pgmap v5634: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:45 smithi078 ceph-mon[91136]: pgmap v5634: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:45 smithi078 ceph-mon[99959]: pgmap v5634: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:47 smithi146 ceph-mon[103278]: pgmap v5635: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:47 smithi078 ceph-mon[91136]: pgmap v5635: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:47 smithi078 ceph-mon[99959]: pgmap v5635: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:49 smithi146 ceph-mon[103278]: pgmap v5636: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:49 smithi078 ceph-mon[91136]: pgmap v5636: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:27:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:49 smithi078 ceph-mon[99959]: pgmap v5636: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:50.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:27:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:27:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:50.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:50.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:27:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:27:50.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:27:52.156 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:51 smithi078 ceph-mon[99959]: pgmap v5637: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:52.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:51 smithi078 ceph-mon[91136]: pgmap v5637: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:51 smithi146 ceph-mon[103278]: pgmap v5637: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:53 smithi146 ceph-mon[103278]: pgmap v5638: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:53 smithi078 ceph-mon[91136]: pgmap v5638: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:53 smithi078 ceph-mon[99959]: pgmap v5638: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:55 smithi078 ceph-mon[91136]: pgmap v5639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:55 smithi078 ceph-mon[99959]: pgmap v5639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:55 smithi146 ceph-mon[103278]: pgmap v5639: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:27:57.441 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:27:57.970 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:27:57.970 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 3h - - 2024-09-20T23:27:57.970 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 3h - - 2024-09-20T23:27:57.970 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 19s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 15s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 19s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 19s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 19s ago 3h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 3h 221M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 19s ago 3h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 19s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 15s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 19s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 19s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 19s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:27:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 19s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 3h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 15s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 15s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 15s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 3h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 19s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 15s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 19s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:27:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 15s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:27:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:57 smithi146 ceph-mon[103278]: pgmap v5640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:57 smithi146 ceph-mon[103278]: from='client.56823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:57 smithi078 ceph-mon[99959]: pgmap v5640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:57 smithi078 ceph-mon[99959]: from='client.56823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:57 smithi078 ceph-mon[91136]: pgmap v5640: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:27:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:57 smithi078 ceph-mon[91136]: from='client.56823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:58.302 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:27:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:27:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:27:58.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:27:58.304 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:27:58.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:27:58.574 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:27:58.882 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:27:58.882 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:27:58.882 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:27:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='client.56829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='client.56835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3522018300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:59.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:27:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/114300451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='client.56829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='client.56835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3522018300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/114300451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='client.56829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='client.56835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:27:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3522018300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:27:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:27:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:27:59.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:27:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/114300451' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:00.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-mon[91136]: from='client.76766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:00.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:00.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-mon[91136]: pgmap v5641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:00.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-mon[99959]: from='client.76766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:00.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:00.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-mon[99959]: pgmap v5641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:00.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:00 smithi146 ceph-mon[103278]: from='client.76766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:00.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:00.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:00 smithi146 ceph-mon[103278]: pgmap v5641: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:00.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:28:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:28:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:00.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:00.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:00.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:02.601 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:02 smithi078 ceph-mon[91136]: pgmap v5642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:02.601 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:02 smithi078 ceph-mon[99959]: pgmap v5642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:02 smithi146 ceph-mon[103278]: pgmap v5642: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:03 smithi146 ceph-mon[103278]: pgmap v5643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:03 smithi078 ceph-mon[91136]: pgmap v5643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:03 smithi078 ceph-mon[99959]: pgmap v5643: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:06 smithi146 ceph-mon[103278]: pgmap v5644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:06 smithi078 ceph-mon[99959]: pgmap v5644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:06 smithi078 ceph-mon[91136]: pgmap v5644: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:08 smithi146 ceph-mon[103278]: pgmap v5645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:08 smithi078 ceph-mon[91136]: pgmap v5645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:08 smithi078 ceph-mon[99959]: pgmap v5645: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:10 smithi146 ceph-mon[103278]: pgmap v5646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-mon[99959]: pgmap v5646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-mon[91136]: pgmap v5646: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:10.916 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:28:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:28:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:10.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:10.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:10.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:12 smithi146 ceph-mon[103278]: pgmap v5647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:12 smithi078 ceph-mon[99959]: pgmap v5647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:12 smithi078 ceph-mon[91136]: pgmap v5647: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:14 smithi078 ceph-mon[99959]: pgmap v5648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:14 smithi078 ceph-mon[91136]: pgmap v5648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:14 smithi146 ceph-mon[103278]: pgmap v5648: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:16 smithi078 ceph-mon[91136]: pgmap v5649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:16 smithi078 ceph-mon[99959]: pgmap v5649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:16 smithi146 ceph-mon[103278]: pgmap v5649: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:18 smithi078 ceph-mon[91136]: pgmap v5650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:18 smithi078 ceph-mon[99959]: pgmap v5650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:18.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:18 smithi146 ceph-mon[103278]: pgmap v5650: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:20.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:20.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-mon[91136]: pgmap v5651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:20.710 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:28:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:28:20.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:20.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-mon[99959]: pgmap v5651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:20 smithi146 ceph-mon[103278]: pgmap v5651: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:20.915Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:20.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:21.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:20.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:22 smithi146 ceph-mon[103278]: pgmap v5652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:22 smithi078 ceph-mon[91136]: pgmap v5652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:22 smithi078 ceph-mon[99959]: pgmap v5652: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:24 smithi146 ceph-mon[103278]: pgmap v5653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[99959]: pgmap v5653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[91136]: pgmap v5653: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:26 smithi146 ceph-mon[103278]: pgmap v5654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:26 smithi078 ceph-mon[99959]: pgmap v5654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:26 smithi078 ceph-mon[91136]: pgmap v5654: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:28 smithi146 ceph-mon[103278]: pgmap v5655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:28 smithi078 ceph-mon[91136]: pgmap v5655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:28 smithi078 ceph-mon[99959]: pgmap v5655: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:29.180 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:28:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 3h - - 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 3h - - 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 11s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 6s ago 3h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 11s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:28:29.706 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 6s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 11s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 11s ago 3h 347M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 6s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 11s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 11s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 6s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 11s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 11s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 11s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 11s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 6s ago 3h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (2h) 6s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 6s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:28:29.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 6s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:28:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 6s ago 3h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:28:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 11s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:28:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 6s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:28:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 11s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:28:29.708 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 6s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:28:30.018 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:28:30.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:28:30.018 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:28:30.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:28:30.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:28:30.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:28:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:28:30.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:28:30.602 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:28:30.602 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:28:30.602 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:28:30.602 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:28:30.603 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:28:30.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:30.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[99959]: pgmap v5656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:30.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[99959]: from='client.56859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:30.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[99959]: from='client.56865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:30.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4054887754' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:30.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:30.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[91136]: pgmap v5656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:30.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[91136]: from='client.56859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:30.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[91136]: from='client.56865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:30.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4054887754' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:30 smithi146 ceph-mon[103278]: pgmap v5656: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:30 smithi146 ceph-mon[103278]: from='client.56859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:30 smithi146 ceph-mon[103278]: from='client.56865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4054887754' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:28:30.916 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:28:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:28:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:30.915Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:30.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:30.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:31 smithi146 ceph-mon[103278]: from='client.56871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:31 smithi146 ceph-mon[103278]: from='client.76802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3163170071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:31 smithi078 ceph-mon[91136]: from='client.56871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:31 smithi078 ceph-mon[91136]: from='client.76802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3163170071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:31 smithi078 ceph-mon[99959]: from='client.56871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:31 smithi078 ceph-mon[99959]: from='client.76802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:28:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3163170071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:28:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:32 smithi146 ceph-mon[103278]: pgmap v5657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:32 smithi078 ceph-mon[91136]: pgmap v5657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:32 smithi078 ceph-mon[99959]: pgmap v5657: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:34 smithi146 ceph-mon[103278]: pgmap v5658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:34.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:34 smithi078 ceph-mon[91136]: pgmap v5658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:34.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:34 smithi078 ceph-mon[99959]: pgmap v5658: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:36.673 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:36 smithi078 ceph-mon[99959]: pgmap v5659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:36.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:36 smithi078 ceph-mon[91136]: pgmap v5659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:36 smithi146 ceph-mon[103278]: pgmap v5659: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:38.662 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:38 smithi146 ceph-mon[103278]: pgmap v5660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:38 smithi078 ceph-mon[91136]: pgmap v5660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:38 smithi078 ceph-mon[99959]: pgmap v5660: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:40.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:40.918 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-mon[99959]: pgmap v5661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:40.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:40.918 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-mon[91136]: pgmap v5661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:40.918 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:28:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:28:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:40 smithi146 ceph-mon[103278]: pgmap v5661: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:40.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:40.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:40.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:42 smithi146 ceph-mon[103278]: pgmap v5662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:42 smithi078 ceph-mon[91136]: pgmap v5662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:42 smithi078 ceph-mon[99959]: pgmap v5662: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[91136]: pgmap v5663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[99959]: pgmap v5663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:44 smithi146 ceph-mon[103278]: pgmap v5663: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:46 smithi078 ceph-mon[91136]: pgmap v5664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:46 smithi078 ceph-mon[99959]: pgmap v5664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:46 smithi146 ceph-mon[103278]: pgmap v5664: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:48 smithi078 ceph-mon[91136]: pgmap v5665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:48 smithi078 ceph-mon[99959]: pgmap v5665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:48 smithi146 ceph-mon[103278]: pgmap v5665: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-mon[99959]: pgmap v5666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-mon[91136]: pgmap v5666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:28:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:50 smithi146 ceph-mon[103278]: pgmap v5666: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:50.918 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:28:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:28:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:50.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:50.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:28:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:28:50.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:28:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:52 smithi078 ceph-mon[91136]: pgmap v5667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:52 smithi078 ceph-mon[99959]: pgmap v5667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:52 smithi146 ceph-mon[103278]: pgmap v5667: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:54 smithi078 ceph-mon[91136]: pgmap v5668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:54 smithi078 ceph-mon[99959]: pgmap v5668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:54 smithi146 ceph-mon[103278]: pgmap v5668: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:56.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:56 smithi078 ceph-mon[99959]: pgmap v5669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:56.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:56 smithi078 ceph-mon[91136]: pgmap v5669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:56 smithi146 ceph-mon[103278]: pgmap v5669: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:28:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:58 smithi146 ceph-mon[103278]: pgmap v5670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:58.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:58 smithi078 ceph-mon[99959]: pgmap v5670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:58.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:58 smithi078 ceph-mon[91136]: pgmap v5670: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:28:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:28:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:28:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:28:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:28:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:00 smithi146 ceph-mon[103278]: pgmap v5671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-mon[99959]: pgmap v5671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-mon[91136]: pgmap v5671: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:00.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:29:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:29:00.897 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:29:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:00.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:00.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:00.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 3h - - 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 3h - - 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 2s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 18s ago 3h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 18s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 3h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 18s ago 3h 223M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:29:01.423 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 2s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 18s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 2s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 2s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 2s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 18s ago 3h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 18s ago 3h 72.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 18s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 18s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 18s ago 3h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 2s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 18s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 2s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:29:01.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 18s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:29:01.734 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:29:01.734 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:29:01.735 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:29:02.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:29:02.002 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:29:02.003 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:29:02.316 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:29:02.316 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:02.316 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:02.316 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:02.316 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:02 smithi146 ceph-mon[103278]: pgmap v5672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:02 smithi146 ceph-mon[103278]: from='client.56895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:02 smithi146 ceph-mon[103278]: from='client.56901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:02 smithi146 ceph-mon[103278]: from='client.76826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1734663779' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[91136]: pgmap v5672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[91136]: from='client.56895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[91136]: from='client.56901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[91136]: from='client.76826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1734663779' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[99959]: pgmap v5672: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[99959]: from='client.56895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[99959]: from='client.56901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[99959]: from='client.76826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1734663779' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:03 smithi146 ceph-mon[103278]: from='client.76838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1135964557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:03 smithi078 ceph-mon[91136]: from='client.76838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1135964557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:03 smithi078 ceph-mon[99959]: from='client.76838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1135964557' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:04 smithi146 ceph-mon[103278]: pgmap v5673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[91136]: pgmap v5673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[99959]: pgmap v5673: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:06 smithi146 ceph-mon[103278]: pgmap v5674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:06 smithi078 ceph-mon[91136]: pgmap v5674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:06 smithi078 ceph-mon[99959]: pgmap v5674: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:08.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:08 smithi078 ceph-mon[91136]: pgmap v5675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:08.655 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:08 smithi078 ceph-mon[99959]: pgmap v5675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:08 smithi146 ceph-mon[103278]: pgmap v5675: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:10 smithi146 ceph-mon[103278]: pgmap v5676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-mon[91136]: pgmap v5676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:29:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:29:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-mon[99959]: pgmap v5676: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:10.918Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:10.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:10.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:12 smithi146 ceph-mon[103278]: pgmap v5677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:12 smithi078 ceph-mon[91136]: pgmap v5677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:12 smithi078 ceph-mon[99959]: pgmap v5677: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:14 smithi146 ceph-mon[103278]: pgmap v5678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:14 smithi078 ceph-mon[91136]: pgmap v5678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:14 smithi078 ceph-mon[99959]: pgmap v5678: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:16 smithi078 ceph-mon[91136]: pgmap v5679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:16 smithi078 ceph-mon[99959]: pgmap v5679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:16 smithi146 ceph-mon[103278]: pgmap v5679: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:18.648 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:18 smithi078 ceph-mon[99959]: pgmap v5680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:18.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:18 smithi078 ceph-mon[91136]: pgmap v5680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:18.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:18 smithi146 ceph-mon[103278]: pgmap v5680: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:19 smithi146 ceph-mon[103278]: pgmap v5681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[91136]: pgmap v5681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[99959]: pgmap v5681: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:20.920 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:29:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:29:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:20.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:20.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:20.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:22.160 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:21 smithi078 ceph-mon[91136]: pgmap v5682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:22.160 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:21 smithi078 ceph-mon[99959]: pgmap v5682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:21 smithi146 ceph-mon[103278]: pgmap v5682: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[99959]: pgmap v5683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[91136]: pgmap v5683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:24.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:24.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:24.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:24 smithi146 ceph-mon[103278]: pgmap v5683: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:26 smithi078 ceph-mon[99959]: pgmap v5684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:26 smithi078 ceph-mon[91136]: pgmap v5684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:26 smithi146 ceph-mon[103278]: pgmap v5684: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:28 smithi078 ceph-mon[91136]: pgmap v5685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:28 smithi078 ceph-mon[99959]: pgmap v5685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:28.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:28 smithi146 ceph-mon[103278]: pgmap v5685: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:28.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:30.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-mon[91136]: pgmap v5686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:30.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:30.921 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:29:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:29:30.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-mon[99959]: pgmap v5686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:30.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:30 smithi146 ceph-mon[103278]: pgmap v5686: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:30.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:30.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:30.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:32.618 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:29:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:32 smithi146 ceph-mon[103278]: pgmap v5687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:32 smithi078 ceph-mon[91136]: pgmap v5687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:32 smithi078 ceph-mon[99959]: pgmap v5687: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 3h - - 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 3h - - 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 14s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 9s ago 3h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 14s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 9s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 14s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 14s ago 3h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 9s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:29:33.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 14s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 14s ago 3h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 9s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 14s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 14s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 14s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 14s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 9s ago 3h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 9s ago 3h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (2h) 9s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 9s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 9s ago 3h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 14s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 9s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 14s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:29:33.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 9s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:29:33.459 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:29:33.459 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:29:33.460 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:29:33.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:29:33.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:29:33.461 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:29:33.734 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:29:33.734 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:29:33.734 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:29:33.735 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:29:33.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/769022011' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:33.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/769022011' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/769022011' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:29:34.044 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:29:34.044 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:34.044 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:29:34.044 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:34.044 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:29:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:34 smithi146 ceph-mon[103278]: from='client.56931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:34 smithi146 ceph-mon[103278]: pgmap v5688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:34 smithi146 ceph-mon[103278]: from='client.56937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:34 smithi146 ceph-mon[103278]: from='client.76862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3427684400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[91136]: from='client.56931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[91136]: pgmap v5688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[91136]: from='client.56937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[91136]: from='client.76862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3427684400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[99959]: from='client.56931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[99959]: pgmap v5688: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[99959]: from='client.56937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[99959]: from='client.76862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3427684400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:29:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:35 smithi146 ceph-mon[103278]: from='client.76874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:35 smithi078 ceph-mon[99959]: from='client.76874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:35 smithi078 ceph-mon[91136]: from='client.76874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:29:36.824 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:36 smithi078 ceph-mon[99959]: pgmap v5689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:36.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:36 smithi078 ceph-mon[91136]: pgmap v5689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:36 smithi146 ceph-mon[103278]: pgmap v5689: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:38.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:38 smithi078 ceph-mon[91136]: pgmap v5690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:38.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:38 smithi078 ceph-mon[99959]: pgmap v5690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:38 smithi146 ceph-mon[103278]: pgmap v5690: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:39 smithi146 ceph-mon[103278]: pgmap v5691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[91136]: pgmap v5691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:40.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:40.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.243 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[99959]: pgmap v5691: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:40.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:40.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:40.921 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:29:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:29:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:40.920Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:40.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:40.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:41.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:41 smithi146 ceph-mon[103278]: pgmap v5692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:41 smithi078 ceph-mon[91136]: pgmap v5692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:41 smithi078 ceph-mon[99959]: pgmap v5692: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:43 smithi146 ceph-mon[103278]: pgmap v5693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[91136]: pgmap v5693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[99959]: pgmap v5693: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:29:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:45 smithi146 ceph-mon[103278]: pgmap v5694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:45 smithi078 ceph-mon[91136]: pgmap v5694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:45 smithi078 ceph-mon[99959]: pgmap v5694: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:47 smithi146 ceph-mon[103278]: pgmap v5695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:47 smithi078 ceph-mon[99959]: pgmap v5695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:47 smithi078 ceph-mon[91136]: pgmap v5695: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:49 smithi146 ceph-mon[103278]: pgmap v5696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:49 smithi078 ceph-mon[91136]: pgmap v5696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:49 smithi078 ceph-mon[99959]: pgmap v5696: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:29:50.922 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:29:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:29:51.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:50.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:51.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:50.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:29:51.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:29:50.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:29:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:51 smithi146 ceph-mon[103278]: pgmap v5697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:51 smithi078 ceph-mon[99959]: pgmap v5697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:51 smithi078 ceph-mon[91136]: pgmap v5697: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:53 smithi146 ceph-mon[103278]: pgmap v5698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:53 smithi078 ceph-mon[99959]: pgmap v5698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:53 smithi078 ceph-mon[91136]: pgmap v5698: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:55 smithi146 ceph-mon[103278]: pgmap v5699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:55 smithi078 ceph-mon[91136]: pgmap v5699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:55 smithi078 ceph-mon[99959]: pgmap v5699: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:29:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:57 smithi146 ceph-mon[103278]: pgmap v5700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:57 smithi078 ceph-mon[91136]: pgmap v5700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:57 smithi078 ceph-mon[99959]: pgmap v5700: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:29:59.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:59.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:29:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:00.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T23:29:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[91136]: pgmap v5701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[99959]: pgmap v5701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:00 smithi146 ceph-mon[103278]: pgmap v5701: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:00.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:00.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:00.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:00.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:30:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:30:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:00.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:00.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:00.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:02.356 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:02 smithi146 ceph-mon[103278]: pgmap v5702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:02 smithi078 ceph-mon[91136]: pgmap v5702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:02 smithi078 ceph-mon[99959]: pgmap v5702: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:04.337 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:30:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:04 smithi146 ceph-mon[103278]: pgmap v5703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[91136]: pgmap v5703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[99959]: pgmap v5703: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 3h - - 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 3h - - 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 5s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 1s ago 3h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 1s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 5s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 5s ago 3h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 1s ago 3h 222M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:30:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 5s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 5s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 1s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 5s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 5s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 5s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 5s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 1s ago 3h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 1s ago 3h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 1s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (2h) 1s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 1s ago 3h 73.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 5s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:30:04.867 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 5s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:30:04.868 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:30:05.174 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:30:05.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:30:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:30:05.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:30:05.176 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:30:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:30:05.445 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:30:05.757 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:05.757 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:30:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:30:05.757 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:05.757 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:05.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:05 smithi078 ceph-mon[99959]: from='client.76886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:05.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3005650311' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:05.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:05 smithi078 ceph-mon[91136]: from='client.76886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:05.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3005650311' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:05 smithi146 ceph-mon[103278]: from='client.76886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3005650311' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:06 smithi146 ceph-mon[103278]: from='client.56973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:06 smithi146 ceph-mon[103278]: pgmap v5704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:06 smithi146 ceph-mon[103278]: from='client.56979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:06 smithi146 ceph-mon[103278]: from='client.76910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/922357893' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[99959]: from='client.56973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[99959]: pgmap v5704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[99959]: from='client.56979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[99959]: from='client.76910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/922357893' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[91136]: from='client.56973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[91136]: pgmap v5704: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[91136]: from='client.56979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[91136]: from='client.76910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:06.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/922357893' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:08.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:08 smithi078 ceph-mon[91136]: pgmap v5705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:08.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:08 smithi078 ceph-mon[99959]: pgmap v5705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:08 smithi146 ceph-mon[103278]: pgmap v5705: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:10 smithi146 ceph-mon[103278]: pgmap v5706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-mon[91136]: pgmap v5706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:30:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:30:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-mon[99959]: pgmap v5706: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:10.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:10.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:10.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:12 smithi146 ceph-mon[103278]: pgmap v5707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:12 smithi078 ceph-mon[99959]: pgmap v5707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:12 smithi078 ceph-mon[91136]: pgmap v5707: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:14 smithi146 ceph-mon[103278]: pgmap v5708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:14 smithi078 ceph-mon[91136]: pgmap v5708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:14 smithi078 ceph-mon[99959]: pgmap v5708: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:16 smithi146 ceph-mon[103278]: pgmap v5709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:16 smithi078 ceph-mon[91136]: pgmap v5709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:16 smithi078 ceph-mon[99959]: pgmap v5709: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:18 smithi146 ceph-mon[103278]: pgmap v5710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:18 smithi078 ceph-mon[91136]: pgmap v5710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:18 smithi078 ceph-mon[99959]: pgmap v5710: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:20.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[91136]: pgmap v5711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:20.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:20.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[99959]: pgmap v5711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:20.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:20.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.476 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:20 smithi146 ceph-mon[103278]: pgmap v5711: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:20.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:20.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:30:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:30:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:20.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:20.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:20.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:22.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:22 smithi146 ceph-mon[103278]: pgmap v5712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:22 smithi078 ceph-mon[91136]: pgmap v5712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:22 smithi078 ceph-mon[99959]: pgmap v5712: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:24 smithi146 ceph-mon[103278]: pgmap v5713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[91136]: pgmap v5713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[99959]: pgmap v5713: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:26 smithi146 ceph-mon[103278]: pgmap v5714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:26 smithi078 ceph-mon[91136]: pgmap v5714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:26 smithi078 ceph-mon[99959]: pgmap v5714: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:28 smithi146 ceph-mon[103278]: pgmap v5715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:28 smithi078 ceph-mon[91136]: pgmap v5715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:28 smithi078 ceph-mon[99959]: pgmap v5715: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:30 smithi146 ceph-mon[103278]: pgmap v5716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-mon[91136]: pgmap v5716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-mon[99959]: pgmap v5716: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:30.922 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:30:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:30:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:30.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:30.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:30.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:32 smithi146 ceph-mon[103278]: pgmap v5717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:32 smithi078 ceph-mon[91136]: pgmap v5717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:32 smithi078 ceph-mon[99959]: pgmap v5717: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:34 smithi146 ceph-mon[103278]: pgmap v5718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:34 smithi078 ceph-mon[91136]: pgmap v5718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:34 smithi078 ceph-mon[99959]: pgmap v5718: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:36.052 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:30:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:36 smithi146 ceph-mon[103278]: pgmap v5719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:36 smithi078 ceph-mon[91136]: pgmap v5719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:36 smithi078 ceph-mon[99959]: pgmap v5719: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:30:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 3h - - 2024-09-20T23:30:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 3h - - 2024-09-20T23:30:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 17s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:30:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 12s ago 3h 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:30:36.581 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 17s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 17s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 17s ago 3h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 3h 224M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 17s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 17s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 12s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 17s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 17s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 17s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 17s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 12s ago 3h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 12s ago 3h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:30:36.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 12s ago 3h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:30:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 12s ago 3h 73.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:30:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 3h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:30:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (2h) 17s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:30:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (2h) 12s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:30:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (2h) 17s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:30:36.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (2h) 12s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:30:36.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:30:36.890 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:30:37.154 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:30:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:30:37.468 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:30:37.468 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:30:37.468 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:30:37.468 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:37.468 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:30:37.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[99959]: from='client.76922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[99959]: from='client.57009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[99959]: from='client.57015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/793448105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[91136]: from='client.76922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[91136]: from='client.57009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[91136]: from='client.57015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/793448105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:37 smithi146 ceph-mon[103278]: from='client.76922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:37 smithi146 ceph-mon[103278]: from='client.57009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:37 smithi146 ceph-mon[103278]: from='client.57015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/793448105' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:30:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:38 smithi146 ceph-mon[103278]: pgmap v5720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:38 smithi146 ceph-mon[103278]: from='client.57027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2269507791' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:38 smithi078 ceph-mon[91136]: pgmap v5720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:38 smithi078 ceph-mon[91136]: from='client.57027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2269507791' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:38 smithi078 ceph-mon[99959]: pgmap v5720: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:38 smithi078 ceph-mon[99959]: from='client.57027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:30:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2269507791' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[91136]: pgmap v5721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[99959]: pgmap v5721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:40.339 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.340 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.340 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:40 smithi146 ceph-mon[103278]: pgmap v5721: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:40.927 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:30:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:30:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:40.925Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:40.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:40.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:42.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:42 smithi146 ceph-mon[103278]: pgmap v5722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:42 smithi078 ceph-mon[99959]: pgmap v5722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:42 smithi078 ceph-mon[91136]: pgmap v5722: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:44 smithi146 ceph-mon[103278]: pgmap v5723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[91136]: pgmap v5723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[99959]: pgmap v5723: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:30:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:46 smithi146 ceph-mon[103278]: pgmap v5724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:46 smithi078 ceph-mon[91136]: pgmap v5724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:46 smithi078 ceph-mon[99959]: pgmap v5724: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:48.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:48 smithi078 ceph-mon[91136]: pgmap v5725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:48.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:48 smithi078 ceph-mon[99959]: pgmap v5725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:48 smithi146 ceph-mon[103278]: pgmap v5725: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-mon[99959]: pgmap v5726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-mon[91136]: pgmap v5726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:50 smithi146 ceph-mon[103278]: pgmap v5726: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:30:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:30:50.927 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:30:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:30:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:50.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:50.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:30:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:30:50.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:30:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:52 smithi146 ceph-mon[103278]: pgmap v5727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:52 smithi078 ceph-mon[91136]: pgmap v5727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:52 smithi078 ceph-mon[99959]: pgmap v5727: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:54 smithi146 ceph-mon[103278]: pgmap v5728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:54 smithi078 ceph-mon[91136]: pgmap v5728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:54 smithi078 ceph-mon[99959]: pgmap v5728: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:30:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:56 smithi146 ceph-mon[103278]: pgmap v5729: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:30:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:56 smithi078 ceph-mon[99959]: pgmap v5729: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:30:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:56 smithi078 ceph-mon[91136]: pgmap v5729: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:30:58.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:58 smithi078 ceph-mon[99959]: pgmap v5730: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T23:30:58.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:58.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:58 smithi078 ceph-mon[91136]: pgmap v5730: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T23:30:58.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:30:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:58 smithi146 ceph-mon[103278]: pgmap v5730: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-20T23:30:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:00 smithi146 ceph-mon[103278]: pgmap v5731: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T23:31:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:31:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[91136]: pgmap v5731: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[99959]: pgmap v5731: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:01.181 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:00.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:01.181 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:00.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:01.182 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:00.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:02.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:02 smithi078 ceph-mon[91136]: pgmap v5732: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T23:31:02.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:02 smithi078 ceph-mon[99959]: pgmap v5732: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T23:31:02.716 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:02 smithi146 ceph-mon[103278]: pgmap v5732: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T23:31:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:04 smithi146 ceph-mon[103278]: pgmap v5733: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T23:31:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[91136]: pgmap v5733: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[99959]: pgmap v5733: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 93 op/s 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:06 smithi146 ceph-mon[103278]: pgmap v5734: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:31:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:06 smithi078 ceph-mon[91136]: pgmap v5734: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:31:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:06 smithi078 ceph-mon[99959]: pgmap v5734: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-20T23:31:07.785 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:31:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:31:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 3h - - 2024-09-20T23:31:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 3h - - 2024-09-20T23:31:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 8s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:31:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 4s ago 3h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:31:08.338 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 8s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 4s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 8s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 8s ago 3h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 4s ago 3h 225M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 8s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 8s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 4s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 8s ago 3h 78.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 8s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 8s ago 3h 71.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 8s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 4s ago 3h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:31:08.339 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 4s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 4s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 4s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 4s ago 3h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 8s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 8s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:31:08.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:31:08.663 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:31:08.663 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:31:08.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:31:08.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:31:08.665 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:31:08.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:08 smithi078 ceph-mon[91136]: pgmap v5735: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-20T23:31:08.665 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:08 smithi078 ceph-mon[99959]: pgmap v5735: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-20T23:31:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:08 smithi146 ceph-mon[103278]: pgmap v5735: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-20T23:31:08.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:31:08.934 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:31:08.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:31:09.248 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:31:09.248 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:09.248 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:09.248 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:09.249 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[99959]: from='client.57039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[99959]: from='client.76964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[99959]: from='client.76970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/800984632' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2084069070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[91136]: from='client.57039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[91136]: from='client.76964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[91136]: from='client.76970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/800984632' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2084069070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:09 smithi146 ceph-mon[103278]: from='client.57039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:09 smithi146 ceph-mon[103278]: from='client.76964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:09 smithi146 ceph-mon[103278]: from='client.76970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/800984632' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2084069070' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:10 smithi146 ceph-mon[103278]: pgmap v5736: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T23:31:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:10.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:10 smithi146 ceph-mon[103278]: from='client.76982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-mon[91136]: pgmap v5736: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T23:31:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-mon[91136]: from='client.76982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:31:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-mon[99959]: pgmap v5736: 161 pgs: 161 active+clean; 585 KiB data, 259 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s 2024-09-20T23:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-mon[99959]: from='client.76982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:10.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:10.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:10.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:12 smithi146 ceph-mon[103278]: pgmap v5737: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 189 op/s 2024-09-20T23:31:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:12 smithi078 ceph-mon[91136]: pgmap v5737: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 189 op/s 2024-09-20T23:31:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:12 smithi078 ceph-mon[99959]: pgmap v5737: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 0 B/s wr, 189 op/s 2024-09-20T23:31:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:14 smithi146 ceph-mon[103278]: pgmap v5738: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T23:31:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:14 smithi078 ceph-mon[91136]: pgmap v5738: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T23:31:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:14 smithi078 ceph-mon[99959]: pgmap v5738: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-20T23:31:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:16 smithi146 ceph-mon[103278]: pgmap v5739: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T23:31:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:16 smithi078 ceph-mon[91136]: pgmap v5739: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T23:31:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:16 smithi078 ceph-mon[99959]: pgmap v5739: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-20T23:31:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:18 smithi146 ceph-mon[103278]: pgmap v5740: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 158 op/s 2024-09-20T23:31:18.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:18 smithi078 ceph-mon[99959]: pgmap v5740: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 158 op/s 2024-09-20T23:31:18.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:18 smithi078 ceph-mon[91136]: pgmap v5740: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 158 op/s 2024-09-20T23:31:20.890 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:31:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[99959]: pgmap v5741: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[91136]: pgmap v5741: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:20 smithi146 ceph-mon[103278]: pgmap v5741: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-20T23:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:21.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:20.927Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:21.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:20.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:21.147 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:20.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:21.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:21 smithi078 ceph-mon[91136]: pgmap v5742: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:31:21.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:21 smithi078 ceph-mon[99959]: pgmap v5742: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:31:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:21 smithi146 ceph-mon[103278]: pgmap v5742: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-20T23:31:23.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:23 smithi146 ceph-mon[103278]: pgmap v5743: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:31:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:23 smithi078 ceph-mon[91136]: pgmap v5743: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:31:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:23 smithi078 ceph-mon[99959]: pgmap v5743: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-20T23:31:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:26 smithi146 ceph-mon[103278]: pgmap v5744: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T23:31:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:26 smithi078 ceph-mon[91136]: pgmap v5744: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T23:31:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:26 smithi078 ceph-mon[99959]: pgmap v5744: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-20T23:31:28.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:28 smithi146 ceph-mon[103278]: pgmap v5745: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:31:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:28 smithi078 ceph-mon[91136]: pgmap v5745: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:31:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:28 smithi078 ceph-mon[99959]: pgmap v5745: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:31:29.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-mon[91136]: pgmap v5746: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:31:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-mon[99959]: pgmap v5746: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:31:30.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:30.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:30 smithi146 ceph-mon[103278]: pgmap v5746: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:31:30.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:30.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:31:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:31:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:30.928Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:30.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:30.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:32 smithi078 ceph-mon[99959]: pgmap v5747: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:32 smithi078 ceph-mon[91136]: pgmap v5747: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:32 smithi146 ceph-mon[103278]: pgmap v5747: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:34 smithi146 ceph-mon[103278]: pgmap v5748: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:34 smithi078 ceph-mon[99959]: pgmap v5748: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:34 smithi078 ceph-mon[91136]: pgmap v5748: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:36 smithi146 ceph-mon[103278]: pgmap v5749: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:36 smithi078 ceph-mon[99959]: pgmap v5749: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:36 smithi078 ceph-mon[91136]: pgmap v5749: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:38 smithi146 ceph-mon[103278]: pgmap v5750: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:38 smithi078 ceph-mon[99959]: pgmap v5750: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:38 smithi078 ceph-mon[91136]: pgmap v5750: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:39.543 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 3h - - 2024-09-20T23:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 3h - - 2024-09-20T23:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 0s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 15s ago 3h 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 0s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 0s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 0s ago 3h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 3h 226M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 0s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 0s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 15s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 0s ago 3h 79.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 0s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 0s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 0s ago 3h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 3h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 15s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 15s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 15s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 3h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 0s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 15s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 0s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 15s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:31:40.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:31:40.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:31:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:40 smithi146 ceph-mon[103278]: pgmap v5751: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:40 smithi146 ceph-mon[103278]: from='client.57075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[91136]: pgmap v5751: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[91136]: from='client.57075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[99959]: pgmap v5751: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[99959]: from='client.57075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:40.671 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:31:40.671 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:31:40.671 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:31:40.671 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:31:40.672 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:31:40.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:31:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:31:40.981 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:31:40.981 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:40.981 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:31:40.981 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:40.981 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:31:41.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:40.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:41.203 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:40.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:41.203 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:40.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:41.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[99959]: from='client.57081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:41.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[99959]: from='client.57087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:41.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3276818611' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:41.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3546003782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:41.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[91136]: from='client.57081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:41.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[91136]: from='client.57087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:41.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3276818611' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:41.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3546003782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:41 smithi146 ceph-mon[103278]: from='client.57081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:41 smithi146 ceph-mon[103278]: from='client.57087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3276818611' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:31:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3546003782' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:31:42.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:42 smithi078 ceph-mon[91136]: from='client.57099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:42.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:42 smithi078 ceph-mon[91136]: pgmap v5752: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:42.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:42 smithi078 ceph-mon[99959]: from='client.57099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:42.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:42 smithi078 ceph-mon[99959]: pgmap v5752: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:42 smithi146 ceph-mon[103278]: from='client.57099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:31:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:42 smithi146 ceph-mon[103278]: pgmap v5752: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:43.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:43.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:44.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:44 smithi146 ceph-mon[103278]: pgmap v5753: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:44 smithi078 ceph-mon[91136]: pgmap v5753: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:44 smithi078 ceph-mon[99959]: pgmap v5753: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:31:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:46 smithi146 ceph-mon[103278]: pgmap v5754: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:46 smithi078 ceph-mon[91136]: pgmap v5754: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:46 smithi078 ceph-mon[99959]: pgmap v5754: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:48 smithi146 ceph-mon[103278]: pgmap v5755: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:48 smithi078 ceph-mon[91136]: pgmap v5755: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:48 smithi078 ceph-mon[99959]: pgmap v5755: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:50 smithi146 ceph-mon[103278]: pgmap v5756: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-mon[91136]: pgmap v5756: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:31:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:31:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-mon[99959]: pgmap v5756: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:31:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:50.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:50.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:31:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:31:50.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:31:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:52 smithi146 ceph-mon[103278]: pgmap v5757: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:52 smithi078 ceph-mon[91136]: pgmap v5757: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:52 smithi078 ceph-mon[99959]: pgmap v5757: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:54 smithi146 ceph-mon[103278]: pgmap v5758: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:54.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:54 smithi078 ceph-mon[91136]: pgmap v5758: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:54.879 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:54 smithi078 ceph-mon[99959]: pgmap v5758: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:56 smithi146 ceph-mon[103278]: pgmap v5759: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:56 smithi078 ceph-mon[91136]: pgmap v5759: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:56 smithi078 ceph-mon[99959]: pgmap v5759: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:31:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:58 smithi146 ceph-mon[103278]: pgmap v5760: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:58.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:31:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:58 smithi078 ceph-mon[91136]: pgmap v5760: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:31:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:31:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:58 smithi078 ceph-mon[99959]: pgmap v5760: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:31:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:31:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:00 smithi146 ceph-mon[103278]: pgmap v5761: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:32:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[91136]: pgmap v5761: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[99959]: pgmap v5761: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:00.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:00.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:00.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:02 smithi146 ceph-mon[103278]: pgmap v5762: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:02 smithi078 ceph-mon[91136]: pgmap v5762: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:02 smithi078 ceph-mon[99959]: pgmap v5762: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:04 smithi146 ceph-mon[103278]: pgmap v5763: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:04 smithi078 ceph-mon[91136]: pgmap v5763: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:04 smithi078 ceph-mon[99959]: pgmap v5763: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:06.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:06 smithi146 ceph-mon[103278]: pgmap v5764: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:06 smithi078 ceph-mon[91136]: pgmap v5764: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:06 smithi078 ceph-mon[99959]: pgmap v5764: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:08.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:08 smithi078 ceph-mon[91136]: pgmap v5765: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:08.818 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:08 smithi078 ceph-mon[99959]: pgmap v5765: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:08 smithi146 ceph-mon[103278]: pgmap v5765: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:10.933 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:32:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:32:10.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-mon[99959]: pgmap v5766: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:10.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:10.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-mon[91136]: pgmap v5766: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:10.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:10 smithi146 ceph-mon[103278]: pgmap v5766: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:11.277 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:10.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:11.277 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:10.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:11.277 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:10.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:11.277 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:32:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 3h - - 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 3h - - 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 11s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 7s ago 3h 87.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 11s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 7s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 11s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 11s ago 3h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 7s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 11s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 11s ago 3h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 7s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:32:11.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 11s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 11s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 11s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 11s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 7s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 7s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 7s ago 3h 78.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 7s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 7s ago 3h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 11s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 11s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:32:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 7s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:32:12.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:32:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:32:12.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:32:12.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:32:12.384 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:32:12.385 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:12.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[99959]: pgmap v5767: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[99959]: from='client.57111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[99959]: from='client.77036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2536954313' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[91136]: pgmap v5767: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[91136]: from='client.57111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[91136]: from='client.77036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:12.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2536954313' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:12 smithi146 ceph-mon[103278]: pgmap v5767: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:12 smithi146 ceph-mon[103278]: from='client.57111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:12 smithi146 ceph-mon[103278]: from='client.77036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2536954313' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:13 smithi146 ceph-mon[103278]: from='client.57123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:13 smithi146 ceph-mon[103278]: from='client.57135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4286293475' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[91136]: from='client.57123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[91136]: from='client.57135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4286293475' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[99959]: from='client.57123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[99959]: from='client.57135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4286293475' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:14 smithi146 ceph-mon[103278]: pgmap v5768: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:14 smithi078 ceph-mon[91136]: pgmap v5768: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:14 smithi078 ceph-mon[99959]: pgmap v5768: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:16 smithi146 ceph-mon[103278]: pgmap v5769: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:16 smithi078 ceph-mon[91136]: pgmap v5769: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:16 smithi078 ceph-mon[99959]: pgmap v5769: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:18 smithi146 ceph-mon[103278]: pgmap v5770: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:18 smithi078 ceph-mon[91136]: pgmap v5770: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:18 smithi078 ceph-mon[99959]: pgmap v5770: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:20.934 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:32:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[91136]: pgmap v5771: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[99959]: pgmap v5771: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:20.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:20 smithi146 ceph-mon[103278]: pgmap v5771: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:20.933Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:20.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:20.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:22.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:22 smithi078 ceph-mon[99959]: pgmap v5772: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:22.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:22 smithi078 ceph-mon[91136]: pgmap v5772: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:22 smithi146 ceph-mon[103278]: pgmap v5772: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:24.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:24 smithi078 ceph-mon[99959]: pgmap v5773: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:24.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:24 smithi078 ceph-mon[91136]: pgmap v5773: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:24.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:24 smithi146 ceph-mon[103278]: pgmap v5773: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:25 smithi146 ceph-mon[103278]: pgmap v5774: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[91136]: pgmap v5774: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:25 smithi078 ceph-mon[99959]: pgmap v5774: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:27 smithi146 ceph-mon[103278]: pgmap v5775: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:27 smithi078 ceph-mon[91136]: pgmap v5775: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:27 smithi078 ceph-mon[99959]: pgmap v5775: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:30 smithi146 ceph-mon[103278]: pgmap v5776: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-mon[91136]: pgmap v5776: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-mon[99959]: pgmap v5776: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:32:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:32:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:30.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:30.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:30.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:32 smithi146 ceph-mon[103278]: pgmap v5777: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:32 smithi078 ceph-mon[91136]: pgmap v5777: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:32 smithi078 ceph-mon[99959]: pgmap v5777: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:34 smithi146 ceph-mon[103278]: pgmap v5778: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:34 smithi078 ceph-mon[99959]: pgmap v5778: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:34 smithi078 ceph-mon[91136]: pgmap v5778: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:36 smithi146 ceph-mon[103278]: pgmap v5779: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:36 smithi078 ceph-mon[91136]: pgmap v5779: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:36 smithi078 ceph-mon[99959]: pgmap v5779: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:38 smithi146 ceph-mon[103278]: pgmap v5780: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:38 smithi078 ceph-mon[91136]: pgmap v5780: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:38 smithi078 ceph-mon[99959]: pgmap v5780: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:40 smithi146 ceph-mon[103278]: pgmap v5781: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:32:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[91136]: pgmap v5781: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[99959]: pgmap v5781: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:40.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:40.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:40.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:42 smithi146 ceph-mon[103278]: pgmap v5782: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:42 smithi078 ceph-mon[91136]: pgmap v5782: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:42 smithi078 ceph-mon[99959]: pgmap v5782: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:42.992 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:32:43.544 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:32:43.544 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 3h - - 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 3h - - 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 3s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 18s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 3s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 18s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 3s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 3s ago 3h 354M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 18s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 3s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 3s ago 3h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 18s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 3s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 3s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 3s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 3s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:32:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 18s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 18s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 18s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 18s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 18s ago 3h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 3s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 18s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 3s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:32:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 18s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:32:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:32:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:32:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:32:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:32:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:32:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:32:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:32:44.133 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:32:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:32:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:32:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:32:44.465 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:32:44.465 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:32:44.466 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:32:44.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[99959]: pgmap v5783: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:44.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[99959]: from='client.77066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[99959]: from='client.57153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[99959]: from='client.57159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1410543107' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:44.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/47263831' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[91136]: pgmap v5783: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[91136]: from='client.77066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[91136]: from='client.57153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[91136]: from='client.57159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1410543107' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:44.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/47263831' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:44 smithi146 ceph-mon[103278]: pgmap v5783: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:44 smithi146 ceph-mon[103278]: from='client.77066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:44 smithi146 ceph-mon[103278]: from='client.57153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:44 smithi146 ceph-mon[103278]: from='client.57159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1410543107' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:32:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/47263831' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:32:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:45 smithi146 ceph-mon[103278]: from='client.57171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[91136]: from='client.57171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[99959]: from='client.57171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:32:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:46 smithi146 ceph-mon[103278]: pgmap v5784: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:46 smithi078 ceph-mon[91136]: pgmap v5784: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:46 smithi078 ceph-mon[99959]: pgmap v5784: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:48 smithi146 ceph-mon[103278]: pgmap v5785: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:48 smithi078 ceph-mon[91136]: pgmap v5785: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:48 smithi078 ceph-mon[99959]: pgmap v5785: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-mon[91136]: pgmap v5786: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-mon[99959]: pgmap v5786: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:32:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:32:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:50 smithi146 ceph-mon[103278]: pgmap v5786: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:32:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:50.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:50.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:32:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:32:50.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:32:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:52 smithi078 ceph-mon[91136]: pgmap v5787: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:52 smithi078 ceph-mon[99959]: pgmap v5787: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:52 smithi146 ceph-mon[103278]: pgmap v5787: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:54 smithi078 ceph-mon[91136]: pgmap v5788: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:54 smithi078 ceph-mon[99959]: pgmap v5788: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:54 smithi146 ceph-mon[103278]: pgmap v5788: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:56 smithi078 ceph-mon[91136]: pgmap v5789: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:56 smithi078 ceph-mon[99959]: pgmap v5789: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:56 smithi146 ceph-mon[103278]: pgmap v5789: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:32:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:58 smithi078 ceph-mon[91136]: pgmap v5790: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:58 smithi078 ceph-mon[99959]: pgmap v5790: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:32:58.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:58 smithi146 ceph-mon[103278]: pgmap v5790: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:32:58.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:00.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:33:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:33:00.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[99959]: pgmap v5791: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[91136]: pgmap v5791: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:00.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:00 smithi146 ceph-mon[103278]: pgmap v5791: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:00.936Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:00.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:00.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:02 smithi078 ceph-mon[91136]: pgmap v5792: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:02 smithi078 ceph-mon[99959]: pgmap v5792: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:02 smithi146 ceph-mon[103278]: pgmap v5792: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:04.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:04 smithi146 ceph-mon[103278]: pgmap v5793: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:04 smithi078 ceph-mon[91136]: pgmap v5793: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:04 smithi078 ceph-mon[99959]: pgmap v5793: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:06.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[91136]: pgmap v5794: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:06.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[99959]: pgmap v5794: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:06.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:05 smithi146 ceph-mon[103278]: pgmap v5794: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:07 smithi146 ceph-mon[103278]: pgmap v5795: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:07 smithi078 ceph-mon[91136]: pgmap v5795: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:07 smithi078 ceph-mon[99959]: pgmap v5795: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:09 smithi146 ceph-mon[103278]: pgmap v5796: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:09 smithi078 ceph-mon[99959]: pgmap v5796: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:09 smithi078 ceph-mon[91136]: pgmap v5796: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:10.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:33:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:33:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:10.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:10.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:10.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:11 smithi146 ceph-mon[103278]: pgmap v5797: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:11 smithi078 ceph-mon[91136]: pgmap v5797: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:11 smithi078 ceph-mon[99959]: pgmap v5797: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:13 smithi146 ceph-mon[103278]: pgmap v5798: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:13 smithi078 ceph-mon[91136]: pgmap v5798: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:13 smithi078 ceph-mon[99959]: pgmap v5798: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:14.766 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 3h - - 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 15s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 15s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 15s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 15s ago 3h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:33:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 15s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 15s ago 3h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 15s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 15s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 15s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 15s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 10s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 15s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:33:15.288 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 15s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:33:15.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:33:15.597 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:33:15.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:33:15.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:33:15.599 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:33:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:33:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:33:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:33:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:33:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:33:15.867 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:33:16.173 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:33:16.173 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:16.174 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:16.174 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:16.174 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:16.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[99959]: pgmap v5799: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:16.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[99959]: from='client.77102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[99959]: from='client.57189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[99959]: from='client.57195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3971443133' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:16.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[91136]: pgmap v5799: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:16.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[91136]: from='client.77102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[91136]: from='client.57189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[91136]: from='client.57195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3971443133' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:15 smithi146 ceph-mon[103278]: pgmap v5799: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:15 smithi146 ceph-mon[103278]: from='client.77102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:15 smithi146 ceph-mon[103278]: from='client.57189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:15 smithi146 ceph-mon[103278]: from='client.57195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3971443133' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:16 smithi146 ceph-mon[103278]: from='client.57207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3805760488' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:16 smithi078 ceph-mon[91136]: from='client.57207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3805760488' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:16 smithi078 ceph-mon[99959]: from='client.57207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3805760488' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:17 smithi146 ceph-mon[103278]: pgmap v5800: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:17 smithi078 ceph-mon[91136]: pgmap v5800: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:17 smithi078 ceph-mon[99959]: pgmap v5800: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:19 smithi146 ceph-mon[103278]: pgmap v5801: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:19 smithi078 ceph-mon[99959]: pgmap v5801: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:19 smithi078 ceph-mon[91136]: pgmap v5801: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:33:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:33:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:20.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:20.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:20.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:22 smithi146 ceph-mon[103278]: pgmap v5802: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:22 smithi078 ceph-mon[91136]: pgmap v5802: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:22 smithi078 ceph-mon[99959]: pgmap v5802: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:24.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:24 smithi078 ceph-mon[91136]: pgmap v5803: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:24.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:24 smithi078 ceph-mon[99959]: pgmap v5803: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:24 smithi146 ceph-mon[103278]: pgmap v5803: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:26.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[91136]: pgmap v5804: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:26.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[99959]: pgmap v5804: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:26.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:26 smithi146 ceph-mon[103278]: pgmap v5804: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:28 smithi146 ceph-mon[103278]: pgmap v5805: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:28 smithi078 ceph-mon[91136]: pgmap v5805: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:28 smithi078 ceph-mon[99959]: pgmap v5805: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:29.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:30.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:30 smithi146 ceph-mon[103278]: pgmap v5806: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:30.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-mon[91136]: pgmap v5806: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:30.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-mon[99959]: pgmap v5806: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:30.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:33:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:33:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:30.939Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:30.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:30.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:32 smithi146 ceph-mon[103278]: pgmap v5807: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:32 smithi078 ceph-mon[99959]: pgmap v5807: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:32 smithi078 ceph-mon[91136]: pgmap v5807: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:34 smithi146 ceph-mon[103278]: pgmap v5808: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:34 smithi078 ceph-mon[99959]: pgmap v5808: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:34 smithi078 ceph-mon[91136]: pgmap v5808: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:36 smithi146 ceph-mon[103278]: pgmap v5809: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:36 smithi078 ceph-mon[99959]: pgmap v5809: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:36 smithi078 ceph-mon[91136]: pgmap v5809: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:38 smithi146 ceph-mon[103278]: pgmap v5810: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:38 smithi078 ceph-mon[91136]: pgmap v5810: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:38 smithi078 ceph-mon[99959]: pgmap v5810: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:40 smithi146 ceph-mon[103278]: pgmap v5811: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:40.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-mon[99959]: pgmap v5811: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:40.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:40.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-mon[91136]: pgmap v5811: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:40.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:33:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:33:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:40.940Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:40.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:40.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:42 smithi078 ceph-mon[91136]: pgmap v5812: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:42 smithi078 ceph-mon[99959]: pgmap v5812: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:42 smithi146 ceph-mon[103278]: pgmap v5812: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:44 smithi146 ceph-mon[103278]: pgmap v5813: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:44 smithi078 ceph-mon[91136]: pgmap v5813: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:44 smithi078 ceph-mon[99959]: pgmap v5813: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:46.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[91136]: pgmap v5814: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:46.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[99959]: pgmap v5814: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:46.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.470 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:33:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:46 smithi146 ceph-mon[103278]: pgmap v5814: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:33:47.027 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:33:47.027 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 3h - - 2024-09-20T23:33:47.027 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 3h - - 2024-09-20T23:33:47.027 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 6s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:33:47.027 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 1s ago 3h 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 6s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 1s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 6s ago 3h 574M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 6s ago 3h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 1s ago 3h 227M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 6s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 6s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 1s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 6s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 6s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 6s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 6s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:33:47.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 1s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 1s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 1s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 1s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 1s ago 3h 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 6s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 6s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:33:47.029 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 1s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:33:47.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:47 smithi078 ceph-mon[99959]: from='client.77135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:47.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:47 smithi078 ceph-mon[91136]: from='client.77135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:47.370 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:33:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:33:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:33:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:33:47.372 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:33:47.487 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:47 smithi146 ceph-mon[103278]: from='client.77135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:33:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:33:47.651 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:33:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:33:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:33:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[91136]: from='client.77141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[91136]: pgmap v5815: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[91136]: from='client.57231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3949546238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/16037390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[99959]: from='client.77141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[99959]: pgmap v5815: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[99959]: from='client.57231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3949546238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/16037390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:48 smithi146 ceph-mon[103278]: from='client.77141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:48 smithi146 ceph-mon[103278]: pgmap v5815: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:48 smithi146 ceph-mon[103278]: from='client.57231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3949546238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:33:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/16037390' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:33:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:49 smithi146 ceph-mon[103278]: from='client.57243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:49 smithi078 ceph-mon[91136]: from='client.57243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:49 smithi078 ceph-mon[99959]: from='client.57243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:33:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:50 smithi146 ceph-mon[103278]: pgmap v5816: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-mon[91136]: pgmap v5816: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-mon[99959]: pgmap v5816: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:33:50.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:33:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:33:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:50.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:50.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:33:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:33:50.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:33:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:52 smithi146 ceph-mon[103278]: pgmap v5817: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:52 smithi078 ceph-mon[91136]: pgmap v5817: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:52 smithi078 ceph-mon[99959]: pgmap v5817: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:54 smithi146 ceph-mon[103278]: pgmap v5818: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:54 smithi078 ceph-mon[91136]: pgmap v5818: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:54 smithi078 ceph-mon[99959]: pgmap v5818: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:56 smithi146 ceph-mon[103278]: pgmap v5819: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:56 smithi078 ceph-mon[91136]: pgmap v5819: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:56 smithi078 ceph-mon[99959]: pgmap v5819: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:33:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:58 smithi146 ceph-mon[103278]: pgmap v5820: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:58 smithi078 ceph-mon[91136]: pgmap v5820: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:58 smithi078 ceph-mon[99959]: pgmap v5820: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:33:59.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:33:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:00 smithi146 ceph-mon[103278]: pgmap v5821: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-mon[99959]: pgmap v5821: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-mon[91136]: pgmap v5821: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:00.943 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:34:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:34:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:00.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:00.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:00.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[99959]: pgmap v5822: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:01 smithi078 ceph-mon[91136]: pgmap v5822: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:01 smithi146 ceph-mon[103278]: pgmap v5822: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:03 smithi146 ceph-mon[103278]: pgmap v5823: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:03 smithi078 ceph-mon[99959]: pgmap v5823: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:03 smithi078 ceph-mon[91136]: pgmap v5823: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[91136]: pgmap v5824: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[99959]: pgmap v5824: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:06 smithi146 ceph-mon[103278]: pgmap v5824: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:08.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:08 smithi146 ceph-mon[103278]: pgmap v5825: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:08 smithi078 ceph-mon[91136]: pgmap v5825: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:08 smithi078 ceph-mon[99959]: pgmap v5825: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-mon[91136]: pgmap v5826: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-mon[99959]: pgmap v5826: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:10 smithi146 ceph-mon[103278]: pgmap v5826: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:10.944 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:34:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:34:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:10.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:10.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:10.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:12 smithi146 ceph-mon[103278]: pgmap v5827: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:12 smithi078 ceph-mon[91136]: pgmap v5827: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:12 smithi078 ceph-mon[99959]: pgmap v5827: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:14 smithi146 ceph-mon[103278]: pgmap v5828: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:14 smithi078 ceph-mon[99959]: pgmap v5828: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:14 smithi078 ceph-mon[91136]: pgmap v5828: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:16 smithi146 ceph-mon[103278]: pgmap v5829: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:16 smithi078 ceph-mon[91136]: pgmap v5829: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:16 smithi078 ceph-mon[99959]: pgmap v5829: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:18.270 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:34:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:18 smithi078 ceph-mon[99959]: pgmap v5830: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:18 smithi078 ceph-mon[91136]: pgmap v5830: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:18 smithi146 ceph-mon[103278]: pgmap v5830: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:18.803 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 3h - - 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 3h - - 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 18s ago 3h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 13s ago 3h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 18s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 13s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 18s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 18s ago 3h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 13s ago 3h 228M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 18s ago 3h 235M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 18s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:34:18.804 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 13s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 18s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 18s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 13s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 13s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 13s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 13s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 13s ago 3h 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:34:18.805 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 13s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:34:19.110 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:34:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:34:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:34:19.112 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:34:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:34:19.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:34:19.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:19 smithi078 ceph-mon[99959]: from='client.57255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:19.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:19 smithi078 ceph-mon[99959]: from='client.77177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:19.382 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3371675775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:19.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:19 smithi078 ceph-mon[91136]: from='client.57255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:19.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:19 smithi078 ceph-mon[91136]: from='client.77177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:19.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3371675775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:19.688 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:34:19.688 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:19.688 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:19.688 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:19.689 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:19 smithi146 ceph-mon[103278]: from='client.57255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:19 smithi146 ceph-mon[103278]: from='client.77177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3371675775' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[99959]: pgmap v5831: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[99959]: from='client.77183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[99959]: from='client.57279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1099339746' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[91136]: pgmap v5831: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[91136]: from='client.77183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:20.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:20.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[91136]: from='client.57279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:20.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1099339746' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:20 smithi146 ceph-mon[103278]: pgmap v5831: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:20 smithi146 ceph-mon[103278]: from='client.77183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:20 smithi146 ceph-mon[103278]: from='client.57279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1099339746' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:20.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:34:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:34:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:20.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:20.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:20.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[99959]: pgmap v5832: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[91136]: pgmap v5832: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:21 smithi146 ceph-mon[103278]: pgmap v5832: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:23 smithi078 ceph-mon[91136]: pgmap v5833: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:23 smithi078 ceph-mon[99959]: pgmap v5833: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:23 smithi146 ceph-mon[103278]: pgmap v5833: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 23:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T23:34:25.847103215Z level=info msg="Completed cleanup jobs" duration=11.94379ms 2024-09-20T23:34:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:26 smithi146 ceph-mon[103278]: pgmap v5834: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[99959]: pgmap v5834: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[91136]: pgmap v5834: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:28 smithi078 ceph-mon[91136]: pgmap v5835: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:28 smithi078 ceph-mon[99959]: pgmap v5835: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:28 smithi146 ceph-mon[103278]: pgmap v5835: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:28.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:30 smithi146 ceph-mon[103278]: pgmap v5836: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-mon[91136]: pgmap v5836: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-mon[99959]: pgmap v5836: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:34:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:34:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:30.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:30.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:30.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:32 smithi146 ceph-mon[103278]: pgmap v5837: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:32 smithi078 ceph-mon[91136]: pgmap v5837: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:32 smithi078 ceph-mon[99959]: pgmap v5837: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:34 smithi146 ceph-mon[103278]: pgmap v5838: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:34 smithi078 ceph-mon[91136]: pgmap v5838: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:34 smithi078 ceph-mon[99959]: pgmap v5838: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:36 smithi146 ceph-mon[103278]: pgmap v5839: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:36 smithi078 ceph-mon[91136]: pgmap v5839: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:36 smithi078 ceph-mon[99959]: pgmap v5839: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:38 smithi146 ceph-mon[103278]: pgmap v5840: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:38 smithi078 ceph-mon[91136]: pgmap v5840: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:38 smithi078 ceph-mon[99959]: pgmap v5840: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:40 smithi146 ceph-mon[103278]: pgmap v5841: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:34:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:34:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-mon[91136]: pgmap v5841: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-mon[99959]: pgmap v5841: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:40.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:40.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:40.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:41 smithi146 ceph-mon[103278]: pgmap v5842: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[91136]: pgmap v5842: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[99959]: pgmap v5842: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:43 smithi146 ceph-mon[103278]: pgmap v5843: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:43 smithi078 ceph-mon[91136]: pgmap v5843: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:43 smithi078 ceph-mon[99959]: pgmap v5843: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:46 smithi146 ceph-mon[103278]: pgmap v5844: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[91136]: pgmap v5844: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[99959]: pgmap v5844: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:34:48.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:48 smithi078 ceph-mon[99959]: pgmap v5845: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:48.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:48 smithi078 ceph-mon[91136]: pgmap v5845: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:48.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:48 smithi146 ceph-mon[103278]: pgmap v5845: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:49.976 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 3h - - 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 3h - - 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 9s ago 3h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 4s ago 3h 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 9s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 4s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 9s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 9s ago 3h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:34:50.504 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 4s ago 3h 229M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 9s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 9s ago 3h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 4s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 9s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 9s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 9s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 9s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 4s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 4s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 4s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 4s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:34:50.505 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 4s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:34:50.506 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 9s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:34:50.506 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:34:50.506 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 9s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:34:50.506 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 4s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:34:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:50 smithi146 ceph-mon[103278]: pgmap v5846: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:34:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:34:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:34:50.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-mon[99959]: pgmap v5846: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:50.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:50.946 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:34:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:34:50.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-mon[91136]: pgmap v5846: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:50.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:34:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:34:51.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:34:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:50.944Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:50.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:34:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:34:50.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:34:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:34:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:34:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:34:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:51 smithi146 ceph-mon[103278]: from='client.77207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:51 smithi146 ceph-mon[103278]: from='client.57297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:51 smithi146 ceph-mon[103278]: from='client.57303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3111189341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2940890857' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[91136]: from='client.77207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[91136]: from='client.57297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[91136]: from='client.57303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3111189341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2940890857' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[99959]: from='client.77207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[99959]: from='client.57297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[99959]: from='client.57303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3111189341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:34:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2940890857' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:34:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:52 smithi078 ceph-mon[99959]: pgmap v5847: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:52 smithi078 ceph-mon[99959]: from='client.77231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:52 smithi078 ceph-mon[91136]: pgmap v5847: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:52 smithi078 ceph-mon[91136]: from='client.77231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:52 smithi146 ceph-mon[103278]: pgmap v5847: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:52 smithi146 ceph-mon[103278]: from='client.77231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:34:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:54 smithi146 ceph-mon[103278]: pgmap v5848: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:54 smithi078 ceph-mon[91136]: pgmap v5848: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:54 smithi078 ceph-mon[99959]: pgmap v5848: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:56 smithi146 ceph-mon[103278]: pgmap v5849: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:57.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:56 smithi078 ceph-mon[99959]: pgmap v5849: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:57.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:56 smithi078 ceph-mon[91136]: pgmap v5849: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:34:58.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:58 smithi078 ceph-mon[91136]: pgmap v5850: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:58.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:58.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:58 smithi078 ceph-mon[99959]: pgmap v5850: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:58.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:34:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:58 smithi146 ceph-mon[103278]: pgmap v5850: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:34:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:00.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-mon[99959]: pgmap v5851: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:00.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:00.947 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:35:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:35:00.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-mon[91136]: pgmap v5851: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:00.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:00 smithi146 ceph-mon[103278]: pgmap v5851: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:01.011 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:00.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:00.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:00.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[91136]: pgmap v5852: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[99959]: pgmap v5852: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:02 smithi146 ceph-mon[103278]: pgmap v5852: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:04 smithi146 ceph-mon[103278]: pgmap v5853: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:04 smithi078 ceph-mon[91136]: pgmap v5853: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:04 smithi078 ceph-mon[99959]: pgmap v5853: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:06 smithi146 ceph-mon[103278]: pgmap v5854: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[91136]: pgmap v5854: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[99959]: pgmap v5854: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:08 smithi078 ceph-mon[91136]: pgmap v5855: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:08 smithi078 ceph-mon[99959]: pgmap v5855: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:08.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:08 smithi146 ceph-mon[103278]: pgmap v5855: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:10.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-mon[91136]: pgmap v5856: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:10.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:10.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-mon[99959]: pgmap v5856: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:10.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:10.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:10 smithi146 ceph-mon[103278]: pgmap v5856: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:10.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:10.781 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:35:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:35:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:10.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:11.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:10.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:11.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:10.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:12 smithi078 ceph-mon[91136]: pgmap v5857: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:12 smithi078 ceph-mon[99959]: pgmap v5857: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:12 smithi146 ceph-mon[103278]: pgmap v5857: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:14 smithi146 ceph-mon[103278]: pgmap v5858: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:14 smithi078 ceph-mon[99959]: pgmap v5858: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:14 smithi078 ceph-mon[91136]: pgmap v5858: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:16 smithi146 ceph-mon[103278]: pgmap v5859: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:16 smithi078 ceph-mon[91136]: pgmap v5859: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:16 smithi078 ceph-mon[99959]: pgmap v5859: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:18 smithi146 ceph-mon[103278]: pgmap v5860: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:18 smithi078 ceph-mon[91136]: pgmap v5860: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:18 smithi078 ceph-mon[99959]: pgmap v5860: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:20 smithi146 ceph-mon[103278]: pgmap v5861: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-mon[99959]: pgmap v5861: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-mon[91136]: pgmap v5861: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:20.947 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:35:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:35:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:20.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:20.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:20.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:21.698 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:35:22.226 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:35:22.226 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 3h - - 2024-09-20T23:35:22.226 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 3h - - 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 1s ago 3h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 16s ago 3h 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 1s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 16s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 1s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 1s ago 3h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 16s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 1s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 1s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 16s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 1s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 1s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 1s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:35:22.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 1s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 16s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 16s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 16s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 16s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 16s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:35:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 16s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:35:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[99959]: pgmap v5862: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[91136]: pgmap v5862: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:22 smithi146 ceph-mon[103278]: pgmap v5862: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:22.534 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:35:22.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:22.536 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:35:22.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:35:22.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:35:22.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:35:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:35:22.804 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:35:22.804 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:35:22.805 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:35:23.113 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:35:23.113 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:23.113 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:23.113 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:23 smithi146 ceph-mon[103278]: from='client.77243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:23 smithi146 ceph-mon[103278]: from='client.77249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:23 smithi146 ceph-mon[103278]: from='client.77255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/578489605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/631747197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[91136]: from='client.77243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[91136]: from='client.77249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[91136]: from='client.77255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/578489605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/631747197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[99959]: from='client.77243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[99959]: from='client.77249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[99959]: from='client.77255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/578489605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/631747197' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:24 smithi146 ceph-mon[103278]: pgmap v5863: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:24 smithi146 ceph-mon[103278]: from='client.57351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:24 smithi078 ceph-mon[91136]: pgmap v5863: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:24 smithi078 ceph-mon[91136]: from='client.57351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:24 smithi078 ceph-mon[99959]: pgmap v5863: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:24 smithi078 ceph-mon[99959]: from='client.57351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:26.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:26 smithi146 ceph-mon[103278]: pgmap v5864: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:26.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[91136]: pgmap v5864: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[99959]: pgmap v5864: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:28 smithi146 ceph-mon[103278]: pgmap v5865: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:28 smithi078 ceph-mon[99959]: pgmap v5865: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:28 smithi078 ceph-mon[91136]: pgmap v5865: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:30.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-mon[99959]: pgmap v5866: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:30.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:30.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-mon[91136]: pgmap v5866: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:30.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:30 smithi146 ceph-mon[103278]: pgmap v5866: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:35:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:35:31.150 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:30.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:31.150 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:30.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:31.150 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:30.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:32 smithi146 ceph-mon[103278]: pgmap v5867: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:32 smithi078 ceph-mon[99959]: pgmap v5867: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:32 smithi078 ceph-mon[91136]: pgmap v5867: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:34 smithi146 ceph-mon[103278]: pgmap v5868: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:34 smithi078 ceph-mon[99959]: pgmap v5868: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:34 smithi078 ceph-mon[91136]: pgmap v5868: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:36 smithi146 ceph-mon[103278]: pgmap v5869: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:36 smithi078 ceph-mon[99959]: pgmap v5869: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:36 smithi078 ceph-mon[91136]: pgmap v5869: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:38 smithi146 ceph-mon[103278]: pgmap v5870: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:38 smithi078 ceph-mon[91136]: pgmap v5870: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:38 smithi078 ceph-mon[99959]: pgmap v5870: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:40 smithi146 ceph-mon[103278]: pgmap v5871: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-mon[91136]: pgmap v5871: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-mon[99959]: pgmap v5871: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:40.949 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:35:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:35:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:40.948Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:40.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:40.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:42 smithi146 ceph-mon[103278]: pgmap v5872: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[91136]: pgmap v5872: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[99959]: pgmap v5872: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:44 smithi146 ceph-mon[103278]: pgmap v5873: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:44 smithi078 ceph-mon[91136]: pgmap v5873: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:44 smithi078 ceph-mon[99959]: pgmap v5873: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:46 smithi146 ceph-mon[103278]: pgmap v5874: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[91136]: pgmap v5874: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[99959]: pgmap v5874: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:35:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:48 smithi146 ceph-mon[103278]: pgmap v5875: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:48 smithi078 ceph-mon[99959]: pgmap v5875: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:48 smithi078 ceph-mon[91136]: pgmap v5875: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-mon[91136]: pgmap v5876: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-mon[99959]: pgmap v5876: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:50 smithi146 ceph-mon[103278]: pgmap v5876: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:35:50.834 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:35:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:35:51.087 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:50.948Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:51.087 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:50.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:35:51.087 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:35:50.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:35:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:52 smithi078 ceph-mon[91136]: pgmap v5877: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:52 smithi078 ceph-mon[99959]: pgmap v5877: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:52.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:52 smithi146 ceph-mon[103278]: pgmap v5877: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 3h - - 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 3h - - 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 12s ago 3h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 7s ago 3h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 12s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 7s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 12s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 12s ago 3h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 7s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 12s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:35:53.943 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 12s ago 3h 14.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 7s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 12s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 12s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 12s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 12s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 7s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 7s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 7s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 7s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 7s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 12s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 12s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:35:53.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 7s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:54.254 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:35:54.255 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:35:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:35:54.527 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:35:54.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:54 smithi078 ceph-mon[99959]: pgmap v5878: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:54.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:54 smithi078 ceph-mon[99959]: from='client.57363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:54.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1393048715' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:54.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:54 smithi078 ceph-mon[91136]: pgmap v5878: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:54.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:54 smithi078 ceph-mon[91136]: from='client.57363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:54.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1393048715' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:54 smithi146 ceph-mon[103278]: pgmap v5878: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:54 smithi146 ceph-mon[103278]: from='client.57363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1393048715' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:35:54.840 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:35:54.840 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:35:54.840 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:35:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:55 smithi146 ceph-mon[103278]: from='client.77285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:55 smithi146 ceph-mon[103278]: from='client.77291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:55 smithi146 ceph-mon[103278]: from='client.77303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2522970833' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[99959]: from='client.77285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[99959]: from='client.77291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[99959]: from='client.77303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2522970833' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[91136]: from='client.77285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[91136]: from='client.77291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[91136]: from='client.77303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:35:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2522970833' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:35:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:56 smithi146 ceph-mon[103278]: pgmap v5879: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:56 smithi078 ceph-mon[99959]: pgmap v5879: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:56 smithi078 ceph-mon[91136]: pgmap v5879: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:35:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:58 smithi146 ceph-mon[103278]: pgmap v5880: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:58 smithi078 ceph-mon[91136]: pgmap v5880: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:35:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:58 smithi078 ceph-mon[99959]: pgmap v5880: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:35:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-mon[91136]: pgmap v5881: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:00.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-mon[99959]: pgmap v5881: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:00.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:00 smithi146 ceph-mon[103278]: pgmap v5881: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:00.951 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:36:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:36:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:00.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:00.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:00.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:02 smithi146 ceph-mon[103278]: pgmap v5882: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[91136]: pgmap v5882: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[99959]: pgmap v5882: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:04 smithi146 ceph-mon[103278]: pgmap v5883: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:04 smithi078 ceph-mon[91136]: pgmap v5883: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:04 smithi078 ceph-mon[99959]: pgmap v5883: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:06 smithi146 ceph-mon[103278]: pgmap v5884: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[91136]: pgmap v5884: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[99959]: pgmap v5884: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:08 smithi146 ceph-mon[103278]: pgmap v5885: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:08 smithi078 ceph-mon[91136]: pgmap v5885: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:08 smithi078 ceph-mon[99959]: pgmap v5885: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-mon[91136]: pgmap v5886: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-mon[99959]: pgmap v5886: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:10.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:10 smithi146 ceph-mon[103278]: pgmap v5886: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:10.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:10.952 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:36:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:36:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:10.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:10.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:10.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:12 smithi078 ceph-mon[91136]: pgmap v5887: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:12 smithi078 ceph-mon[99959]: pgmap v5887: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:12 smithi146 ceph-mon[103278]: pgmap v5887: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:14 smithi146 ceph-mon[103278]: pgmap v5888: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:14 smithi078 ceph-mon[91136]: pgmap v5888: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:14 smithi078 ceph-mon[99959]: pgmap v5888: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:16 smithi146 ceph-mon[103278]: pgmap v5889: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:16 smithi078 ceph-mon[91136]: pgmap v5889: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:16 smithi078 ceph-mon[99959]: pgmap v5889: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:18 smithi146 ceph-mon[103278]: pgmap v5890: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:18 smithi078 ceph-mon[91136]: pgmap v5890: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:18 smithi078 ceph-mon[99959]: pgmap v5890: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:20 smithi146 ceph-mon[103278]: pgmap v5891: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-mon[91136]: pgmap v5891: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-mon[99959]: pgmap v5891: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:36:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T23:36:21.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:20.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:21.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:20.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:21.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:20.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:22 smithi146 ceph-mon[103278]: pgmap v5892: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[91136]: pgmap v5892: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[99959]: pgmap v5892: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:24 smithi146 ceph-mon[103278]: pgmap v5893: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:24 smithi078 ceph-mon[91136]: pgmap v5893: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:24 smithi078 ceph-mon[99959]: pgmap v5893: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:25.135 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:36:25.661 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 3h - - 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 3h - - 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 4s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 19s ago 3h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 4s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 19s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 4s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 4s ago 3h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 19s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 4s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 4s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:36:25.662 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 19s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 4s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 4s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 4s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 4s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 19s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 19s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 19s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 19s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 19s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:36:25.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 19s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:36:25.970 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:25.971 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:36:25.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:36:25.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:36:25.972 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:36:26.236 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:36:26.236 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:36:26.236 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:36:26.236 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:36:26.236 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:36:26.237 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:36:26.545 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:36:26.546 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:26.546 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: pgmap v5894: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='client.77315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='client.57405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3403622802' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2596978190' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: pgmap v5894: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='client.77315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='client.57405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3403622802' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2596978190' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: pgmap v5894: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='client.77315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='client.57405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3403622802' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:27.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2596978190' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:27 smithi146 ceph-mon[103278]: from='client.77327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:27 smithi146 ceph-mon[103278]: from='client.77339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:27 smithi078 ceph-mon[99959]: from='client.77327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:27 smithi078 ceph-mon[99959]: from='client.77339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:27 smithi078 ceph-mon[91136]: from='client.77327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:27 smithi078 ceph-mon[91136]: from='client.77339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:28.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:28 smithi146 ceph-mon[103278]: pgmap v5895: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:28 smithi078 ceph-mon[99959]: pgmap v5895: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:28 smithi078 ceph-mon[91136]: pgmap v5895: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-mon[91136]: pgmap v5896: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:30.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:30.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-mon[99959]: pgmap v5896: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:30.954 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:30.954 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:36:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:36:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:30 smithi146 ceph-mon[103278]: pgmap v5896: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:31.271 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:30.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:31.271 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:30.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:31.271 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:30.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:32.658 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:32 smithi078 ceph-mon[91136]: pgmap v5897: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:32.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:32 smithi078 ceph-mon[99959]: pgmap v5897: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:32 smithi146 ceph-mon[103278]: pgmap v5897: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:35.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:34 smithi146 ceph-mon[103278]: pgmap v5898: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:34 smithi078 ceph-mon[91136]: pgmap v5898: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:34 smithi078 ceph-mon[99959]: pgmap v5898: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:36 smithi146 ceph-mon[103278]: pgmap v5899: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:36 smithi078 ceph-mon[91136]: pgmap v5899: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:36 smithi078 ceph-mon[99959]: pgmap v5899: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:38 smithi146 ceph-mon[103278]: pgmap v5900: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:38 smithi078 ceph-mon[91136]: pgmap v5900: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:38 smithi078 ceph-mon[99959]: pgmap v5900: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-mon[99959]: pgmap v5901: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:40.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-mon[91136]: pgmap v5901: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:40.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:40.954 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:36:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:36:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:40 smithi146 ceph-mon[103278]: pgmap v5901: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:40.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:40.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:40.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:42 smithi146 ceph-mon[103278]: pgmap v5902: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[91136]: pgmap v5902: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[99959]: pgmap v5902: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:44 smithi146 ceph-mon[103278]: pgmap v5903: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:44 smithi078 ceph-mon[91136]: pgmap v5903: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:44 smithi078 ceph-mon[99959]: pgmap v5903: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:46 smithi146 ceph-mon[103278]: pgmap v5904: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[91136]: pgmap v5904: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[99959]: pgmap v5904: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:36:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:48 smithi146 ceph-mon[103278]: pgmap v5905: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:48 smithi078 ceph-mon[91136]: pgmap v5905: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:48 smithi078 ceph-mon[99959]: pgmap v5905: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:50.954 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-mon[91136]: pgmap v5906: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:50.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:50.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-mon[99959]: pgmap v5906: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:50.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:50.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:36:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:50 smithi146 ceph-mon[103278]: pgmap v5906: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:36:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:50.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:50.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:36:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:36:50.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:36:51.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:51 smithi146 ceph-mon[103278]: pgmap v5907: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:51.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:51 smithi078 ceph-mon[91136]: pgmap v5907: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:51.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:51 smithi078 ceph-mon[99959]: pgmap v5907: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:54.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:53 smithi146 ceph-mon[103278]: pgmap v5908: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:54.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:53 smithi078 ceph-mon[91136]: pgmap v5908: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:54.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:53 smithi078 ceph-mon[99959]: pgmap v5908: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:55 smithi146 ceph-mon[103278]: pgmap v5909: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:55 smithi078 ceph-mon[91136]: pgmap v5909: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:55 smithi078 ceph-mon[99959]: pgmap v5909: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:36:56.838 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:36:57.362 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:36:57.362 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 3h - - 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 15s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 15s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 15s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 15s ago 3h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 15s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 15s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 15s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:36:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 15s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 15s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 15s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 10s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 15s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 15s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:36:57.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:36:57.672 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:36:57.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:36:57.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:36:57.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:36:57.942 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:36:57.942 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:36:57.942 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:36:57.942 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:36:57.943 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:36:57.943 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[99959]: from='client.57435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[99959]: pgmap v5910: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[99959]: from='client.57441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[99959]: from='client.57447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2461164161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[91136]: from='client.57435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[91136]: pgmap v5910: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[91136]: from='client.57441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[91136]: from='client.57447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2461164161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:58.252 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:36:58.252 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:36:58.252 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:36:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:57 smithi146 ceph-mon[103278]: from='client.57435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:57 smithi146 ceph-mon[103278]: pgmap v5910: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:36:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:57 smithi146 ceph-mon[103278]: from='client.57441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:57 smithi146 ceph-mon[103278]: from='client.57447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2461164161' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:36:59.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:58 smithi146 ceph-mon[103278]: from='client.57459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:59.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4098767058' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:59.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:58 smithi078 ceph-mon[99959]: from='client.57459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4098767058' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:36:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:58 smithi078 ceph-mon[91136]: from='client.57459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4098767058' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:36:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:59 smithi146 ceph-mon[103278]: pgmap v5911: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:36:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:59 smithi078 ceph-mon[99959]: pgmap v5911: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:36:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:59 smithi078 ceph-mon[91136]: pgmap v5911: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:36:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:00.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:37:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:37:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:00.954Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:00.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:00.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:01 smithi146 ceph-mon[103278]: pgmap v5912: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[91136]: pgmap v5912: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[99959]: pgmap v5912: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:03 smithi146 ceph-mon[103278]: pgmap v5913: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:03 smithi078 ceph-mon[99959]: pgmap v5913: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:03 smithi078 ceph-mon[91136]: pgmap v5913: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:05 smithi146 ceph-mon[103278]: pgmap v5914: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:05 smithi078 ceph-mon[91136]: pgmap v5914: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:05 smithi078 ceph-mon[99959]: pgmap v5914: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:08.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:08 smithi146 ceph-mon[103278]: pgmap v5915: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:08 smithi078 ceph-mon[91136]: pgmap v5915: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:08 smithi078 ceph-mon[99959]: pgmap v5915: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:10 smithi146 ceph-mon[103278]: pgmap v5916: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:37:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:37:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-mon[99959]: pgmap v5916: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-mon[91136]: pgmap v5916: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:10.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:10.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:10.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:12.641 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:12 smithi146 ceph-mon[103278]: pgmap v5917: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:12 smithi078 ceph-mon[91136]: pgmap v5917: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:12 smithi078 ceph-mon[99959]: pgmap v5917: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:13.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:14.678 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:14 smithi146 ceph-mon[103278]: pgmap v5918: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:14 smithi078 ceph-mon[99959]: pgmap v5918: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:14 smithi078 ceph-mon[91136]: pgmap v5918: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:16 smithi078 ceph-mon[91136]: pgmap v5919: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:16 smithi078 ceph-mon[99959]: pgmap v5919: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:16 smithi146 ceph-mon[103278]: pgmap v5919: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:18 smithi078 ceph-mon[91136]: pgmap v5920: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:18 smithi078 ceph-mon[99959]: pgmap v5920: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:18 smithi146 ceph-mon[103278]: pgmap v5920: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-mon[91136]: pgmap v5921: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:37:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:37:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-mon[99959]: pgmap v5921: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:20 smithi146 ceph-mon[103278]: pgmap v5921: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:20.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:20.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:20.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[91136]: pgmap v5922: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[99959]: pgmap v5922: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:22 smithi146 ceph-mon[103278]: pgmap v5922: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:24 smithi078 ceph-mon[91136]: pgmap v5923: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:24 smithi078 ceph-mon[99959]: pgmap v5923: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:24 smithi146 ceph-mon[103278]: pgmap v5923: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:26 smithi146 ceph-mon[103278]: pgmap v5924: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:26 smithi078 ceph-mon[91136]: pgmap v5924: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:26 smithi078 ceph-mon[99959]: pgmap v5924: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:28.556 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:37:28.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:28 smithi146 ceph-mon[103278]: pgmap v5925: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:28 smithi078 ceph-mon[99959]: pgmap v5925: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:28 smithi078 ceph-mon[91136]: pgmap v5925: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:29.087 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:37:29.087 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 3h - - 2024-09-20T23:37:29.087 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 3h - - 2024-09-20T23:37:29.087 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 7s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:37:29.087 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 2s ago 3h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:37:29.087 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 7s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 2s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 7s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 7s ago 3h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 2s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 7s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 7s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 2s ago 3h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 7s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 7s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 7s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 7s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 2s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 2s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:37:29.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 2s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:37:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 2s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:37:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 2s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:37:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:37:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:37:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:37:29.089 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 2s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:37:29.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:37:29.398 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:37:29.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:37:29.669 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:37:29.669 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:37:29.670 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:37:29.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:29 smithi146 ceph-mon[103278]: from='client.57471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:29.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3963303581' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:29.988 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:37:29.988 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:37:29.988 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:37:29.988 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:37:29.989 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:37:29.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:29 smithi078 ceph-mon[99959]: from='client.57471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:29.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3963303581' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:29.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:29 smithi078 ceph-mon[91136]: from='client.57471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:29.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3963303581' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:37:30.871 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:37:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:37:30.871 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[99959]: from='client.77393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:30.871 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[99959]: pgmap v5926: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:30.871 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[99959]: from='client.57483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:30.871 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[99959]: from='client.77411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/376423522' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[91136]: from='client.77393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[91136]: pgmap v5926: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[91136]: from='client.57483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[91136]: from='client.77411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:30.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/376423522' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:30 smithi146 ceph-mon[103278]: from='client.77393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:30 smithi146 ceph-mon[103278]: pgmap v5926: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:30 smithi146 ceph-mon[103278]: from='client.57483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:30 smithi146 ceph-mon[103278]: from='client.77411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/376423522' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:37:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:30.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:30.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:30.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:31 smithi146 ceph-mon[103278]: pgmap v5927: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:31 smithi078 ceph-mon[99959]: pgmap v5927: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:31 smithi078 ceph-mon[91136]: pgmap v5927: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:33 smithi146 ceph-mon[103278]: pgmap v5928: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:34.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:33 smithi078 ceph-mon[99959]: pgmap v5928: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:34.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:33 smithi078 ceph-mon[91136]: pgmap v5928: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:35 smithi078 ceph-mon[91136]: pgmap v5929: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:35 smithi078 ceph-mon[99959]: pgmap v5929: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:35 smithi146 ceph-mon[103278]: pgmap v5929: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:37 smithi146 ceph-mon[103278]: pgmap v5930: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:37 smithi078 ceph-mon[91136]: pgmap v5930: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:37 smithi078 ceph-mon[99959]: pgmap v5930: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:39 smithi146 ceph-mon[103278]: pgmap v5931: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:39 smithi078 ceph-mon[91136]: pgmap v5931: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:39 smithi078 ceph-mon[99959]: pgmap v5931: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:40.958 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:37:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:37:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:40.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:40.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:40.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:41 smithi146 ceph-mon[103278]: pgmap v5932: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:41 smithi078 ceph-mon[91136]: pgmap v5932: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:41 smithi078 ceph-mon[99959]: pgmap v5932: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:44 smithi146 ceph-mon[103278]: pgmap v5933: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:44 smithi078 ceph-mon[91136]: pgmap v5933: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:44 smithi078 ceph-mon[99959]: pgmap v5933: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:46 smithi146 ceph-mon[103278]: pgmap v5934: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:46 smithi078 ceph-mon[91136]: pgmap v5934: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:46 smithi078 ceph-mon[99959]: pgmap v5934: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:47 smithi146 ceph-mon[103278]: pgmap v5935: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[91136]: pgmap v5935: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:37:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:47 smithi078 ceph-mon[99959]: pgmap v5935: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:49 smithi146 ceph-mon[103278]: pgmap v5936: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:49 smithi078 ceph-mon[91136]: pgmap v5936: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:49 smithi078 ceph-mon[99959]: pgmap v5936: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:37:50.959 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:37:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:37:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:50.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:50.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:37:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:37:50.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:37:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:51 smithi146 ceph-mon[103278]: pgmap v5937: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:51 smithi078 ceph-mon[91136]: pgmap v5937: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:51 smithi078 ceph-mon[99959]: pgmap v5937: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:53 smithi146 ceph-mon[103278]: pgmap v5938: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:54.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:53 smithi078 ceph-mon[99959]: pgmap v5938: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:54.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:53 smithi078 ceph-mon[91136]: pgmap v5938: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:55 smithi146 ceph-mon[103278]: pgmap v5939: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:55 smithi078 ceph-mon[91136]: pgmap v5939: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:55 smithi078 ceph-mon[99959]: pgmap v5939: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:37:58.276 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:57 smithi146 ceph-mon[103278]: pgmap v5940: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:58.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:57 smithi078 ceph-mon[91136]: pgmap v5940: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:58.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:57 smithi078 ceph-mon[99959]: pgmap v5940: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:37:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:37:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:59 smithi146 ceph-mon[103278]: pgmap v5941: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:37:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:00.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:59 smithi078 ceph-mon[99959]: pgmap v5941: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:00.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:37:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:00.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:59 smithi078 ceph-mon[91136]: pgmap v5941: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:00.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:37:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:00.281 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:38:00.809 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:38:00.809 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 3h - - 2024-09-20T23:38:00.809 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 3h - - 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 18s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 14s ago 3h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 18s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 14s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 18s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 18s ago 3h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 14s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 18s ago 3h 237M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 18s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 14s ago 3h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 18s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 18s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:38:00.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 14s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 14s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 14s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 14s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:38:00.811 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:38:00.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:38:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:38:01.120 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:38:01.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:01.121 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:38:01.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:38:01.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:38:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:38:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:00 smithi146 ceph-mon[103278]: from='client.77423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:00 smithi146 ceph-mon[103278]: from='client.77429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-mon[99959]: from='client.77423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-mon[99959]: from='client.77429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-mon[91136]: from='client.77423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-mon[91136]: from='client.77429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:00.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:00.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:00.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:38:01.389 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:38:01.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:38:01.700 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:38:01.700 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:01.700 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:01.700 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:01.700 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[91136]: from='client.77435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[91136]: pgmap v5942: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1466587570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[91136]: from='client.57531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/253093957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[99959]: from='client.77435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[99959]: pgmap v5942: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1466587570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[99959]: from='client.57531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/253093957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:01 smithi146 ceph-mon[103278]: from='client.77435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:01 smithi146 ceph-mon[103278]: pgmap v5942: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1466587570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:01 smithi146 ceph-mon[103278]: from='client.57531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/253093957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:04 smithi146 ceph-mon[103278]: pgmap v5943: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:04 smithi078 ceph-mon[91136]: pgmap v5943: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:04 smithi078 ceph-mon[99959]: pgmap v5943: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:06 smithi146 ceph-mon[103278]: pgmap v5944: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:06 smithi078 ceph-mon[91136]: pgmap v5944: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:06 smithi078 ceph-mon[99959]: pgmap v5944: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:07 smithi146 ceph-mon[103278]: pgmap v5945: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[91136]: pgmap v5945: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[99959]: pgmap v5945: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:09 smithi146 ceph-mon[103278]: pgmap v5946: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:09 smithi078 ceph-mon[91136]: pgmap v5946: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:09 smithi078 ceph-mon[99959]: pgmap v5946: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:10.959 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:38:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:38:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:10.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:10.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:10.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:11 smithi146 ceph-mon[103278]: pgmap v5947: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:11 smithi078 ceph-mon[99959]: pgmap v5947: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:11 smithi078 ceph-mon[91136]: pgmap v5947: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:14.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:13 smithi146 ceph-mon[103278]: pgmap v5948: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:14.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:13 smithi078 ceph-mon[99959]: pgmap v5948: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:13 smithi078 ceph-mon[91136]: pgmap v5948: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:15.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:15 smithi078 ceph-mon[99959]: pgmap v5949: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:15.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:15 smithi078 ceph-mon[91136]: pgmap v5949: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:15 smithi146 ceph-mon[103278]: pgmap v5949: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:17 smithi146 ceph-mon[103278]: pgmap v5950: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:17 smithi078 ceph-mon[91136]: pgmap v5950: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:17 smithi078 ceph-mon[99959]: pgmap v5950: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:19 smithi146 ceph-mon[103278]: pgmap v5951: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:19 smithi078 ceph-mon[91136]: pgmap v5951: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:19 smithi078 ceph-mon[99959]: pgmap v5951: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:20.960 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:38:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:38:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:20.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:20.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:20.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:21 smithi078 ceph-mon[91136]: pgmap v5952: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:21 smithi078 ceph-mon[99959]: pgmap v5952: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:21 smithi146 ceph-mon[103278]: pgmap v5952: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:24 smithi146 ceph-mon[103278]: pgmap v5953: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:24 smithi078 ceph-mon[91136]: pgmap v5953: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:24 smithi078 ceph-mon[99959]: pgmap v5953: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:26 smithi146 ceph-mon[103278]: pgmap v5954: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:26 smithi078 ceph-mon[91136]: pgmap v5954: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:26 smithi078 ceph-mon[99959]: pgmap v5954: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[91136]: pgmap v5955: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[99959]: pgmap v5955: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:28 smithi146 ceph-mon[103278]: pgmap v5955: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:28.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:28.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:29.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:30 smithi146 ceph-mon[103278]: pgmap v5956: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-mon[99959]: pgmap v5956: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-mon[91136]: pgmap v5956: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:30.961 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:38:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:38:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:30.960Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:30.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:30.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:31.994 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:38:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:32 smithi078 ceph-mon[99959]: pgmap v5957: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:32 smithi078 ceph-mon[91136]: pgmap v5957: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:32 smithi146 ceph-mon[103278]: pgmap v5957: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 3h - - 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 3h - - 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 10s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 5s ago 3h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 10s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:38:32.521 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 5s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 10s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 10s ago 3h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 5s ago 3h 231M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 10s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 10s ago 3h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 5s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 10s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 10s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 10s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 10s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 5s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 5s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 5s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 5s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:38:32.522 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 5s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:38:32.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 10s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:38:32.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:38:32.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 10s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:38:32.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 5s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:38:32.835 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:38:32.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:38:32.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:38:32.837 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:38:33.106 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:38:33.106 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:38:33.106 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:38:33.106 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:38:33.107 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:38:33.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[91136]: from='client.57543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[91136]: from='client.57549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.108 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[91136]: from='client.77471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.108 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/310611081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:33.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[99959]: from='client.57543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[99959]: from='client.57549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[99959]: from='client.77471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/310611081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:33.416 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:38:33.416 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:33.416 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:38:33.416 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:33.416 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:38:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:33 smithi146 ceph-mon[103278]: from='client.57543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:33 smithi146 ceph-mon[103278]: from='client.57549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:33 smithi146 ceph-mon[103278]: from='client.77471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/310611081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:38:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:34 smithi146 ceph-mon[103278]: pgmap v5958: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:34 smithi146 ceph-mon[103278]: from='client.77483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2046015777' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:34 smithi078 ceph-mon[91136]: pgmap v5958: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:34 smithi078 ceph-mon[91136]: from='client.77483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2046015777' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:34 smithi078 ceph-mon[99959]: pgmap v5958: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:34 smithi078 ceph-mon[99959]: from='client.77483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2046015777' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:38:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:36 smithi146 ceph-mon[103278]: pgmap v5959: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:36.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:36 smithi078 ceph-mon[91136]: pgmap v5959: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:36.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:36 smithi078 ceph-mon[99959]: pgmap v5959: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:38 smithi146 ceph-mon[103278]: pgmap v5960: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:38 smithi078 ceph-mon[91136]: pgmap v5960: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:38 smithi078 ceph-mon[99959]: pgmap v5960: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:40 smithi146 ceph-mon[103278]: pgmap v5961: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-mon[91136]: pgmap v5961: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-mon[99959]: pgmap v5961: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:40.961 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:38:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:38:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:40.960Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:40.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:40.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:42.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:42 smithi078 ceph-mon[91136]: pgmap v5962: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:42.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:42 smithi078 ceph-mon[99959]: pgmap v5962: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:42 smithi146 ceph-mon[103278]: pgmap v5962: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:44 smithi146 ceph-mon[103278]: pgmap v5963: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:44 smithi078 ceph-mon[91136]: pgmap v5963: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:44 smithi078 ceph-mon[99959]: pgmap v5963: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:46 smithi146 ceph-mon[103278]: pgmap v5964: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:46 smithi078 ceph-mon[91136]: pgmap v5964: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:46 smithi078 ceph-mon[99959]: pgmap v5964: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:48 smithi146 ceph-mon[103278]: pgmap v5965: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[99959]: pgmap v5965: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[91136]: pgmap v5965: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:38:50.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-mon[99959]: pgmap v5966: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:50.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:50.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-mon[91136]: pgmap v5966: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:50.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:50 smithi146 ceph-mon[103278]: pgmap v5966: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:38:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:38:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:38:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:50.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:50.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:38:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:38:50.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:38:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:52 smithi146 ceph-mon[103278]: pgmap v5967: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:52 smithi078 ceph-mon[91136]: pgmap v5967: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:52 smithi078 ceph-mon[99959]: pgmap v5967: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:54 smithi146 ceph-mon[103278]: pgmap v5968: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:54 smithi078 ceph-mon[91136]: pgmap v5968: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:54 smithi078 ceph-mon[99959]: pgmap v5968: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:56.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:56 smithi146 ceph-mon[103278]: pgmap v5969: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:56 smithi078 ceph-mon[91136]: pgmap v5969: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:56 smithi078 ceph-mon[99959]: pgmap v5969: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:38:58.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:58 smithi078 ceph-mon[91136]: pgmap v5970: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:58.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:58 smithi078 ceph-mon[99959]: pgmap v5970: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:58 smithi146 ceph-mon[103278]: pgmap v5970: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:38:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:38:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:38:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:38:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:38:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:00 smithi146 ceph-mon[103278]: pgmap v5971: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-mon[91136]: pgmap v5971: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-mon[99959]: pgmap v5971: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:00.963 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:39:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:39:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:00.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:00.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:00.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:02 smithi146 ceph-mon[103278]: pgmap v5972: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:02 smithi078 ceph-mon[91136]: pgmap v5972: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:02 smithi078 ceph-mon[99959]: pgmap v5972: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:39:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:04.243 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:39:04.243 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 3h - - 2024-09-20T23:39:04.243 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 3h - - 2024-09-20T23:39:04.243 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 1s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:39:04.243 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 17s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:39:04.243 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 1s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:39:04.244 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 17s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:39:04.244 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 1s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:39:04.244 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 1s ago 3h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:39:04.245 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 17s ago 3h 230M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:39:04.245 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 1s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:39:04.245 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 1s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:39:04.245 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 17s ago 3h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:39:04.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 1s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 1s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 1s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 1s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 17s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 17s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 17s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 17s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 17s ago 3h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 17s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:39:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 17s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:39:04.553 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:39:04.554 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:39:04.819 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:39:04.819 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:39:04.819 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:39:04.819 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:39:04.819 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:39:04.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:39:04.820 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:04 smithi078 ceph-mon[99959]: pgmap v5973: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:04.820 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/78212728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:04.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:04 smithi078 ceph-mon[91136]: pgmap v5973: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:04.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/78212728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:04 smithi146 ceph-mon[103278]: pgmap v5973: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/78212728' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:05.128 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:39:05.128 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:39:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:39:05.128 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:39:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:39:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:05 smithi146 ceph-mon[103278]: from='client.77495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:05 smithi146 ceph-mon[103278]: from='client.57585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:05 smithi146 ceph-mon[103278]: from='client.57591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/527388535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[91136]: from='client.77495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[91136]: from='client.57585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[91136]: from='client.57591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/527388535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[99959]: from='client.77495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[99959]: from='client.57585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[99959]: from='client.57591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/527388535' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:06 smithi146 ceph-mon[103278]: from='client.57603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:06 smithi146 ceph-mon[103278]: pgmap v5974: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:06 smithi078 ceph-mon[99959]: from='client.57603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:06 smithi078 ceph-mon[99959]: pgmap v5974: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:06 smithi078 ceph-mon[91136]: from='client.57603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:06 smithi078 ceph-mon[91136]: pgmap v5974: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[91136]: pgmap v5975: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[99959]: pgmap v5975: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:08 smithi146 ceph-mon[103278]: pgmap v5975: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-mon[99959]: pgmap v5976: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-mon[91136]: pgmap v5976: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:10 smithi146 ceph-mon[103278]: pgmap v5976: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:10.963 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:39:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:39:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:10.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:10.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:10.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:12 smithi146 ceph-mon[103278]: pgmap v5977: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:12 smithi078 ceph-mon[91136]: pgmap v5977: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:12 smithi078 ceph-mon[99959]: pgmap v5977: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:14 smithi146 ceph-mon[103278]: pgmap v5978: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:14 smithi078 ceph-mon[91136]: pgmap v5978: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:14 smithi078 ceph-mon[99959]: pgmap v5978: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:16 smithi146 ceph-mon[103278]: pgmap v5979: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:16.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:16 smithi078 ceph-mon[91136]: pgmap v5979: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:16.641 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:16 smithi078 ceph-mon[99959]: pgmap v5979: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:18.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:18 smithi078 ceph-mon[91136]: pgmap v5980: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:18.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:18 smithi078 ceph-mon[99959]: pgmap v5980: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:18.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:18 smithi146 ceph-mon[103278]: pgmap v5980: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-mon[91136]: pgmap v5981: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-mon[99959]: pgmap v5981: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:20 smithi146 ceph-mon[103278]: pgmap v5981: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:20.964 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:39:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:39:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:20.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:20.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:20.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:22.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:22 smithi078 ceph-mon[99959]: pgmap v5982: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:22.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:22 smithi078 ceph-mon[91136]: pgmap v5982: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:22 smithi146 ceph-mon[103278]: pgmap v5982: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:23 smithi146 ceph-mon[103278]: pgmap v5983: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[91136]: pgmap v5983: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:23 smithi078 ceph-mon[99959]: pgmap v5983: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:25 smithi146 ceph-mon[103278]: pgmap v5984: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:25 smithi078 ceph-mon[99959]: pgmap v5984: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:25 smithi078 ceph-mon[91136]: pgmap v5984: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:28 smithi146 ceph-mon[103278]: pgmap v5985: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[99959]: pgmap v5985: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[91136]: pgmap v5985: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:30 smithi146 ceph-mon[103278]: pgmap v5986: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-mon[91136]: pgmap v5986: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:39:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:39:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-mon[99959]: pgmap v5986: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:30.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:30.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:30.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:32 smithi146 ceph-mon[103278]: pgmap v5987: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:32 smithi078 ceph-mon[91136]: pgmap v5987: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:32 smithi078 ceph-mon[99959]: pgmap v5987: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:34 smithi146 ceph-mon[103278]: pgmap v5988: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:34 smithi078 ceph-mon[91136]: pgmap v5988: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:34 smithi078 ceph-mon[99959]: pgmap v5988: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:35.419 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 3h - - 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 3h - - 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 8s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 8s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 3h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:39:35.949 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 8s ago 3h 232M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 8s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 13s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 13s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 13s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 8s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 8s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 8s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 8s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 8s ago 3h 66.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 8s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:39:35.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:39:35.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 8s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:36.258 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:39:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:39:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:39:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:39:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:39:36.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:39:36.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:39:36.526 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:39:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:39:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:39:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:39:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:39:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:39:36.527 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:39:36.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:36 smithi078 ceph-mon[91136]: pgmap v5989: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:36.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:36 smithi078 ceph-mon[91136]: from='client.77531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:36.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/798901122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:36.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:36 smithi078 ceph-mon[99959]: pgmap v5989: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:36.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:36 smithi078 ceph-mon[99959]: from='client.77531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:36.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/798901122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:36 smithi146 ceph-mon[103278]: pgmap v5989: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:36 smithi146 ceph-mon[103278]: from='client.77531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/798901122' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:39:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:39:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:39:36.835 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:39:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:39:36.835 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:39:37.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:37 smithi146 ceph-mon[103278]: from='client.77537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:37 smithi146 ceph-mon[103278]: from='client.77543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:37 smithi146 ceph-mon[103278]: from='client.57639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3954652848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[99959]: from='client.77537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[99959]: from='client.77543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[99959]: from='client.57639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3954652848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[91136]: from='client.77537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[91136]: from='client.77543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[91136]: from='client.57639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3954652848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:39:38.699 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:38 smithi146 ceph-mon[103278]: pgmap v5990: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:38.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:38 smithi078 ceph-mon[91136]: pgmap v5990: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:38.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:38 smithi078 ceph-mon[99959]: pgmap v5990: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:40 smithi146 ceph-mon[103278]: pgmap v5991: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-mon[99959]: pgmap v5991: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-mon[91136]: pgmap v5991: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:39:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:39:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:40.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:40.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:40.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:42 smithi146 ceph-mon[103278]: pgmap v5992: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:42 smithi078 ceph-mon[91136]: pgmap v5992: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:42 smithi078 ceph-mon[99959]: pgmap v5992: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:43 smithi146 ceph-mon[103278]: pgmap v5993: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[91136]: pgmap v5993: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[99959]: pgmap v5993: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:45 smithi146 ceph-mon[103278]: pgmap v5994: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:45 smithi078 ceph-mon[91136]: pgmap v5994: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:45 smithi078 ceph-mon[99959]: pgmap v5994: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:48 smithi146 ceph-mon[103278]: pgmap v5995: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[99959]: pgmap v5995: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[91136]: pgmap v5995: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:39:50.966 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:39:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:39:50.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-mon[99959]: pgmap v5996: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:50.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:50.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-mon[91136]: pgmap v5996: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:50.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:50 smithi146 ceph-mon[103278]: pgmap v5996: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:39:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:50.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:50.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:39:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:39:50.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:39:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:52 smithi146 ceph-mon[103278]: pgmap v5997: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:52 smithi078 ceph-mon[91136]: pgmap v5997: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:52 smithi078 ceph-mon[99959]: pgmap v5997: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:54 smithi146 ceph-mon[103278]: pgmap v5998: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:54 smithi078 ceph-mon[91136]: pgmap v5998: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:54 smithi078 ceph-mon[99959]: pgmap v5998: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:56.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:56 smithi146 ceph-mon[103278]: pgmap v5999: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:56 smithi078 ceph-mon[91136]: pgmap v5999: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:56 smithi078 ceph-mon[99959]: pgmap v5999: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:39:58.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:58 smithi146 ceph-mon[103278]: pgmap v6000: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:58.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:58.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:58 smithi078 ceph-mon[99959]: pgmap v6000: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:58.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:39:58.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:58 smithi078 ceph-mon[91136]: pgmap v6000: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:39:58.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:00.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T23:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:00 smithi146 ceph-mon[103278]: pgmap v6001: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:00.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:00.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-mon[99959]: pgmap v6001: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-mon[91136]: pgmap v6001: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:00.967 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:40:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:40:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:00.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:00.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:00.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:03.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:02 smithi078 ceph-mon[91136]: pgmap v6002: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:03.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:02 smithi078 ceph-mon[99959]: pgmap v6002: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:02 smithi146 ceph-mon[103278]: pgmap v6002: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:04 smithi146 ceph-mon[103278]: pgmap v6003: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[91136]: pgmap v6003: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[99959]: pgmap v6003: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:06 smithi146 ceph-mon[103278]: pgmap v6004: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:06 smithi078 ceph-mon[91136]: pgmap v6004: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:06 smithi078 ceph-mon[99959]: pgmap v6004: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:40:07.654 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 3h - - 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 3h - - 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 4s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 20s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 4s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 20s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 4s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 4s ago 3h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 20s ago 3h 233M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 4s ago 3h 241M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 4s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 20s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:40:07.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 4s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 4s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 4s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 4s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 20s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 20s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 20s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 20s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 20s ago 3h 66.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 20s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:40:07.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 20s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:40:07.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:40:07.968 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:40:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:40:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:40:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:40:08.240 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: pgmap v6005: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='client.77567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='client.77573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='client.77579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1291872732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: pgmap v6005: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='client.77567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='client.77573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='client.77579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1291872732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: pgmap v6005: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='client.77567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='client.77573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='client.77579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1291872732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:08.552 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:08.552 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:08.552 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:09 smithi078 ceph-mon[91136]: from='client.57675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2337214585' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:09 smithi078 ceph-mon[99959]: from='client.57675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2337214585' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:09 smithi146 ceph-mon[103278]: from='client.57675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2337214585' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-mon[91136]: pgmap v6006: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-mon[99959]: pgmap v6006: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:10.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:10 smithi146 ceph-mon[103278]: pgmap v6006: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:10.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:10.967 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:40:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:40:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:10.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:10.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:10.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:12 smithi078 ceph-mon[91136]: pgmap v6007: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:12 smithi078 ceph-mon[99959]: pgmap v6007: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:12 smithi146 ceph-mon[103278]: pgmap v6007: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:14 smithi078 ceph-mon[91136]: pgmap v6008: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:14 smithi078 ceph-mon[99959]: pgmap v6008: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:14 smithi146 ceph-mon[103278]: pgmap v6008: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:16 smithi078 ceph-mon[91136]: pgmap v6009: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:16 smithi078 ceph-mon[99959]: pgmap v6009: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:16.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:16 smithi146 ceph-mon[103278]: pgmap v6009: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:18 smithi078 ceph-mon[91136]: pgmap v6010: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:18 smithi078 ceph-mon[99959]: pgmap v6010: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:18 smithi146 ceph-mon[103278]: pgmap v6010: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:20.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-mon[91136]: pgmap v6011: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:20.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:20.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-mon[99959]: pgmap v6011: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:20.204 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:20 smithi146 ceph-mon[103278]: pgmap v6011: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:20.968 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:40:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:40:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:20.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:20.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:20.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:22 smithi146 ceph-mon[103278]: pgmap v6012: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:22 smithi078 ceph-mon[91136]: pgmap v6012: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:22 smithi078 ceph-mon[99959]: pgmap v6012: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:24 smithi146 ceph-mon[103278]: pgmap v6013: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[91136]: pgmap v6013: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[99959]: pgmap v6013: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:26 smithi146 ceph-mon[103278]: pgmap v6014: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:26.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:26 smithi078 ceph-mon[91136]: pgmap v6014: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:26.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:26 smithi078 ceph-mon[99959]: pgmap v6014: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:28 smithi146 ceph-mon[103278]: pgmap v6015: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[91136]: pgmap v6015: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[99959]: pgmap v6015: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:30.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-mon[91136]: pgmap v6016: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:30.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:30.901 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:40:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:40:30.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-mon[99959]: pgmap v6016: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:30.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:30 smithi146 ceph-mon[103278]: pgmap v6016: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:30.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:30.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:30.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:32 smithi146 ceph-mon[103278]: pgmap v6017: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:32 smithi078 ceph-mon[91136]: pgmap v6017: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:32 smithi078 ceph-mon[99959]: pgmap v6017: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:34 smithi146 ceph-mon[103278]: pgmap v6018: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:34 smithi078 ceph-mon[91136]: pgmap v6018: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:34 smithi078 ceph-mon[99959]: pgmap v6018: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:36 smithi146 ceph-mon[103278]: pgmap v6019: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:36 smithi078 ceph-mon[91136]: pgmap v6019: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:36 smithi078 ceph-mon[99959]: pgmap v6019: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:38.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:38 smithi146 ceph-mon[103278]: pgmap v6020: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:38.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:38 smithi078 ceph-mon[99959]: pgmap v6020: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:38.849 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:38 smithi078 ceph-mon[91136]: pgmap v6020: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:38.850 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 3h - - 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 3h - - 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 16s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 11s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 11s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 3h 575M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 3h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 11s ago 3h 234M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 16s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 16s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:40:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 11s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 16s ago 3h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 16s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 16s ago 3h 73.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 16s ago 3h 82.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 11s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 11s ago 3h 73.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 11s ago 3h 78.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 11s ago 3h 74.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 11s ago 3h 69.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 16s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 11s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 16s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:40:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 11s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:40:39.705 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:40:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:40:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:40:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:40:39.706 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:40:39.980 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:40:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:40:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:40:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:40:39.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:40:40.305 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:40:40.305 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:40.305 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:40:40.305 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:40.305 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:40:40.970 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:40:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:40:40.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: from='client.57687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:40.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: pgmap v6021: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:40.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: from='client.77609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:40.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: from='client.57699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/919921612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/606023068' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: from='client.57687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: pgmap v6021: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: from='client.77609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: from='client.57699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/919921612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:40.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/606023068' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: from='client.57687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: pgmap v6021: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: from='client.77609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: from='client.57699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/919921612' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:40:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/606023068' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:40:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:40.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:40.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:40.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:41 smithi146 ceph-mon[103278]: from='client.57711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:41 smithi078 ceph-mon[91136]: from='client.57711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:41 smithi078 ceph-mon[99959]: from='client.57711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:42 smithi146 ceph-mon[103278]: pgmap v6022: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:42 smithi078 ceph-mon[91136]: pgmap v6022: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:42 smithi078 ceph-mon[99959]: pgmap v6022: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[91136]: pgmap v6023: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[99959]: pgmap v6023: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:44 smithi146 ceph-mon[103278]: pgmap v6023: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:46 smithi078 ceph-mon[91136]: pgmap v6024: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:46 smithi078 ceph-mon[99959]: pgmap v6024: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:46 smithi146 ceph-mon[103278]: pgmap v6024: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[99959]: pgmap v6025: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[91136]: pgmap v6025: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:48 smithi146 ceph-mon[103278]: pgmap v6025: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:40:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-mon[91136]: pgmap v6026: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-mon[99959]: pgmap v6026: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:50 smithi146 ceph-mon[103278]: pgmap v6026: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:40:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:40:50.970 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:40:50] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-20T23:40:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:50.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:50.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:40:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:40:50.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:40:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:52 smithi078 ceph-mon[91136]: pgmap v6027: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:52 smithi078 ceph-mon[99959]: pgmap v6027: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:52 smithi146 ceph-mon[103278]: pgmap v6027: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:54 smithi078 ceph-mon[91136]: pgmap v6028: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:54 smithi078 ceph-mon[99959]: pgmap v6028: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:54 smithi146 ceph-mon[103278]: pgmap v6028: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:40:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:56 smithi146 ceph-mon[103278]: pgmap v6029: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:40:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:56 smithi078 ceph-mon[91136]: pgmap v6029: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:40:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:56 smithi078 ceph-mon[99959]: pgmap v6029: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:40:58.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:58 smithi146 ceph-mon[103278]: pgmap v6030: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:40:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:58 smithi078 ceph-mon[91136]: pgmap v6030: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:40:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:58 smithi078 ceph-mon[99959]: pgmap v6030: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:40:59.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:40:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:59.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:40:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:40:59.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:40:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:00.374 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:00 smithi146 ceph-mon[103278]: pgmap v6031: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:41:00.374 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:00.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-mon[91136]: pgmap v6031: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:41:00.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:00.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-mon[99959]: pgmap v6031: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-09-20T23:41:00.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:41:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:41:01.186 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:00.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:01.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:00.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:01.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:00.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:02.413 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:02 smithi078 ceph-mon[99959]: pgmap v6032: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:02.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:02 smithi078 ceph-mon[91136]: pgmap v6032: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:02 smithi146 ceph-mon[103278]: pgmap v6032: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:04 smithi146 ceph-mon[103278]: pgmap v6033: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[91136]: pgmap v6033: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[99959]: pgmap v6033: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:06 smithi146 ceph-mon[103278]: pgmap v6034: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T23:41:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:06 smithi078 ceph-mon[91136]: pgmap v6034: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T23:41:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:06 smithi078 ceph-mon[99959]: pgmap v6034: 161 pgs: 161 active+clean; 585 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-20T23:41:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:08 smithi146 ceph-mon[103278]: pgmap v6035: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T23:41:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[91136]: pgmap v6035: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T23:41:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[99959]: pgmap v6035: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-20T23:41:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:10.602 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:41:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:10 smithi146 ceph-mon[103278]: pgmap v6036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T23:41:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-mon[91136]: pgmap v6036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T23:41:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:41:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:41:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-mon[99959]: pgmap v6036: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-20T23:41:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 3h - - 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 3h - - 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 7s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 3s ago 3h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 7s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 3s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 7s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 7s ago 3h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 3s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 7s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:41:11.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 7s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 3s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 7s ago 3h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 7s ago 3h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 7s ago 3h 72.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 7s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 3s ago 3h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 3s ago 3h 74.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 3s ago 3h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 3s ago 3h 74.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 3s ago 3h 69.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 3s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:41:11.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 3s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:41:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:10.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:10.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:10.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:41:11.446 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:41:11.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:41:11.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:41:11.719 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:41:11.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:11 smithi078 ceph-mon[99959]: from='client.57723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:11.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:11 smithi078 ceph-mon[91136]: from='client.57723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:11 smithi146 ceph-mon[103278]: from='client.57723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.028 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:41:12.028 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:12.028 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:12.028 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:12.028 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:12 smithi146 ceph-mon[103278]: from='client.77645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:12 smithi146 ceph-mon[103278]: pgmap v6037: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-20T23:41:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:12 smithi146 ceph-mon[103278]: from='client.57735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3636521677' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3747128793' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[91136]: from='client.77645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[91136]: pgmap v6037: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-20T23:41:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[91136]: from='client.57735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3636521677' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3747128793' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[99959]: from='client.77645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[99959]: pgmap v6037: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-20T23:41:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[99959]: from='client.57735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3636521677' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3747128793' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:13 smithi146 ceph-mon[103278]: from='client.57747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:13 smithi078 ceph-mon[91136]: from='client.57747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:13 smithi078 ceph-mon[99959]: from='client.57747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:14 smithi146 ceph-mon[103278]: pgmap v6038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T23:41:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:14 smithi078 ceph-mon[99959]: pgmap v6038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T23:41:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:14 smithi078 ceph-mon[91136]: pgmap v6038: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T23:41:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:16 smithi146 ceph-mon[103278]: pgmap v6039: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:41:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:16 smithi078 ceph-mon[91136]: pgmap v6039: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:41:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:16 smithi078 ceph-mon[99959]: pgmap v6039: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:41:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:18 smithi146 ceph-mon[103278]: pgmap v6040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:41:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:18 smithi078 ceph-mon[99959]: pgmap v6040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:41:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:18 smithi078 ceph-mon[91136]: pgmap v6040: 161 pgs: 161 active+clean; 585 KiB data, 283 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-20T23:41:20.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:20 smithi146 ceph-mon[103278]: pgmap v6041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T23:41:20.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:20.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-mon[99959]: pgmap v6041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T23:41:20.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:20.687 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:41:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-20T23:41:20.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-mon[91136]: pgmap v6041: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-20T23:41:20.688 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:20.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:20.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:20.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:22.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:22 smithi078 ceph-mon[91136]: pgmap v6042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:41:22.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:22 smithi078 ceph-mon[99959]: pgmap v6042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:41:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:22 smithi146 ceph-mon[103278]: pgmap v6042: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-20T23:41:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[91136]: pgmap v6043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[99959]: pgmap v6043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:24 smithi146 ceph-mon[103278]: pgmap v6043: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-20T23:41:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:26 smithi078 ceph-mon[91136]: pgmap v6044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T23:41:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:26 smithi078 ceph-mon[99959]: pgmap v6044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T23:41:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:26 smithi146 ceph-mon[103278]: pgmap v6044: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T23:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[91136]: pgmap v6045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[99959]: pgmap v6045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:41:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:28 smithi146 ceph-mon[103278]: pgmap v6045: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:41:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-mon[91136]: pgmap v6046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:41:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:41:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:41:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-mon[99959]: pgmap v6046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:41:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:30 smithi146 ceph-mon[103278]: pgmap v6046: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:41:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:30.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:30.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:30.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:32 smithi078 ceph-mon[91136]: pgmap v6047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:32 smithi078 ceph-mon[99959]: pgmap v6047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:32 smithi146 ceph-mon[103278]: pgmap v6047: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:34 smithi146 ceph-mon[103278]: pgmap v6048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:34 smithi078 ceph-mon[91136]: pgmap v6048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:34 smithi078 ceph-mon[99959]: pgmap v6048: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:36 smithi146 ceph-mon[103278]: pgmap v6049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:36 smithi078 ceph-mon[91136]: pgmap v6049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:36 smithi078 ceph-mon[99959]: pgmap v6049: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:38 smithi146 ceph-mon[103278]: pgmap v6050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:38 smithi078 ceph-mon[91136]: pgmap v6050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:38 smithi078 ceph-mon[99959]: pgmap v6050: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:40.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:40 smithi146 ceph-mon[103278]: pgmap v6051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:40.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:40.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-mon[99959]: pgmap v6051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:40.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:40.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-mon[91136]: pgmap v6051: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:40.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:40.974 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:41:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:41:41.241 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:40.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:41.241 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:40.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:41.241 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:40.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:42.346 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:41:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:42 smithi078 ceph-mon[99959]: pgmap v6052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:42 smithi078 ceph-mon[91136]: pgmap v6052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:42.909 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:41:42.909 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 3h - - 2024-09-20T23:41:42.909 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 3h - - 2024-09-20T23:41:42.909 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 19s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 14s ago 3h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 19s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 14s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 19s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 19s ago 3h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 14s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 19s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 19s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 14s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 19s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 19s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 19s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 19s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 14s ago 3h 92.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 14s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:41:42.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 14s ago 3h 80.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:41:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:41:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 14s ago 3h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:41:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 19s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:41:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:41:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 19s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:41:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:41:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:42 smithi146 ceph-mon[103278]: pgmap v6052: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:41:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:41:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:41:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:41:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:41:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:41:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:41:43.503 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:41:43.504 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:41:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[91136]: from='client.57759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[91136]: from='client.57765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/841254265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[99959]: from='client.57759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[99959]: from='client.57765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/841254265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:43.829 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:41:43.830 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:41:43.830 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:43 smithi146 ceph-mon[103278]: from='client.57759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:43 smithi146 ceph-mon[103278]: from='client.57765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/841254265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: from='client.57771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: pgmap v6053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: from='client.77699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2681985230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: from='client.57771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: pgmap v6053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: from='client.77699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2681985230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: from='client.57771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: pgmap v6053: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: from='client.77699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2681985230' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:41:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:46 smithi146 ceph-mon[103278]: pgmap v6054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:46 smithi078 ceph-mon[91136]: pgmap v6054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:46 smithi078 ceph-mon[99959]: pgmap v6054: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:48 smithi146 ceph-mon[103278]: pgmap v6055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[91136]: pgmap v6055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[99959]: pgmap v6055: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:41:50.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-mon[91136]: pgmap v6056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:50.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:50.927 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:41:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:41:50.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-mon[99959]: pgmap v6056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:50.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:50 smithi146 ceph-mon[103278]: pgmap v6056: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:41:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:50.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:50.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:41:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:41:50.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:41:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:52 smithi146 ceph-mon[103278]: pgmap v6057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:52 smithi078 ceph-mon[91136]: pgmap v6057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:52 smithi078 ceph-mon[99959]: pgmap v6057: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:54 smithi146 ceph-mon[103278]: pgmap v6058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:54 smithi078 ceph-mon[91136]: pgmap v6058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:54 smithi078 ceph-mon[99959]: pgmap v6058: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:56 smithi146 ceph-mon[103278]: pgmap v6059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:56 smithi078 ceph-mon[91136]: pgmap v6059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:56 smithi078 ceph-mon[99959]: pgmap v6059: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:41:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:58 smithi146 ceph-mon[103278]: pgmap v6060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:59.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:58 smithi078 ceph-mon[91136]: pgmap v6060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:41:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:58 smithi078 ceph-mon[99959]: pgmap v6060: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:41:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:00.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-mon[99959]: pgmap v6061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:00.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:00.975 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:42:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:42:00.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-mon[91136]: pgmap v6061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:00.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:00 smithi146 ceph-mon[103278]: pgmap v6061: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:00.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:00.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:00.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:02.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:02 smithi078 ceph-mon[91136]: pgmap v6062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:02.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:02 smithi078 ceph-mon[99959]: pgmap v6062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:02 smithi146 ceph-mon[103278]: pgmap v6062: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:04 smithi146 ceph-mon[103278]: pgmap v6063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[91136]: pgmap v6063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[99959]: pgmap v6063: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:06 smithi146 ceph-mon[103278]: pgmap v6064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:06 smithi078 ceph-mon[91136]: pgmap v6064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:06 smithi078 ceph-mon[99959]: pgmap v6064: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:08 smithi146 ceph-mon[103278]: pgmap v6065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:09.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[91136]: pgmap v6065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:09.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:09.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:09.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:09.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[99959]: pgmap v6065: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:09.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:09.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:09.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:10.975 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:42:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:42:10.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-mon[99959]: pgmap v6066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:10.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:10.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-mon[91136]: pgmap v6066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:10.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:10 smithi146 ceph-mon[103278]: pgmap v6066: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:10.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:10.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:10.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:12 smithi146 ceph-mon[103278]: pgmap v6067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:12 smithi078 ceph-mon[91136]: pgmap v6067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:12 smithi078 ceph-mon[99959]: pgmap v6067: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:13 smithi078 ceph-mon[99959]: pgmap v6068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:13.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:13 smithi078 ceph-mon[91136]: pgmap v6068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:13.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:13 smithi146 ceph-mon[103278]: pgmap v6068: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 3h - - 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 3h - - 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 10s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 6s ago 3h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 10s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 6s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 10s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 10s ago 3h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:42:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 6s ago 3h 238M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 10s ago 3h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 10s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 6s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 10s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 10s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 10s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 10s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 6s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 6s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 6s ago 3h 80.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 6s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 6s ago 3h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 10s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:42:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 6s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:42:14.662 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 10s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:42:14.662 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 6s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:42:14.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:42:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:14 smithi078 ceph-mon[99959]: from='client.77711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:14 smithi078 ceph-mon[99959]: from='client.57801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:14.972 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:14 smithi078 ceph-mon[99959]: from='client.77723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:14.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:14 smithi078 ceph-mon[91136]: from='client.77711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:14.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:14 smithi078 ceph-mon[91136]: from='client.57801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:14.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:14 smithi078 ceph-mon[91136]: from='client.77723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:14 smithi146 ceph-mon[103278]: from='client.77711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:14 smithi146 ceph-mon[103278]: from='client.57801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:14 smithi146 ceph-mon[103278]: from='client.77723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:42:15.242 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:42:15.243 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:42:15.243 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:42:15.243 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:42:15.243 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:42:15.243 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:42:15.551 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:42:15.551 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:15.551 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:15 smithi146 ceph-mon[103278]: pgmap v6069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2626574277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:15 smithi146 ceph-mon[103278]: from='client.57819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3126847600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[91136]: pgmap v6069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2626574277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[91136]: from='client.57819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3126847600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[99959]: pgmap v6069: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2626574277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[99959]: from='client.57819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3126847600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:17 smithi146 ceph-mon[103278]: pgmap v6070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:17 smithi078 ceph-mon[91136]: pgmap v6070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:17 smithi078 ceph-mon[99959]: pgmap v6070: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:19 smithi146 ceph-mon[103278]: pgmap v6071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:19 smithi078 ceph-mon[91136]: pgmap v6071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:19 smithi078 ceph-mon[99959]: pgmap v6071: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:20.976 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:42:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:42:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:20.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:20.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:20.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:21 smithi146 ceph-mon[103278]: pgmap v6072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:21 smithi078 ceph-mon[91136]: pgmap v6072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:21 smithi078 ceph-mon[99959]: pgmap v6072: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:24 smithi146 ceph-mon[103278]: pgmap v6073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[91136]: pgmap v6073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[99959]: pgmap v6073: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:26 smithi146 ceph-mon[103278]: pgmap v6074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:26 smithi078 ceph-mon[91136]: pgmap v6074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:26 smithi078 ceph-mon[99959]: pgmap v6074: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:28 smithi146 ceph-mon[103278]: pgmap v6075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:28 smithi078 ceph-mon[91136]: pgmap v6075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:28 smithi078 ceph-mon[99959]: pgmap v6075: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:30.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-mon[91136]: pgmap v6076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:30.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:30.976 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:42:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:42:30.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-mon[99959]: pgmap v6076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:30.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:30 smithi146 ceph-mon[103278]: pgmap v6076: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:30.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:30.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:30.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:32 smithi146 ceph-mon[103278]: pgmap v6077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:32 smithi078 ceph-mon[99959]: pgmap v6077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:32 smithi078 ceph-mon[91136]: pgmap v6077: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:34 smithi146 ceph-mon[103278]: pgmap v6078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:34 smithi078 ceph-mon[99959]: pgmap v6078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:34 smithi078 ceph-mon[91136]: pgmap v6078: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:36 smithi146 ceph-mon[103278]: pgmap v6079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:36 smithi078 ceph-mon[91136]: pgmap v6079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:36 smithi078 ceph-mon[99959]: pgmap v6079: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:38 smithi146 ceph-mon[103278]: pgmap v6080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:38 smithi078 ceph-mon[91136]: pgmap v6080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:38 smithi078 ceph-mon[99959]: pgmap v6080: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:40.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-mon[99959]: pgmap v6081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:40.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:40.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-mon[91136]: pgmap v6081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:40.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:40.978 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:42:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:42:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:40 smithi146 ceph-mon[103278]: pgmap v6081: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:40.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:40.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:40.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:42.867 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:42 smithi146 ceph-mon[103278]: pgmap v6082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:42 smithi078 ceph-mon[99959]: pgmap v6082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:42 smithi078 ceph-mon[91136]: pgmap v6082: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:43.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:44 smithi146 ceph-mon[103278]: pgmap v6083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:45.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[91136]: pgmap v6083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[99959]: pgmap v6083: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:45.847 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 3h - - 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 3h - - 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 2s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 17s ago 3h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 17s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 3h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 17s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:42:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 2s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 17s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 2s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 2s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 2s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 17s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 17s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 17s ago 3h 80.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 17s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 17s ago 3h 70.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 2s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:42:46.468 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 17s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:42:46.469 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 2s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:42:46.469 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 17s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:42:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:42:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:42:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:42:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:42:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:42:46.779 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:42:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:46 smithi146 ceph-mon[103278]: pgmap v6084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:46 smithi078 ceph-mon[99959]: pgmap v6084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:46 smithi078 ceph-mon[91136]: pgmap v6084: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:42:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:42:47.047 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:42:47.047 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:42:47.047 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:42:47.353 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:42:47.353 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:42:47.353 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:42:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:47 smithi146 ceph-mon[103278]: from='client.77747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:47 smithi146 ceph-mon[103278]: from='client.57837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:47 smithi146 ceph-mon[103278]: from='client.57843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4207432641' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:48.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/644743971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[91136]: from='client.77747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[91136]: from='client.57837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[91136]: from='client.57843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4207432641' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/644743971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[99959]: from='client.77747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[99959]: from='client.57837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[99959]: from='client.57843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4207432641' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/644743971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:42:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:48 smithi146 ceph-mon[103278]: pgmap v6085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:48 smithi146 ceph-mon[103278]: from='client.57855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:48 smithi078 ceph-mon[99959]: pgmap v6085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:48 smithi078 ceph-mon[99959]: from='client.57855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:48 smithi078 ceph-mon[91136]: pgmap v6085: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:48 smithi078 ceph-mon[91136]: from='client.57855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:49 smithi146 ceph-mon[103278]: pgmap v6086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[91136]: pgmap v6086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:42:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[99959]: pgmap v6086: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:42:50.978 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:42:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:42:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:50.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:50.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:42:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:42:50.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:42:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:52 smithi146 ceph-mon[103278]: pgmap v6087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:52 smithi078 ceph-mon[91136]: pgmap v6087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:52 smithi078 ceph-mon[99959]: pgmap v6087: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:54 smithi146 ceph-mon[103278]: pgmap v6088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:54 smithi078 ceph-mon[91136]: pgmap v6088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:54 smithi078 ceph-mon[99959]: pgmap v6088: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:42:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:56 smithi146 ceph-mon[103278]: pgmap v6089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:56 smithi078 ceph-mon[91136]: pgmap v6089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:56 smithi078 ceph-mon[99959]: pgmap v6089: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:58 smithi146 ceph-mon[103278]: pgmap v6090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:58 smithi078 ceph-mon[91136]: pgmap v6090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:58 smithi078 ceph-mon[99959]: pgmap v6090: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:42:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:42:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-mon[91136]: pgmap v6091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-mon[99959]: pgmap v6091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:00 smithi146 ceph-mon[103278]: pgmap v6091: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:00.979 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:43:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:43:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:00.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:00.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:00.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:02 smithi078 ceph-mon[91136]: pgmap v6092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:02 smithi078 ceph-mon[99959]: pgmap v6092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:02.337 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:02 smithi146 ceph-mon[103278]: pgmap v6092: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:04.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:04 smithi146 ceph-mon[103278]: pgmap v6093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:04.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:04 smithi078 ceph-mon[99959]: pgmap v6093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:04.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:04 smithi078 ceph-mon[91136]: pgmap v6093: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.532 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:05.532 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:06 smithi146 ceph-mon[103278]: pgmap v6094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:06.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:06 smithi078 ceph-mon[99959]: pgmap v6094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:06.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:06 smithi078 ceph-mon[91136]: pgmap v6094: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:08 smithi146 ceph-mon[103278]: pgmap v6095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:08 smithi078 ceph-mon[91136]: pgmap v6095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:08 smithi078 ceph-mon[99959]: pgmap v6095: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:09 smithi146 ceph-mon[103278]: pgmap v6096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[91136]: pgmap v6096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[99959]: pgmap v6096: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:10.979 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:43:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:43:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:10.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:10.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:10.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:12 smithi146 ceph-mon[103278]: pgmap v6097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:12 smithi078 ceph-mon[91136]: pgmap v6097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:12 smithi078 ceph-mon[99959]: pgmap v6097: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:14 smithi146 ceph-mon[103278]: pgmap v6098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:14 smithi078 ceph-mon[91136]: pgmap v6098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:14 smithi078 ceph-mon[99959]: pgmap v6098: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:16 smithi078 ceph-mon[91136]: pgmap v6099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:16 smithi078 ceph-mon[99959]: pgmap v6099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:16 smithi146 ceph-mon[103278]: pgmap v6099: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:17.649 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:43:18.175 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:43:18.175 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 3h - - 2024-09-20T23:43:18.175 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 3h - - 2024-09-20T23:43:18.175 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 9s ago 3h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 9s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 3h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 9s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 9s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 13s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 13s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 13s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 9s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:43:18.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 9s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 9s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 9s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 9s ago 3h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 9s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:43:18.177 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 9s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:43:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:18 smithi078 ceph-mon[91136]: pgmap v6100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:18 smithi078 ceph-mon[91136]: from='client.77783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:18 smithi078 ceph-mon[99959]: pgmap v6100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:18 smithi078 ceph-mon[99959]: from='client.77783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:43:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:43:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:43:18.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:43:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:18 smithi146 ceph-mon[103278]: pgmap v6100: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:18 smithi146 ceph-mon[103278]: from='client.77783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:43:18.757 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:43:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:43:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:43:18.758 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:43:18.758 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:43:19.069 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:43:19.069 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:19.069 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:19.069 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:19 smithi078 ceph-mon[91136]: from='client.77789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:19 smithi078 ceph-mon[91136]: from='client.77795 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2261548065' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:19 smithi078 ceph-mon[99959]: from='client.77789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:19 smithi078 ceph-mon[99959]: from='client.77795 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2261548065' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:19 smithi146 ceph-mon[103278]: from='client.77789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:19 smithi146 ceph-mon[103278]: from='client.77795 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2261548065' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[91136]: from='client.57891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[91136]: pgmap v6101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1791213405' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[99959]: from='client.57891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[99959]: pgmap v6101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1791213405' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:20 smithi146 ceph-mon[103278]: from='client.57891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:20 smithi146 ceph-mon[103278]: pgmap v6101: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1791213405' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:20.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:43:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:43:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:20.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:20.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:20.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:22 smithi078 ceph-mon[99959]: pgmap v6102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:22 smithi078 ceph-mon[91136]: pgmap v6102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:22 smithi146 ceph-mon[103278]: pgmap v6102: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:24 smithi146 ceph-mon[103278]: pgmap v6103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:24 smithi078 ceph-mon[99959]: pgmap v6103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:24 smithi078 ceph-mon[91136]: pgmap v6103: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:25.597 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.597 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.597 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:25.618 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:26.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:26 smithi078 ceph-mon[91136]: pgmap v6104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:26.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:26 smithi078 ceph-mon[99959]: pgmap v6104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:26 smithi146 ceph-mon[103278]: pgmap v6104: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:28.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:28 smithi146 ceph-mon[103278]: pgmap v6105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:28 smithi078 ceph-mon[91136]: pgmap v6105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:28 smithi078 ceph-mon[99959]: pgmap v6105: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-mon[91136]: pgmap v6106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-mon[99959]: pgmap v6106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:30 smithi146 ceph-mon[103278]: pgmap v6106: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:30.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:43:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:43:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:30.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:30.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:30.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:32 smithi146 ceph-mon[103278]: pgmap v6107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:32 smithi078 ceph-mon[91136]: pgmap v6107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:32 smithi078 ceph-mon[99959]: pgmap v6107: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:34 smithi146 ceph-mon[103278]: pgmap v6108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:34 smithi078 ceph-mon[91136]: pgmap v6108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:34 smithi078 ceph-mon[99959]: pgmap v6108: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:36 smithi078 ceph-mon[91136]: pgmap v6109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:36 smithi078 ceph-mon[99959]: pgmap v6109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:36 smithi146 ceph-mon[103278]: pgmap v6109: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:38 smithi146 ceph-mon[103278]: pgmap v6110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:38 smithi078 ceph-mon[91136]: pgmap v6110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:38 smithi078 ceph-mon[99959]: pgmap v6110: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:40.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-mon[91136]: pgmap v6111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:40.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:40.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-mon[99959]: pgmap v6111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:40.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:40.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:43:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:43:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:40 smithi146 ceph-mon[103278]: pgmap v6111: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:40.980Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:40.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:40.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:42 smithi146 ceph-mon[103278]: pgmap v6112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:42 smithi078 ceph-mon[91136]: pgmap v6112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:42 smithi078 ceph-mon[99959]: pgmap v6112: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:43.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[99959]: pgmap v6113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[91136]: pgmap v6113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:44 smithi146 ceph-mon[103278]: pgmap v6113: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:46.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:46 smithi078 ceph-mon[91136]: pgmap v6114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:46.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:46 smithi078 ceph-mon[99959]: pgmap v6114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:46 smithi146 ceph-mon[103278]: pgmap v6114: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:48 smithi146 ceph-mon[103278]: pgmap v6115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:48 smithi078 ceph-mon[91136]: pgmap v6115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:48 smithi078 ceph-mon[99959]: pgmap v6115: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:43:49.897 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:43:49.897 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 3h - - 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 3h - - 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 5s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 0s ago 3h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 0s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 5s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 5s ago 3h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 0s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 5s ago 3h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 5s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 0s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 5s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:43:49.898 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 5s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 5s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 5s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 0s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 0s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 0s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 0s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 0s ago 3h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 5s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 0s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 5s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:43:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 0s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:43:50.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:43:50.212 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:43:50.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: pgmap v6116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:50.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: from='client.57903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[99959]: from='client.77825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: pgmap v6116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: from='client.57903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:50.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-mon[91136]: from='client.77825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:50.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:43:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:43:50.495 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: pgmap v6116: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: from='client.57903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:43:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:50 smithi146 ceph-mon[103278]: from='client.77825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:43:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:43:50.815 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:43:50.816 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:43:50.816 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:50.816 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:43:51.127 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:50.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:51.127 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:50.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:43:51.127 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:43:50.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:43:51.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[99959]: from='client.77831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:51.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1319345051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:51.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[99959]: from='client.57927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:51.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2626060063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:51.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[91136]: from='client.77831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:51.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1319345051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:51.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[91136]: from='client.57927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:51.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2626060063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:51 smithi146 ceph-mon[103278]: from='client.77831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1319345051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:43:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:51 smithi146 ceph-mon[103278]: from='client.57927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2626060063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:43:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:52 smithi146 ceph-mon[103278]: pgmap v6117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:52 smithi078 ceph-mon[91136]: pgmap v6117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:52 smithi078 ceph-mon[99959]: pgmap v6117: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:54 smithi146 ceph-mon[103278]: pgmap v6118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:54 smithi078 ceph-mon[91136]: pgmap v6118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:54 smithi078 ceph-mon[99959]: pgmap v6118: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:43:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:56 smithi146 ceph-mon[103278]: pgmap v6119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:56 smithi078 ceph-mon[91136]: pgmap v6119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:56 smithi078 ceph-mon[99959]: pgmap v6119: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:58 smithi146 ceph-mon[103278]: pgmap v6120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:58 smithi078 ceph-mon[91136]: pgmap v6120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:58 smithi078 ceph-mon[99959]: pgmap v6120: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:43:59.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:43:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:59.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:43:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:43:59.487 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:43:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:00 smithi146 ceph-mon[103278]: pgmap v6121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-mon[91136]: pgmap v6121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-mon[99959]: pgmap v6121: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:00.908 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:44:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:44:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:00.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:00.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:00.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:02 smithi146 ceph-mon[103278]: pgmap v6122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:02 smithi078 ceph-mon[91136]: pgmap v6122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:02 smithi078 ceph-mon[99959]: pgmap v6122: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:04 smithi146 ceph-mon[103278]: pgmap v6123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:04 smithi078 ceph-mon[99959]: pgmap v6123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:04 smithi078 ceph-mon[91136]: pgmap v6123: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:05.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:05.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:05.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:06.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:06 smithi078 ceph-mon[99959]: pgmap v6124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:06.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:06 smithi078 ceph-mon[91136]: pgmap v6124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:06 smithi146 ceph-mon[103278]: pgmap v6124: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:08.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:08 smithi078 ceph-mon[91136]: pgmap v6125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:08.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:08 smithi078 ceph-mon[99959]: pgmap v6125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:08 smithi146 ceph-mon[103278]: pgmap v6125: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:10 smithi146 ceph-mon[103278]: pgmap v6126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[91136]: pgmap v6126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[99959]: pgmap v6126: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:10.983 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:44:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:44:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:10.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:10.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:10.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:12 smithi146 ceph-mon[103278]: pgmap v6127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:12 smithi078 ceph-mon[91136]: pgmap v6127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:12 smithi078 ceph-mon[99959]: pgmap v6127: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:14 smithi146 ceph-mon[103278]: pgmap v6128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:14 smithi078 ceph-mon[91136]: pgmap v6128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:14 smithi078 ceph-mon[99959]: pgmap v6128: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:16 smithi146 ceph-mon[103278]: pgmap v6129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:16 smithi078 ceph-mon[91136]: pgmap v6129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:16 smithi078 ceph-mon[99959]: pgmap v6129: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:18 smithi146 ceph-mon[103278]: pgmap v6130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:18 smithi078 ceph-mon[91136]: pgmap v6130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:18 smithi078 ceph-mon[99959]: pgmap v6130: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-mon[91136]: pgmap v6131: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-mon[99959]: pgmap v6131: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:20 smithi146 ceph-mon[103278]: pgmap v6131: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:20.801 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:44:20] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:44:21.115 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:20.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:21.116 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:20.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:21.116 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:20.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:21.116 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:44:21.679 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:44:21.679 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 3h - - 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 3h - - 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 17s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 12s ago 3h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 17s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 17s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 17s ago 3h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 17s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 17s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 12s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:44:21.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 17s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 17s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 17s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 17s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 12s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 12s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 12s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 12s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 3h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 17s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 12s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 17s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:44:21.681 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 12s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:44:21.998 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:44:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:44:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:44:21.999 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:44:22.267 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:44:22.420 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:44:22.420 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:44:22.420 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:44:22.420 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:44:22.421 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:44:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[91136]: pgmap v6132: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[91136]: from='client.77855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[91136]: from='client.57945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[91136]: from='client.77867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1550001772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[99959]: pgmap v6132: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[99959]: from='client.77855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[99959]: from='client.57945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[99959]: from='client.77867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1550001772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:22.575 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:44:22.575 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:22.575 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:22 smithi146 ceph-mon[103278]: pgmap v6132: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:22 smithi146 ceph-mon[103278]: from='client.77855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:22 smithi146 ceph-mon[103278]: from='client.57945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:22 smithi146 ceph-mon[103278]: from='client.77867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1550001772' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:23 smithi078 ceph-mon[91136]: from='client.77879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2337537931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:23 smithi078 ceph-mon[99959]: from='client.77879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2337537931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:23 smithi146 ceph-mon[103278]: from='client.77879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2337537931' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:24.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:24 smithi146 ceph-mon[103278]: pgmap v6133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:24 smithi078 ceph-mon[99959]: pgmap v6133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:24 smithi078 ceph-mon[91136]: pgmap v6133: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.032 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 23:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T23:44:25.848795553Z level=info msg="Completed cleanup jobs" duration=12.771935ms 2024-09-20T23:44:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:26.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:26 smithi078 ceph-mon[99959]: pgmap v6134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:26.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:26 smithi078 ceph-mon[91136]: pgmap v6134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:26 smithi146 ceph-mon[103278]: pgmap v6134: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:29.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:28 smithi078 ceph-mon[91136]: pgmap v6135: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:29.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:29.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:28 smithi078 ceph-mon[99959]: pgmap v6135: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:29.015 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:28 smithi146 ceph-mon[103278]: pgmap v6135: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:30 smithi146 ceph-mon[103278]: pgmap v6136: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[91136]: pgmap v6136: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:44:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:44:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[99959]: pgmap v6136: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:31.258 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:30.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:31.258 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:30.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:31.258 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:30.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:32 smithi146 ceph-mon[103278]: pgmap v6137: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:32 smithi078 ceph-mon[91136]: pgmap v6137: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:32 smithi078 ceph-mon[99959]: pgmap v6137: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:34 smithi146 ceph-mon[103278]: pgmap v6138: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:34 smithi078 ceph-mon[91136]: pgmap v6138: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:34 smithi078 ceph-mon[99959]: pgmap v6138: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:36 smithi146 ceph-mon[103278]: pgmap v6139: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:36 smithi078 ceph-mon[91136]: pgmap v6139: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:36 smithi078 ceph-mon[99959]: pgmap v6139: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:38 smithi146 ceph-mon[103278]: pgmap v6140: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:38 smithi078 ceph-mon[99959]: pgmap v6140: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:38 smithi078 ceph-mon[91136]: pgmap v6140: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:40.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-mon[91136]: pgmap v6141: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:40.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:40.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-mon[99959]: pgmap v6141: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:40.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:40.986 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:44:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:44:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:40 smithi146 ceph-mon[103278]: pgmap v6141: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:40.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:40.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:40.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:42 smithi146 ceph-mon[103278]: pgmap v6142: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:42 smithi078 ceph-mon[99959]: pgmap v6142: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:42 smithi078 ceph-mon[91136]: pgmap v6142: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:44.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:44 smithi146 ceph-mon[103278]: pgmap v6143: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:44 smithi078 ceph-mon[99959]: pgmap v6143: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:44 smithi078 ceph-mon[91136]: pgmap v6143: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:46.099 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.099 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.099 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.099 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:45 smithi146 ceph-mon[103278]: pgmap v6144: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[99959]: pgmap v6144: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:45 smithi078 ceph-mon[91136]: pgmap v6144: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:48.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:48 smithi078 ceph-mon[99959]: pgmap v6145: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:48.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:48 smithi078 ceph-mon[91136]: pgmap v6145: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:48 smithi146 ceph-mon[103278]: pgmap v6145: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[91136]: pgmap v6146: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[99959]: pgmap v6146: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:50.595 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.596 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.596 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:50 smithi146 ceph-mon[103278]: pgmap v6146: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:44:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:44:50.986 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:44:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:44:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:50.985Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:50.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:44:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:44:50.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:44:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:52 smithi146 ceph-mon[103278]: pgmap v6147: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:52 smithi078 ceph-mon[99959]: pgmap v6147: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:52 smithi078 ceph-mon[91136]: pgmap v6147: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:44:53.394 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 3h - - 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 3h - - 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 8s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 3s ago 3h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 8s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 3s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 8s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 8s ago 3h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 3s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 8s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 8s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:44:53.395 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 3s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 8s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 8s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 8s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 8s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 3s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 3s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 3s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 3s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 3s ago 3h 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 8s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 3s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 8s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:44:53.396 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 3s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:53.708 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:44:53.709 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:44:53.976 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:44:53.977 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:44:53.977 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:44:54.284 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:44:54.284 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:44:54.284 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:44:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[91136]: from='client.57975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[91136]: pgmap v6148: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[91136]: from='client.77897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[91136]: from='client.77903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2044943263' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3388744026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[99959]: from='client.57975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[99959]: pgmap v6148: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[99959]: from='client.77897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[99959]: from='client.77903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2044943263' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3388744026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:54 smithi146 ceph-mon[103278]: from='client.57975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:54 smithi146 ceph-mon[103278]: pgmap v6148: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:44:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:54 smithi146 ceph-mon[103278]: from='client.77897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:54 smithi146 ceph-mon[103278]: from='client.77903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2044943263' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:44:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3388744026' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:44:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:55 smithi146 ceph-mon[103278]: from='client.57999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:55 smithi078 ceph-mon[91136]: from='client.57999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:55 smithi078 ceph-mon[99959]: from='client.57999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:56 smithi146 ceph-mon[103278]: pgmap v6149: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:56 smithi078 ceph-mon[91136]: pgmap v6149: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:56 smithi078 ceph-mon[99959]: pgmap v6149: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:58.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:58 smithi146 ceph-mon[103278]: pgmap v6150: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:58.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:58 smithi078 ceph-mon[91136]: pgmap v6150: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:44:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:58 smithi078 ceph-mon[99959]: pgmap v6150: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:44:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-mon[91136]: pgmap v6151: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-mon[99959]: pgmap v6151: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:45:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:45:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:00 smithi146 ceph-mon[103278]: pgmap v6151: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:00.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:00.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:00.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:02 smithi078 ceph-mon[91136]: pgmap v6152: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:02 smithi078 ceph-mon[99959]: pgmap v6152: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:02 smithi146 ceph-mon[103278]: pgmap v6152: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:04 smithi078 ceph-mon[99959]: pgmap v6153: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:04 smithi078 ceph-mon[91136]: pgmap v6153: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:04 smithi146 ceph-mon[103278]: pgmap v6153: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:06.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:05 smithi146 ceph-mon[103278]: pgmap v6154: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[99959]: pgmap v6154: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:05 smithi078 ceph-mon[91136]: pgmap v6154: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:08.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:08 smithi078 ceph-mon[91136]: pgmap v6155: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:08.393 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:08 smithi078 ceph-mon[99959]: pgmap v6155: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:08.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:08 smithi146 ceph-mon[103278]: pgmap v6155: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:10.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[91136]: pgmap v6156: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:10.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:10.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:10.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:10.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:10.690 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:45:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:45:10.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[99959]: pgmap v6156: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:10.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:10.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:10.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:10.691 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:10 smithi146 ceph-mon[103278]: pgmap v6156: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:10.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:10.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:10.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:12 smithi146 ceph-mon[103278]: pgmap v6157: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:12 smithi078 ceph-mon[91136]: pgmap v6157: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:12 smithi078 ceph-mon[99959]: pgmap v6157: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:14 smithi146 ceph-mon[103278]: pgmap v6158: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:14 smithi078 ceph-mon[91136]: pgmap v6158: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:14 smithi078 ceph-mon[99959]: pgmap v6158: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:16 smithi146 ceph-mon[103278]: pgmap v6159: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:16 smithi078 ceph-mon[91136]: pgmap v6159: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:16 smithi078 ceph-mon[99959]: pgmap v6159: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:18 smithi146 ceph-mon[103278]: pgmap v6160: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:18 smithi078 ceph-mon[91136]: pgmap v6160: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:18 smithi078 ceph-mon[99959]: pgmap v6160: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:20.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-mon[99959]: pgmap v6161: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:20.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:20.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-mon[91136]: pgmap v6161: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:20.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:20.989 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:45:20] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:45:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:20 smithi146 ceph-mon[103278]: pgmap v6161: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:20.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:20.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:20.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:22 smithi146 ceph-mon[103278]: pgmap v6162: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:22 smithi078 ceph-mon[91136]: pgmap v6162: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:22 smithi078 ceph-mon[99959]: pgmap v6162: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:45:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:24 smithi146 ceph-mon[103278]: pgmap v6163: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:24 smithi078 ceph-mon[91136]: pgmap v6163: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:24 smithi078 ceph-mon[99959]: pgmap v6163: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:25.125 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 3h - - 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 3h - - 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 0s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 15s ago 3h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 0s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 0s ago 3h 576M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 0s ago 3h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 3h 236M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 0s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 0s ago 3h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:45:25.126 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 15s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 0s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 0s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 0s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 0s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 15s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 15s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 15s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 3h 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 0s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 15s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 0s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:45:25.127 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 15s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:45:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:45:25.440 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:45:25.712 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:45:25.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[99959]: from='client.77927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1755223861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[91136]: from='client.77927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1755223861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:25.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:25 smithi146 ceph-mon[103278]: from='client.77927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:25.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:25.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1755223861' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:26.022 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:45:26.022 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:26.022 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:26.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:26 smithi146 ceph-mon[103278]: from='client.58017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:26.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:26 smithi146 ceph-mon[103278]: pgmap v6164: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:26.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:26 smithi146 ceph-mon[103278]: from='client.77939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:26.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:26 smithi146 ceph-mon[103278]: from='client.77951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:26.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1346565680' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[91136]: from='client.58017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[91136]: pgmap v6164: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[91136]: from='client.77939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[91136]: from='client.77951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1346565680' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[99959]: from='client.58017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[99959]: pgmap v6164: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[99959]: from='client.77939 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[99959]: from='client.77951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1346565680' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:28.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:28 smithi146 ceph-mon[103278]: pgmap v6165: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:28.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:28.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:28 smithi078 ceph-mon[91136]: pgmap v6165: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:28.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:28.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:28 smithi078 ceph-mon[99959]: pgmap v6165: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:28.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[91136]: pgmap v6166: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:45:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[99959]: pgmap v6166: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:30.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:30 smithi146 ceph-mon[103278]: pgmap v6166: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:30.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:30.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:30.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:32 smithi146 ceph-mon[103278]: pgmap v6167: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:32 smithi078 ceph-mon[91136]: pgmap v6167: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:32 smithi078 ceph-mon[99959]: pgmap v6167: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:34 smithi146 ceph-mon[103278]: pgmap v6168: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:34 smithi078 ceph-mon[91136]: pgmap v6168: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:34 smithi078 ceph-mon[99959]: pgmap v6168: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:36 smithi146 ceph-mon[103278]: pgmap v6169: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:36 smithi078 ceph-mon[99959]: pgmap v6169: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:36 smithi078 ceph-mon[91136]: pgmap v6169: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:38 smithi146 ceph-mon[103278]: pgmap v6170: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:38 smithi078 ceph-mon[91136]: pgmap v6170: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:38 smithi078 ceph-mon[99959]: pgmap v6170: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:40.990 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:45:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:45:40.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-mon[99959]: pgmap v6171: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:40.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:40.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-mon[91136]: pgmap v6171: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:40.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:40 smithi146 ceph-mon[103278]: pgmap v6171: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:40.989Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:40.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:40.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:42 smithi146 ceph-mon[103278]: pgmap v6172: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:42 smithi078 ceph-mon[91136]: pgmap v6172: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:42 smithi078 ceph-mon[99959]: pgmap v6172: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:44 smithi146 ceph-mon[103278]: pgmap v6173: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:44 smithi078 ceph-mon[99959]: pgmap v6173: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:44 smithi078 ceph-mon[91136]: pgmap v6173: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:46.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:46 smithi146 ceph-mon[103278]: pgmap v6174: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:46.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[99959]: pgmap v6174: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[91136]: pgmap v6174: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:46.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:48 smithi146 ceph-mon[103278]: pgmap v6175: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:48 smithi078 ceph-mon[99959]: pgmap v6175: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:48 smithi078 ceph-mon[91136]: pgmap v6175: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:50 smithi146 ceph-mon[103278]: pgmap v6176: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[99959]: pgmap v6176: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[91136]: pgmap v6176: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:45:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:45:50.883 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:45:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:45:51.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:50.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:51.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:50.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:45:51.203 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:45:50.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:45:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:52 smithi146 ceph-mon[103278]: pgmap v6177: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:52 smithi078 ceph-mon[91136]: pgmap v6177: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:52 smithi078 ceph-mon[99959]: pgmap v6177: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:54 smithi146 ceph-mon[103278]: pgmap v6178: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:54 smithi078 ceph-mon[91136]: pgmap v6178: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:54 smithi078 ceph-mon[99959]: pgmap v6178: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:45:56.314 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:45:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:56 smithi146 ceph-mon[103278]: pgmap v6179: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:56 smithi078 ceph-mon[99959]: pgmap v6179: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:56 smithi078 ceph-mon[91136]: pgmap v6179: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:56.844 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:45:56.844 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 3h - - 2024-09-20T23:45:56.844 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 3h - - 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 11s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 6s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 11s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 6s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 11s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 11s ago 3h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 6s ago 3h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 11s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 11s ago 3h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 6s ago 3h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 11s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 11s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 11s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:45:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 11s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 6s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 6s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 6s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 6s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 6s ago 3h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 11s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 6s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 11s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:45:56.846 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 6s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:45:57.154 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:45:57.154 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:45:57.155 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:45:57.156 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:45:57.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:45:57.156 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:45:57.426 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:45:57.426 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:45:57.426 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:45:57.426 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:45:57.426 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:45:57.426 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:45:57.427 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:45:57.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:57 smithi078 ceph-mon[99959]: from='client.77963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:57.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:57 smithi078 ceph-mon[99959]: from='client.77969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:57.427 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2656479195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:57.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:57 smithi078 ceph-mon[91136]: from='client.77963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:57.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:57 smithi078 ceph-mon[91136]: from='client.77969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:57.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2656479195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:57 smithi146 ceph-mon[103278]: from='client.77963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:57 smithi146 ceph-mon[103278]: from='client.77969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2656479195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:45:57.739 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:45:57.739 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:57.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:45:57.739 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:57.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:45:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:58 smithi146 ceph-mon[103278]: from='client.77975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:58 smithi146 ceph-mon[103278]: pgmap v6180: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:58 smithi146 ceph-mon[103278]: from='client.77987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3679080775' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[91136]: from='client.77975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[91136]: pgmap v6180: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[91136]: from='client.77987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3679080775' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[99959]: from='client.77975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[99959]: pgmap v6180: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:45:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[99959]: from='client.77987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3679080775' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:45:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:59.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:45:59.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:00 smithi146 ceph-mon[103278]: pgmap v6181: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-mon[91136]: pgmap v6181: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-mon[99959]: pgmap v6181: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:00.991 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:46:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:46:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:00.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:00.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:00.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:02 smithi146 ceph-mon[103278]: pgmap v6182: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:02 smithi078 ceph-mon[91136]: pgmap v6182: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:02 smithi078 ceph-mon[99959]: pgmap v6182: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:04 smithi146 ceph-mon[103278]: pgmap v6183: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:04 smithi078 ceph-mon[99959]: pgmap v6183: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:04 smithi078 ceph-mon[91136]: pgmap v6183: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:06 smithi146 ceph-mon[103278]: pgmap v6184: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[99959]: pgmap v6184: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[91136]: pgmap v6184: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:08 smithi146 ceph-mon[103278]: pgmap v6185: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:08 smithi078 ceph-mon[99959]: pgmap v6185: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:08 smithi078 ceph-mon[91136]: pgmap v6185: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[99959]: pgmap v6186: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[91136]: pgmap v6186: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:10 smithi146 ceph-mon[103278]: pgmap v6186: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:10.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:46:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:46:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:10.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:10.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:10.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:12.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:12 smithi078 ceph-mon[91136]: pgmap v6187: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:12.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:12 smithi078 ceph-mon[99959]: pgmap v6187: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:12 smithi146 ceph-mon[103278]: pgmap v6187: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:14 smithi146 ceph-mon[103278]: pgmap v6188: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:14 smithi078 ceph-mon[91136]: pgmap v6188: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:14 smithi078 ceph-mon[99959]: pgmap v6188: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:16 smithi146 ceph-mon[103278]: pgmap v6189: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:16 smithi078 ceph-mon[91136]: pgmap v6189: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:16 smithi078 ceph-mon[99959]: pgmap v6189: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:18 smithi146 ceph-mon[103278]: pgmap v6190: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:18 smithi078 ceph-mon[91136]: pgmap v6190: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:18 smithi078 ceph-mon[99959]: pgmap v6190: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:20 smithi146 ceph-mon[103278]: pgmap v6191: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-mon[91136]: pgmap v6191: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:46:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:46:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-mon[99959]: pgmap v6191: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:20.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:20.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:20.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:22 smithi146 ceph-mon[103278]: pgmap v6192: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:22 smithi078 ceph-mon[91136]: pgmap v6192: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:22 smithi078 ceph-mon[99959]: pgmap v6192: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:24 smithi146 ceph-mon[103278]: pgmap v6193: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:24 smithi078 ceph-mon[91136]: pgmap v6193: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:24 smithi078 ceph-mon[99959]: pgmap v6193: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:26 smithi146 ceph-mon[103278]: pgmap v6194: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[91136]: pgmap v6194: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[99959]: pgmap v6194: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:28.034 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:46:28.563 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 3h - - 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 3h - - 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 3s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 18s ago 3h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 3s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 18s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 3s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 3s ago 3h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 18s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 3s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 3s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:46:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 18s ago 3h 15.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 3s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 3s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 3s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 3s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 18s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 18s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 18s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 18s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 18s ago 3h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 3s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 18s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 3s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:46:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 18s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:46:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:28 smithi146 ceph-mon[103278]: pgmap v6195: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:28 smithi078 ceph-mon[91136]: pgmap v6195: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:28 smithi078 ceph-mon[99959]: pgmap v6195: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:28.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:46:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:46:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:46:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:46:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:46:28.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:46:29.142 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:46:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:46:29.142 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:46:29.143 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:46:29.449 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:46:29.449 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:46:29.449 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:46:29.449 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:46:29.449 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:46:29.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:29 smithi146 ceph-mon[103278]: from='client.77999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:29 smithi146 ceph-mon[103278]: from='client.78005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:29 smithi146 ceph-mon[103278]: from='client.58095 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3767805168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:29.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3987599907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[91136]: from='client.77999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[91136]: from='client.78005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[91136]: from='client.58095 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3767805168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3987599907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[99959]: from='client.77999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[99959]: from='client.78005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[99959]: from='client.58095 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3767805168' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:46:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3987599907' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:46:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:30 smithi146 ceph-mon[103278]: pgmap v6196: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:30 smithi146 ceph-mon[103278]: from='client.78023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:46:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:46:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[91136]: pgmap v6196: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[91136]: from='client.78023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[99959]: pgmap v6196: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[99959]: from='client.78023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:31.245 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:30.993Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:31.245 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:30.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:31.245 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:30.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:32.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:32 smithi078 ceph-mon[91136]: pgmap v6197: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:32.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:32 smithi078 ceph-mon[99959]: pgmap v6197: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:32 smithi146 ceph-mon[103278]: pgmap v6197: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:34 smithi146 ceph-mon[103278]: pgmap v6198: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:34 smithi078 ceph-mon[91136]: pgmap v6198: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:34 smithi078 ceph-mon[99959]: pgmap v6198: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:36 smithi146 ceph-mon[103278]: pgmap v6199: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:36 smithi078 ceph-mon[99959]: pgmap v6199: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:36 smithi078 ceph-mon[91136]: pgmap v6199: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:38 smithi146 ceph-mon[103278]: pgmap v6200: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:38 smithi078 ceph-mon[91136]: pgmap v6200: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:38 smithi078 ceph-mon[99959]: pgmap v6200: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:40 smithi146 ceph-mon[103278]: pgmap v6201: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-mon[99959]: pgmap v6201: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-mon[91136]: pgmap v6201: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:46:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:46:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:40.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:40.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:40.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:42 smithi146 ceph-mon[103278]: pgmap v6202: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:42 smithi078 ceph-mon[99959]: pgmap v6202: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:42 smithi078 ceph-mon[91136]: pgmap v6202: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:44 smithi146 ceph-mon[103278]: pgmap v6203: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:44 smithi078 ceph-mon[99959]: pgmap v6203: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:44 smithi078 ceph-mon[91136]: pgmap v6203: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:46 smithi146 ceph-mon[103278]: pgmap v6204: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[91136]: pgmap v6204: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[99959]: pgmap v6204: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:48.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:48 smithi146 ceph-mon[103278]: pgmap v6205: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:48 smithi078 ceph-mon[99959]: pgmap v6205: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:48 smithi078 ceph-mon[91136]: pgmap v6205: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:50 smithi146 ceph-mon[103278]: pgmap v6206: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:46:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:46:50.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[99959]: pgmap v6206: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:50.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[91136]: pgmap v6206: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:50.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:46:50.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:50.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:46:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:50.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:50.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:46:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:46:50.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:46:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:52 smithi146 ceph-mon[103278]: pgmap v6207: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:52 smithi078 ceph-mon[91136]: pgmap v6207: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:52 smithi078 ceph-mon[99959]: pgmap v6207: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:54 smithi078 ceph-mon[91136]: pgmap v6208: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:54 smithi078 ceph-mon[99959]: pgmap v6208: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:54 smithi146 ceph-mon[103278]: pgmap v6208: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:46:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:56 smithi146 ceph-mon[103278]: pgmap v6209: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:56 smithi078 ceph-mon[99959]: pgmap v6209: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:56 smithi078 ceph-mon[91136]: pgmap v6209: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:58 smithi146 ceph-mon[103278]: pgmap v6210: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:46:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:58 smithi078 ceph-mon[91136]: pgmap v6210: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:46:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:58 smithi078 ceph-mon[99959]: pgmap v6210: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:46:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:46:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:46:59.745 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:47:00.276 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 3h - - 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 14s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 14s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 14s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 14s ago 3h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 3h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 14s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 14s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 14s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:47:00.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 14s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 14s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 14s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 10s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 14s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 14s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:47:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:47:00.585 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:00.586 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:47:00.587 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:47:00.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:47:00.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:47:00.856 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:47:00.857 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:47:00.857 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:47:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:47:00.857 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[99959]: pgmap v6211: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[99959]: from='client.78035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3914078016' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[91136]: pgmap v6211: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[91136]: from='client.78035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:00.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3914078016' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:00 smithi146 ceph-mon[103278]: pgmap v6211: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:00 smithi146 ceph-mon[103278]: from='client.78035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3914078016' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:01.163 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:47:01.164 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:01.164 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:01.164 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:00.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:01.164 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:00.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:01.164 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:00.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:01 smithi146 ceph-mon[103278]: from='client.78041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:01 smithi146 ceph-mon[103278]: from='client.58131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2394906262' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:01 smithi078 ceph-mon[91136]: from='client.78041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:01 smithi078 ceph-mon[91136]: from='client.58131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2394906262' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:01 smithi078 ceph-mon[99959]: from='client.78041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:01 smithi078 ceph-mon[99959]: from='client.58131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2394906262' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:02 smithi146 ceph-mon[103278]: from='client.78059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:02 smithi146 ceph-mon[103278]: pgmap v6212: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:02 smithi078 ceph-mon[99959]: from='client.78059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:02 smithi078 ceph-mon[99959]: pgmap v6212: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:02 smithi078 ceph-mon[91136]: from='client.78059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:02 smithi078 ceph-mon[91136]: pgmap v6212: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:04.928 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:04 smithi078 ceph-mon[99959]: pgmap v6213: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:04.929 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:04 smithi078 ceph-mon[91136]: pgmap v6213: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:04 smithi146 ceph-mon[103278]: pgmap v6213: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:06.653 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[91136]: pgmap v6214: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:06.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:06.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:06.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:06 smithi146 ceph-mon[103278]: pgmap v6214: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[99959]: pgmap v6214: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:08.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:08 smithi146 ceph-mon[103278]: pgmap v6215: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:08 smithi078 ceph-mon[91136]: pgmap v6215: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:08 smithi078 ceph-mon[99959]: pgmap v6215: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:10.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[99959]: pgmap v6216: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:10.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:10.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:10.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:47:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[91136]: pgmap v6216: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:10.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:10 smithi146 ceph-mon[103278]: pgmap v6216: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:10.995Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:11.376 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:10.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:11.376 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:10.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:12.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:12 smithi078 ceph-mon[91136]: pgmap v6217: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:12.837 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:12 smithi078 ceph-mon[99959]: pgmap v6217: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:12 smithi146 ceph-mon[103278]: pgmap v6217: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:13.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:13.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:14 smithi078 ceph-mon[91136]: pgmap v6218: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:14 smithi078 ceph-mon[99959]: pgmap v6218: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:14 smithi146 ceph-mon[103278]: pgmap v6218: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:16 smithi146 ceph-mon[103278]: pgmap v6219: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:16 smithi078 ceph-mon[91136]: pgmap v6219: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:16 smithi078 ceph-mon[99959]: pgmap v6219: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:18 smithi146 ceph-mon[103278]: pgmap v6220: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:18 smithi078 ceph-mon[91136]: pgmap v6220: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:18 smithi078 ceph-mon[99959]: pgmap v6220: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:20.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-mon[99959]: pgmap v6221: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:20.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:20.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:47:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:47:20.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-mon[91136]: pgmap v6221: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:20.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:20 smithi146 ceph-mon[103278]: pgmap v6221: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:20.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:20.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:20.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:22 smithi146 ceph-mon[103278]: pgmap v6222: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:22 smithi078 ceph-mon[91136]: pgmap v6222: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:22 smithi078 ceph-mon[99959]: pgmap v6222: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:24 smithi146 ceph-mon[103278]: pgmap v6223: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:24 smithi078 ceph-mon[91136]: pgmap v6223: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:24 smithi078 ceph-mon[99959]: pgmap v6223: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:26 smithi146 ceph-mon[103278]: pgmap v6224: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[91136]: pgmap v6224: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[99959]: pgmap v6224: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:27.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:28 smithi146 ceph-mon[103278]: pgmap v6225: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:28 smithi078 ceph-mon[99959]: pgmap v6225: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:28 smithi078 ceph-mon[91136]: pgmap v6225: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:30 smithi146 ceph-mon[103278]: pgmap v6226: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.998 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:47:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[99959]: pgmap v6226: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[91136]: pgmap v6226: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:30.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:30.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:30.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:30.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:31.460 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:47:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 3h - - 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 3h - - 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 6s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 1s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 6s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 1s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 6s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:47:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 6s ago 3h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 1s ago 3h 239M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 6s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 6s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 1s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 6s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 6s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 6s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 6s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 1s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 1s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 1s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 1s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 1s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 6s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:47:31.992 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:47:31.993 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 6s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:47:31.993 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 1s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:47:32.570 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:47:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:47:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:47:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:47:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:47:32.570 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:47:32.571 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:47:32.885 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:47:32.885 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:47:32.885 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:32.885 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[91136]: pgmap v6227: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[91136]: from='client.78071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[91136]: from='client.58161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2487102916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[99959]: pgmap v6227: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[99959]: from='client.78071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[99959]: from='client.58161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:32.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2487102916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:32 smithi146 ceph-mon[103278]: pgmap v6227: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:32 smithi146 ceph-mon[103278]: from='client.78071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:32 smithi146 ceph-mon[103278]: from='client.58161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2487102916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:47:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:33 smithi078 ceph-mon[91136]: from='client.78083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:33 smithi078 ceph-mon[91136]: from='client.78095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4041232169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:33 smithi078 ceph-mon[99959]: from='client.78083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:33 smithi078 ceph-mon[99959]: from='client.78095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4041232169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:33 smithi146 ceph-mon[103278]: from='client.78083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:33 smithi146 ceph-mon[103278]: from='client.78095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4041232169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:47:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:34 smithi146 ceph-mon[103278]: pgmap v6228: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:35.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:34 smithi078 ceph-mon[91136]: pgmap v6228: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:35.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:34 smithi078 ceph-mon[99959]: pgmap v6228: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:36 smithi146 ceph-mon[103278]: pgmap v6229: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:36 smithi078 ceph-mon[91136]: pgmap v6229: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:36 smithi078 ceph-mon[99959]: pgmap v6229: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:38 smithi146 ceph-mon[103278]: pgmap v6230: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:38 smithi078 ceph-mon[91136]: pgmap v6230: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:38 smithi078 ceph-mon[99959]: pgmap v6230: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:40.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-mon[91136]: pgmap v6231: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:40.999 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:40.999 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-mon[99959]: pgmap v6231: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:41.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:41.000 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:47:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:47:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:40 smithi146 ceph-mon[103278]: pgmap v6231: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:40.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:40.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:40.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:42 smithi146 ceph-mon[103278]: pgmap v6232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:42 smithi078 ceph-mon[91136]: pgmap v6232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:42 smithi078 ceph-mon[99959]: pgmap v6232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:44 smithi146 ceph-mon[103278]: pgmap v6233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:44 smithi078 ceph-mon[91136]: pgmap v6233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:44 smithi078 ceph-mon[99959]: pgmap v6233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:46 smithi146 ceph-mon[103278]: pgmap v6234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[91136]: pgmap v6234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[99959]: pgmap v6234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:48 smithi146 ceph-mon[103278]: pgmap v6235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:48 smithi078 ceph-mon[91136]: pgmap v6235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:48 smithi078 ceph-mon[99959]: pgmap v6235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:50.861 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:50 smithi146 ceph-mon[103278]: pgmap v6236: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:50.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:50.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:47:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:47:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[91136]: pgmap v6236: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[99959]: pgmap v6236: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:47:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:50.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:47:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:50.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:50.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:47:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:47:51.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:47:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:52 smithi146 ceph-mon[103278]: pgmap v6237: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:52 smithi078 ceph-mon[99959]: pgmap v6237: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:52 smithi078 ceph-mon[91136]: pgmap v6237: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:54 smithi078 ceph-mon[91136]: pgmap v6238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:54 smithi078 ceph-mon[99959]: pgmap v6238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:54 smithi146 ceph-mon[103278]: pgmap v6238: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:47:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:56 smithi146 ceph-mon[103278]: pgmap v6239: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:56 smithi078 ceph-mon[91136]: pgmap v6239: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:56 smithi078 ceph-mon[99959]: pgmap v6239: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:58 smithi146 ceph-mon[103278]: pgmap v6240: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:58 smithi078 ceph-mon[91136]: pgmap v6240: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:47:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:58 smithi078 ceph-mon[99959]: pgmap v6240: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:47:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:59 smithi078 ceph-mon[91136]: pgmap v6241: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:47:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:59 smithi078 ceph-mon[99959]: pgmap v6241: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:47:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:59 smithi146 ceph-mon[103278]: pgmap v6241: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:47:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:01.000 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:48:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:48:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:00.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:01.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:01.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:02 smithi146 ceph-mon[103278]: pgmap v6242: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:02 smithi078 ceph-mon[91136]: pgmap v6242: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:02 smithi078 ceph-mon[99959]: pgmap v6242: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:03.185 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:48:03.711 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 3h - - 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 3h - - 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 17s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 13s ago 3h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 17s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 13s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 17s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 17s ago 3h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 13s ago 3h 240M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 17s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 17s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:48:03.712 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 13s ago 3h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 17s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 17s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 17s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 17s ago 3h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 13s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 13s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 13s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 13s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 13s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 17s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 17s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:48:03.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 13s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:48:04.024 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:48:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:48:04.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:48:04.026 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:48:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[91136]: pgmap v6243: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[91136]: from='client.78107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[91136]: from='client.58197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[91136]: from='client.78119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1791792436' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[99959]: pgmap v6243: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[99959]: from='client.78107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[99959]: from='client.58197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[99959]: from='client.78119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1791792436' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:48:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:48:04.297 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:48:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:04 smithi146 ceph-mon[103278]: pgmap v6243: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:04 smithi146 ceph-mon[103278]: from='client.78107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:04 smithi146 ceph-mon[103278]: from='client.58197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:04 smithi146 ceph-mon[103278]: from='client.78119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1791792436' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:04.608 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:48:04.608 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:48:04.608 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:48:04.608 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:48:04.608 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:48:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:05 smithi146 ceph-mon[103278]: from='client.78131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3270700411' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:05 smithi078 ceph-mon[91136]: from='client.78131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3270700411' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:05 smithi078 ceph-mon[99959]: from='client.78131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3270700411' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:06 smithi146 ceph-mon[103278]: pgmap v6244: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:06 smithi078 ceph-mon[91136]: pgmap v6244: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:06 smithi078 ceph-mon[99959]: pgmap v6244: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:08 smithi146 ceph-mon[103278]: pgmap v6245: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:08 smithi078 ceph-mon[91136]: pgmap v6245: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:08 smithi078 ceph-mon[99959]: pgmap v6245: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:10 smithi146 ceph-mon[103278]: pgmap v6246: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:10 smithi078 ceph-mon[99959]: pgmap v6246: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:10 smithi078 ceph-mon[91136]: pgmap v6246: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:11.001 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:48:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:48:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:11.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:11.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:11.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:11 smithi146 ceph-mon[103278]: pgmap v6247: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[91136]: pgmap v6247: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:11 smithi078 ceph-mon[99959]: pgmap v6247: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:14.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:14 smithi078 ceph-mon[91136]: pgmap v6248: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:14.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:14.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:14 smithi078 ceph-mon[99959]: pgmap v6248: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:14.485 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:14.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:14 smithi146 ceph-mon[103278]: pgmap v6248: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:14.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:16.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:16 smithi078 ceph-mon[91136]: pgmap v6249: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:16.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:16 smithi078 ceph-mon[99959]: pgmap v6249: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:16 smithi146 ceph-mon[103278]: pgmap v6249: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:18 smithi146 ceph-mon[103278]: pgmap v6250: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:18 smithi078 ceph-mon[91136]: pgmap v6250: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:18 smithi078 ceph-mon[99959]: pgmap v6250: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:20 smithi146 ceph-mon[103278]: pgmap v6251: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:20 smithi078 ceph-mon[91136]: pgmap v6251: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:20 smithi078 ceph-mon[99959]: pgmap v6251: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:21.002 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:48:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-20T23:48:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:21.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:21.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:21.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:22 smithi146 ceph-mon[103278]: pgmap v6252: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:22 smithi078 ceph-mon[91136]: pgmap v6252: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:22 smithi078 ceph-mon[99959]: pgmap v6252: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:24 smithi146 ceph-mon[103278]: pgmap v6253: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:24 smithi078 ceph-mon[91136]: pgmap v6253: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:24 smithi078 ceph-mon[99959]: pgmap v6253: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:26 smithi146 ceph-mon[103278]: pgmap v6254: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[91136]: pgmap v6254: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[99959]: pgmap v6254: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:28 smithi146 ceph-mon[103278]: pgmap v6255: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:28 smithi078 ceph-mon[91136]: pgmap v6255: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:28 smithi078 ceph-mon[99959]: pgmap v6255: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:30.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:30 smithi146 ceph-mon[103278]: pgmap v6256: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:30.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:31.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:30 smithi078 ceph-mon[99959]: pgmap v6256: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:31.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:31.004 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:48:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:48:31.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:30 smithi078 ceph-mon[91136]: pgmap v6256: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:31.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:31.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:31.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:31.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:31 smithi146 ceph-mon[103278]: pgmap v6257: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[91136]: pgmap v6257: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:31 smithi078 ceph-mon[99959]: pgmap v6257: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:34 smithi146 ceph-mon[103278]: pgmap v6258: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:34 smithi078 ceph-mon[91136]: pgmap v6258: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:34 smithi078 ceph-mon[99959]: pgmap v6258: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:34.904 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:48:35.442 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:48:35.442 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 3h - - 2024-09-20T23:48:35.442 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 3h - - 2024-09-20T23:48:35.442 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 9s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:48:35.442 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 4s ago 3h 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:48:35.442 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 9s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 4s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 9s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 9s ago 3h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 4s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 9s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 9s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 4s ago 3h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 9s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 9s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 9s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 9s ago 3h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 4s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 4s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:48:35.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 4s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:48:35.444 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 4s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:48:35.444 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 4s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:48:35.444 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 9s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:48:35.444 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:48:35.444 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 9s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:48:35.444 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 4s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:48:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:48:35.755 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:48:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:48:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:48:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:48:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:48:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:48:36.029 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[91136]: from='client.78143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[91136]: pgmap v6259: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[91136]: from='client.58233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[91136]: from='client.78155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/200210022' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[99959]: from='client.78143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[99959]: pgmap v6259: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[99959]: from='client.58233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[99959]: from='client.78155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/200210022' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:36.349 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:48:36.350 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:48:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:48:36.350 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:48:36.350 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:48:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:36 smithi146 ceph-mon[103278]: from='client.78143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:36 smithi146 ceph-mon[103278]: pgmap v6259: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:36 smithi146 ceph-mon[103278]: from='client.58233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:36 smithi146 ceph-mon[103278]: from='client.78155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:36.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/200210022' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:48:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:37 smithi078 ceph-mon[91136]: from='client.58251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4253000685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:37 smithi078 ceph-mon[99959]: from='client.58251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4253000685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:37 smithi146 ceph-mon[103278]: from='client.58251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4253000685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:48:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:38 smithi146 ceph-mon[103278]: pgmap v6260: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:38 smithi078 ceph-mon[91136]: pgmap v6260: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:38 smithi078 ceph-mon[99959]: pgmap v6260: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:40 smithi146 ceph-mon[103278]: pgmap v6261: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:40 smithi078 ceph-mon[91136]: pgmap v6261: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:40 smithi078 ceph-mon[99959]: pgmap v6261: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:41.003 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:48:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:48:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:41.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:41.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:41.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:42 smithi146 ceph-mon[103278]: pgmap v6262: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:42 smithi078 ceph-mon[91136]: pgmap v6262: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:42 smithi078 ceph-mon[99959]: pgmap v6262: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:44 smithi146 ceph-mon[103278]: pgmap v6263: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:44 smithi078 ceph-mon[91136]: pgmap v6263: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:44 smithi078 ceph-mon[99959]: pgmap v6263: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:46.396 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:46 smithi078 ceph-mon[99959]: pgmap v6264: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:46.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:46 smithi078 ceph-mon[91136]: pgmap v6264: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:46 smithi146 ceph-mon[103278]: pgmap v6264: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:48 smithi146 ceph-mon[103278]: pgmap v6265: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:48 smithi078 ceph-mon[91136]: pgmap v6265: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:48 smithi078 ceph-mon[99959]: pgmap v6265: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:50 smithi146 ceph-mon[103278]: pgmap v6266: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:48:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:48:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:50 smithi078 ceph-mon[91136]: pgmap v6266: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:50 smithi078 ceph-mon[99959]: pgmap v6266: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:48:51.270 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:51.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:51.271 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:51.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:48:51.271 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:48:51.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:48:52.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:52 smithi146 ceph-mon[103278]: pgmap v6267: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[91136]: pgmap v6267: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:48:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:52 smithi078 ceph-mon[99959]: pgmap v6267: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:54.408 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:54 smithi146 ceph-mon[103278]: pgmap v6268: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:54 smithi078 ceph-mon[91136]: pgmap v6268: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:54 smithi078 ceph-mon[99959]: pgmap v6268: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:48:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:56 smithi146 ceph-mon[103278]: pgmap v6269: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:56 smithi078 ceph-mon[91136]: pgmap v6269: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:56 smithi078 ceph-mon[99959]: pgmap v6269: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:58.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:58 smithi078 ceph-mon[91136]: pgmap v6270: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:58.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:58 smithi078 ceph-mon[99959]: pgmap v6270: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:58 smithi146 ceph-mon[103278]: pgmap v6270: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:48:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:48:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:00 smithi146 ceph-mon[103278]: pgmap v6271: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:00 smithi078 ceph-mon[91136]: pgmap v6271: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:00 smithi078 ceph-mon[99959]: pgmap v6271: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:01.005 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:49:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:49:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:01.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:01.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:01.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:02 smithi146 ceph-mon[103278]: pgmap v6272: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:02 smithi078 ceph-mon[91136]: pgmap v6272: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:02 smithi078 ceph-mon[99959]: pgmap v6272: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:04 smithi146 ceph-mon[103278]: pgmap v6273: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:04 smithi078 ceph-mon[99959]: pgmap v6273: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:04 smithi078 ceph-mon[91136]: pgmap v6273: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:06 smithi146 ceph-mon[103278]: pgmap v6274: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:06 smithi078 ceph-mon[99959]: pgmap v6274: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:06 smithi078 ceph-mon[91136]: pgmap v6274: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 3h - - 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 3h - - 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 0s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 16s ago 3h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 0s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 16s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:49:07.220 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 0s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 0s ago 3h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 16s ago 3h 242M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 0s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 0s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 16s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 0s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 0s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 0s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 0s ago 3h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 16s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 16s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 16s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 16s ago 3h 74.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 16s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:49:07.221 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 0s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:49:07.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:49:07.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 0s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:49:07.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 16s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:49:07.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:49:07.528 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:49:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[91136]: from='client.78179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1015022238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[99959]: from='client.78179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1015022238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:07.801 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:49:07.801 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:49:07.801 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:49:07.801 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:49:07.801 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:49:07.801 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:49:07.802 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:49:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:07 smithi146 ceph-mon[103278]: from='client.78179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1015022238' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:08.113 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:49:08.114 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:08.114 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:08.114 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[99959]: from='client.58269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[99959]: pgmap v6275: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[99959]: from='client.78191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4077517848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[91136]: from='client.58269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[91136]: pgmap v6275: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[91136]: from='client.78191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:08.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4077517848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:08 smithi146 ceph-mon[103278]: from='client.58269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:08 smithi146 ceph-mon[103278]: pgmap v6275: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:08 smithi146 ceph-mon[103278]: from='client.78191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4077517848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:09 smithi146 ceph-mon[103278]: from='client.58287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:09 smithi078 ceph-mon[91136]: from='client.58287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:09 smithi078 ceph-mon[99959]: from='client.58287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:11.006 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:49:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:49:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:10 smithi078 ceph-mon[99959]: pgmap v6276: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:11.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:11.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:10 smithi078 ceph-mon[91136]: pgmap v6276: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:11.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:10 smithi146 ceph-mon[103278]: pgmap v6276: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:11.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:11.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:11.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:12 smithi146 ceph-mon[103278]: pgmap v6277: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[91136]: pgmap v6277: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[99959]: pgmap v6277: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:14 smithi146 ceph-mon[103278]: pgmap v6278: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:14 smithi078 ceph-mon[91136]: pgmap v6278: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:14 smithi078 ceph-mon[99959]: pgmap v6278: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:16 smithi146 ceph-mon[103278]: pgmap v6279: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:16 smithi078 ceph-mon[91136]: pgmap v6279: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:16 smithi078 ceph-mon[99959]: pgmap v6279: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:18.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:18 smithi078 ceph-mon[91136]: pgmap v6280: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:18.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:18 smithi078 ceph-mon[99959]: pgmap v6280: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:18 smithi146 ceph-mon[103278]: pgmap v6280: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:20 smithi146 ceph-mon[103278]: pgmap v6281: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:20 smithi078 ceph-mon[91136]: pgmap v6281: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:20.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:20 smithi078 ceph-mon[99959]: pgmap v6281: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:21.007 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:49:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:49:21.283 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:21.006Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:21.284 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:21.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:21.284 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:21.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:22 smithi146 ceph-mon[103278]: pgmap v6282: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:22 smithi078 ceph-mon[91136]: pgmap v6282: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:22 smithi078 ceph-mon[99959]: pgmap v6282: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:24 smithi146 ceph-mon[103278]: pgmap v6283: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:24 smithi078 ceph-mon[99959]: pgmap v6283: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:24 smithi078 ceph-mon[91136]: pgmap v6283: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:26 smithi146 ceph-mon[103278]: pgmap v6284: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:26 smithi078 ceph-mon[99959]: pgmap v6284: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:26 smithi078 ceph-mon[91136]: pgmap v6284: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:28 smithi146 ceph-mon[103278]: pgmap v6285: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:28 smithi078 ceph-mon[91136]: pgmap v6285: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:28 smithi078 ceph-mon[99959]: pgmap v6285: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:31.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:30 smithi078 ceph-mon[91136]: pgmap v6286: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:31.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:31.009 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:49:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:49:31.009 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:30 smithi078 ceph-mon[99959]: pgmap v6286: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:31.009 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:30 smithi146 ceph-mon[103278]: pgmap v6286: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:31.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:31.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:31.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[91136]: pgmap v6287: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[99959]: pgmap v6287: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:32 smithi146 ceph-mon[103278]: pgmap v6287: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:34.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:34 smithi146 ceph-mon[103278]: pgmap v6288: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:34 smithi078 ceph-mon[99959]: pgmap v6288: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:34 smithi078 ceph-mon[91136]: pgmap v6288: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:36 smithi146 ceph-mon[103278]: pgmap v6289: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:36 smithi078 ceph-mon[91136]: pgmap v6289: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:36 smithi078 ceph-mon[99959]: pgmap v6289: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:38.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:38 smithi078 ceph-mon[91136]: pgmap v6290: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:38.407 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:38 smithi078 ceph-mon[99959]: pgmap v6290: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:38.407 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:49:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:38 smithi146 ceph-mon[103278]: pgmap v6290: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:38.997 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:49:38.997 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 3h - - 2024-09-20T23:49:38.997 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 3h - - 2024-09-20T23:49:38.997 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 12s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:49:38.997 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 7s ago 3h 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 12s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 7s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 12s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 12s ago 3h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 7s ago 3h 243M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 12s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 12s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 7s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 12s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 12s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 12s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 12s ago 3h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 7s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:49:38.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 7s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 7s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 7s ago 3h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 7s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 12s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 12s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:49:38.999 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 7s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:49:39.319 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:49:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:49:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:49:39.321 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:49:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:49:39.608 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:49:39.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:39 smithi078 ceph-mon[91136]: from='client.78215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:39.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:39 smithi078 ceph-mon[91136]: from='client.78221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:39.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:39 smithi078 ceph-mon[99959]: from='client.78215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:39.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:39 smithi078 ceph-mon[99959]: from='client.78221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:39 smithi146 ceph-mon[103278]: from='client.78215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:39 smithi146 ceph-mon[103278]: from='client.78221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:39.934 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:49:39.934 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:49:39.934 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:49:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[91136]: from='client.78227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[91136]: pgmap v6291: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/777285004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[91136]: from='client.78239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2238913721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[99959]: from='client.78227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[99959]: pgmap v6291: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/777285004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[99959]: from='client.78239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:40.584 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2238913721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:40 smithi146 ceph-mon[103278]: from='client.78227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:40 smithi146 ceph-mon[103278]: pgmap v6291: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/777285004' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:40 smithi146 ceph-mon[103278]: from='client.78239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2238913721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:49:41.008 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:49:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:49:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:41.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:41.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:41.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:42 smithi146 ceph-mon[103278]: pgmap v6292: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:42 smithi078 ceph-mon[91136]: pgmap v6292: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:42 smithi078 ceph-mon[99959]: pgmap v6292: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:44 smithi146 ceph-mon[103278]: pgmap v6293: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:44 smithi078 ceph-mon[91136]: pgmap v6293: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:44 smithi078 ceph-mon[99959]: pgmap v6293: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:46 smithi146 ceph-mon[103278]: pgmap v6294: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:46 smithi078 ceph-mon[91136]: pgmap v6294: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:46 smithi078 ceph-mon[99959]: pgmap v6294: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:47 smithi146 ceph-mon[103278]: pgmap v6295: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[99959]: pgmap v6295: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:47 smithi078 ceph-mon[91136]: pgmap v6295: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:50 smithi146 ceph-mon[103278]: pgmap v6296: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:50 smithi078 ceph-mon[91136]: pgmap v6296: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:50 smithi078 ceph-mon[99959]: pgmap v6296: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:49:51.009 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:49:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:49:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:51.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:51.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:49:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:49:51.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:49:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:52 smithi146 ceph-mon[103278]: pgmap v6297: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[99959]: pgmap v6297: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[91136]: pgmap v6297: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:49:54.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:54 smithi146 ceph-mon[103278]: pgmap v6298: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:54 smithi078 ceph-mon[91136]: pgmap v6298: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:54 smithi078 ceph-mon[99959]: pgmap v6298: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:49:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:56 smithi146 ceph-mon[103278]: pgmap v6299: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:56 smithi078 ceph-mon[91136]: pgmap v6299: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:56 smithi078 ceph-mon[99959]: pgmap v6299: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:58 smithi078 ceph-mon[91136]: pgmap v6300: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:58 smithi078 ceph-mon[99959]: pgmap v6300: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:49:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:58 smithi146 ceph-mon[103278]: pgmap v6300: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:49:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T23:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:00 smithi146 ceph-mon[103278]: pgmap v6301: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:00 smithi078 ceph-mon[99959]: pgmap v6301: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:00.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:00 smithi078 ceph-mon[91136]: pgmap v6301: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:00.771 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:00.771 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:50:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:50:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:01.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:01.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:01.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:01.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:02 smithi146 ceph-mon[103278]: pgmap v6302: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:02 smithi078 ceph-mon[91136]: pgmap v6302: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:02 smithi078 ceph-mon[99959]: pgmap v6302: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:04 smithi146 ceph-mon[103278]: pgmap v6303: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:04 smithi078 ceph-mon[91136]: pgmap v6303: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:04 smithi078 ceph-mon[99959]: pgmap v6303: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:06 smithi146 ceph-mon[103278]: pgmap v6304: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:06 smithi078 ceph-mon[91136]: pgmap v6304: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:06 smithi078 ceph-mon[99959]: pgmap v6304: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:07 smithi146 ceph-mon[103278]: pgmap v6305: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[91136]: pgmap v6305: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:07 smithi078 ceph-mon[99959]: pgmap v6305: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:10.234 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:50:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:10 smithi146 ceph-mon[103278]: pgmap v6306: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:10 smithi078 ceph-mon[99959]: pgmap v6306: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:10 smithi078 ceph-mon[91136]: pgmap v6306: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 3h - - 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 3h - - 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 3s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 19s ago 3h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 3s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:50:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 19s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 3s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 3s ago 3h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 19s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 3s ago 3h 252M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 3s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 19s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 3s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 3s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 3s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 3s ago 3h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 19s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 19s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:50:10.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 19s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:50:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 19s ago 3h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:50:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 19s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:50:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 3s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:50:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:50:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 3s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:50:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 19s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:50:11.011 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:50:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:50:11.072 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:50:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:50:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:50:11.074 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:50:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[91136]: from='client.58335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[91136]: from='client.78257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[91136]: from='client.78263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2843729129' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[99959]: from='client.58335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[99959]: from='client.78257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[99959]: from='client.78263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2843729129' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:11.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:11.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:11.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:50:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:50:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:50:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:50:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:50:11.348 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:50:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:11 smithi146 ceph-mon[103278]: from='client.58335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:11 smithi146 ceph-mon[103278]: from='client.78257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:11 smithi146 ceph-mon[103278]: from='client.78263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2843729129' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:11.663 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:11.663 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:11.663 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:11.664 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:11.664 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:12 smithi146 ceph-mon[103278]: pgmap v6307: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:12 smithi146 ceph-mon[103278]: from='client.78275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3107383156' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[99959]: pgmap v6307: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[99959]: from='client.78275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3107383156' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[91136]: pgmap v6307: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[91136]: from='client.78275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3107383156' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:14 smithi146 ceph-mon[103278]: pgmap v6308: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:14 smithi078 ceph-mon[91136]: pgmap v6308: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:14 smithi078 ceph-mon[99959]: pgmap v6308: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:16.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:16 smithi146 ceph-mon[103278]: pgmap v6309: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:16 smithi078 ceph-mon[91136]: pgmap v6309: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:16 smithi078 ceph-mon[99959]: pgmap v6309: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:18.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:18 smithi078 ceph-mon[91136]: pgmap v6310: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:18.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:18 smithi078 ceph-mon[99959]: pgmap v6310: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:18 smithi146 ceph-mon[103278]: pgmap v6310: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:20.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:20 smithi078 ceph-mon[91136]: pgmap v6311: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:20.836 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:20 smithi078 ceph-mon[99959]: pgmap v6311: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:20.836 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:50:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T23:50:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:20 smithi146 ceph-mon[103278]: pgmap v6311: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:21.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:21.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:21.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:21.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:22 smithi146 ceph-mon[103278]: pgmap v6312: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:22 smithi078 ceph-mon[91136]: pgmap v6312: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:22 smithi078 ceph-mon[99959]: pgmap v6312: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:24 smithi146 ceph-mon[103278]: pgmap v6313: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:24 smithi078 ceph-mon[91136]: pgmap v6313: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:24 smithi078 ceph-mon[99959]: pgmap v6313: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:26 smithi146 ceph-mon[103278]: pgmap v6314: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:26 smithi078 ceph-mon[91136]: pgmap v6314: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:26 smithi078 ceph-mon[99959]: pgmap v6314: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:28 smithi146 ceph-mon[103278]: pgmap v6315: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[91136]: pgmap v6315: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:28 smithi078 ceph-mon[99959]: pgmap v6315: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:30 smithi146 ceph-mon[103278]: pgmap v6316: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:30 smithi078 ceph-mon[91136]: pgmap v6316: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:30 smithi078 ceph-mon[99959]: pgmap v6316: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:31.012 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:50:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:50:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:31.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:31.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:31.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:32 smithi146 ceph-mon[103278]: pgmap v6317: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[99959]: pgmap v6317: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[91136]: pgmap v6317: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:34 smithi146 ceph-mon[103278]: pgmap v6318: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:34 smithi078 ceph-mon[91136]: pgmap v6318: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:34 smithi078 ceph-mon[99959]: pgmap v6318: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:36 smithi146 ceph-mon[103278]: pgmap v6319: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:36 smithi078 ceph-mon[91136]: pgmap v6319: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:36 smithi078 ceph-mon[99959]: pgmap v6319: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:38 smithi146 ceph-mon[103278]: pgmap v6320: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:38 smithi078 ceph-mon[91136]: pgmap v6320: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:38 smithi078 ceph-mon[99959]: pgmap v6320: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T23:50:40.874 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:40 smithi078 ceph-mon[99959]: pgmap v6321: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:40.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:40 smithi078 ceph-mon[91136]: pgmap v6321: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:40.875 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:50:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:50:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:40 smithi146 ceph-mon[103278]: pgmap v6321: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:41.181 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:41.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:41.265 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:41.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:41.265 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:41.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:41.959 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:50:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 3h - - 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 15s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 15s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 15s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 15s ago 3h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:50:42.487 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 3h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 15s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 15s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 15s ago 3h 79.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 15s ago 3h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 15s ago 3h 72.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 15s ago 3h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 3h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 3h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 3h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 10s ago 3h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:50:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 15s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:50:42.489 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:50:42.489 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 15s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:50:42.489 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:50:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:42 smithi078 ceph-mon[91136]: pgmap v6322: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:42 smithi078 ceph-mon[99959]: pgmap v6322: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:42.799 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:50:42.799 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:50:42.799 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:50:42.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:42.799 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:50:42.800 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:50:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:42 smithi146 ceph-mon[103278]: pgmap v6322: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:43.071 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:50:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:50:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:50:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:50:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:50:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:50:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:50:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:50:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:50:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:50:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:50:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:50:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:50:43.073 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:50:43.385 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:50:43.385 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:50:43.386 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:43 smithi146 ceph-mon[103278]: from='client.78287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:43 smithi146 ceph-mon[103278]: from='client.58377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:43 smithi146 ceph-mon[103278]: from='client.58383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2207198709' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2817059713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[91136]: from='client.78287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[91136]: from='client.58377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[91136]: from='client.58383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2207198709' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2817059713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[99959]: from='client.78287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[99959]: from='client.58377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[99959]: from='client.58383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2207198709' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2817059713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:50:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:44 smithi146 ceph-mon[103278]: from='client.78311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:44 smithi146 ceph-mon[103278]: pgmap v6323: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:44 smithi078 ceph-mon[99959]: from='client.78311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:44 smithi078 ceph-mon[99959]: pgmap v6323: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:44 smithi078 ceph-mon[91136]: from='client.78311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:44 smithi078 ceph-mon[91136]: pgmap v6323: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:46 smithi146 ceph-mon[103278]: pgmap v6324: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:46 smithi078 ceph-mon[91136]: pgmap v6324: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:46 smithi078 ceph-mon[99959]: pgmap v6324: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:48 smithi146 ceph-mon[103278]: pgmap v6325: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[91136]: pgmap v6325: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[99959]: pgmap v6325: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:50 smithi146 ceph-mon[103278]: pgmap v6326: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:50 smithi078 ceph-mon[91136]: pgmap v6326: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:50 smithi078 ceph-mon[99959]: pgmap v6326: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:50:51.013 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:50:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:50:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:51.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:51.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:50:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:50:51.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:50:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:50:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:52 smithi146 ceph-mon[103278]: pgmap v6327: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[99959]: pgmap v6327: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[91136]: pgmap v6327: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:50:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:52.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:50:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:54 smithi146 ceph-mon[103278]: pgmap v6328: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:50:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:54 smithi078 ceph-mon[91136]: pgmap v6328: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:50:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:54 smithi078 ceph-mon[99959]: pgmap v6328: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:50:56.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:56 smithi146 ceph-mon[103278]: pgmap v6329: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:50:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:56 smithi078 ceph-mon[91136]: pgmap v6329: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:50:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:56 smithi078 ceph-mon[99959]: pgmap v6329: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-20T23:50:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:58 smithi146 ceph-mon[103278]: pgmap v6330: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:50:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:58 smithi078 ceph-mon[91136]: pgmap v6330: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:50:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:58 smithi078 ceph-mon[99959]: pgmap v6330: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-20T23:50:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:50:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:50:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:50:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:50:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:00.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:00 smithi078 ceph-mon[99959]: pgmap v6331: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T23:51:00.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:00 smithi078 ceph-mon[91136]: pgmap v6331: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T23:51:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:00 smithi146 ceph-mon[103278]: pgmap v6331: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-20T23:51:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:51:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:51:01.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:01.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:01.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:01.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:01.227 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:01.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:02.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:02 smithi078 ceph-mon[91136]: pgmap v6332: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T23:51:02.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:02 smithi078 ceph-mon[99959]: pgmap v6332: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T23:51:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:02 smithi146 ceph-mon[103278]: pgmap v6332: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-20T23:51:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:04 smithi146 ceph-mon[103278]: pgmap v6333: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T23:51:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:04 smithi078 ceph-mon[91136]: pgmap v6333: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T23:51:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:04 smithi078 ceph-mon[99959]: pgmap v6333: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-20T23:51:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:06 smithi146 ceph-mon[103278]: pgmap v6334: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:51:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:06 smithi078 ceph-mon[91136]: pgmap v6334: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:51:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:06 smithi078 ceph-mon[99959]: pgmap v6334: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 77 op/s 2024-09-20T23:51:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:08 smithi146 ceph-mon[103278]: pgmap v6335: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T23:51:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[91136]: pgmap v6335: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T23:51:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[99959]: pgmap v6335: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-20T23:51:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:10 smithi078 ceph-mon[91136]: pgmap v6336: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:51:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:10 smithi078 ceph-mon[99959]: pgmap v6336: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:51:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:10 smithi146 ceph-mon[103278]: pgmap v6336: 161 pgs: 161 active+clean; 585 KiB data, 291 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-20T23:51:11.014 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:51:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:51:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:11.013Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:11.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:11.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:12 smithi146 ceph-mon[103278]: pgmap v6337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T23:51:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[91136]: pgmap v6337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T23:51:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[99959]: pgmap v6337: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-20T23:51:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:13.681 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:13.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:13.681 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:51:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:14.209 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 3h - - 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 3h - - 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 6s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 2s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 6s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 2s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 6s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 6s ago 3h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 2s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 6s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:51:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 6s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 2s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 6s ago 3h 79.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 6s ago 3h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 6s ago 3h 73.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 6s ago 3h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 2s ago 3h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 2s ago 3h 75.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 2s ago 3h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 2s ago 3h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 2s ago 3h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 6s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 6s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:51:14.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 2s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:51:14.522 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:51:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:51:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:51:14.524 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:51:14.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:14 smithi078 ceph-mon[99959]: pgmap v6338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:51:14.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:14 smithi078 ceph-mon[99959]: from='client.78323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:14.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:14 smithi078 ceph-mon[91136]: pgmap v6338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:51:14.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:14 smithi078 ceph-mon[91136]: from='client.78323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:14 smithi146 ceph-mon[103278]: pgmap v6338: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-20T23:51:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:14 smithi146 ceph-mon[103278]: from='client.78323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:51:14.837 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:51:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:51:14.838 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:51:14.838 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:51:15.147 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:51:15.148 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:15.148 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:15.148 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:15.148 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[99959]: from='client.78329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[99959]: from='client.78335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4084914381' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/543817562' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[91136]: from='client.78329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[91136]: from='client.78335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4084914381' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/543817562' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:15 smithi146 ceph-mon[103278]: from='client.78329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:15 smithi146 ceph-mon[103278]: from='client.78335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4084914381' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/543817562' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:16.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:16 smithi146 ceph-mon[103278]: from='client.58431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:16.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:16 smithi146 ceph-mon[103278]: pgmap v6339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T23:51:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:16 smithi078 ceph-mon[99959]: from='client.58431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:16 smithi078 ceph-mon[99959]: pgmap v6339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T23:51:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:16 smithi078 ceph-mon[91136]: from='client.58431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:16 smithi078 ceph-mon[91136]: pgmap v6339: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-20T23:51:18.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:18 smithi146 ceph-mon[103278]: pgmap v6340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T23:51:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:18 smithi078 ceph-mon[99959]: pgmap v6340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T23:51:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:18 smithi078 ceph-mon[91136]: pgmap v6340: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-20T23:51:20.541 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:20 smithi078 ceph-mon[99959]: pgmap v6341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T23:51:20.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:20 smithi078 ceph-mon[91136]: pgmap v6341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T23:51:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:20 smithi146 ceph-mon[103278]: pgmap v6341: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-20T23:51:21.015 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:51:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:51:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:21.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:21.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:21.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:22.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:22 smithi078 ceph-mon[91136]: pgmap v6342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:51:22.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:22 smithi078 ceph-mon[99959]: pgmap v6342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:51:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:22 smithi146 ceph-mon[103278]: pgmap v6342: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-20T23:51:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:24 smithi146 ceph-mon[103278]: pgmap v6343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T23:51:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:24 smithi078 ceph-mon[99959]: pgmap v6343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T23:51:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:24 smithi078 ceph-mon[91136]: pgmap v6343: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-20T23:51:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:26 smithi146 ceph-mon[103278]: pgmap v6344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:51:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:26 smithi078 ceph-mon[99959]: pgmap v6344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:51:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:26 smithi078 ceph-mon[91136]: pgmap v6344: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-20T23:51:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:28 smithi146 ceph-mon[103278]: pgmap v6345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T23:51:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[91136]: pgmap v6345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T23:51:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[99959]: pgmap v6345: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T23:51:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:30 smithi146 ceph-mon[103278]: pgmap v6346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:51:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:30 smithi078 ceph-mon[91136]: pgmap v6346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:51:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:30 smithi078 ceph-mon[99959]: pgmap v6346: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-20T23:51:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:51:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:51:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:31.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:31.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:31.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:32 smithi146 ceph-mon[103278]: pgmap v6347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[99959]: pgmap v6347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[91136]: pgmap v6347: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:34 smithi146 ceph-mon[103278]: pgmap v6348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:34 smithi078 ceph-mon[99959]: pgmap v6348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:34 smithi078 ceph-mon[91136]: pgmap v6348: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:36 smithi146 ceph-mon[103278]: pgmap v6349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:36 smithi078 ceph-mon[91136]: pgmap v6349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:36 smithi078 ceph-mon[99959]: pgmap v6349: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:38 smithi146 ceph-mon[103278]: pgmap v6350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:38 smithi078 ceph-mon[99959]: pgmap v6350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:38 smithi078 ceph-mon[91136]: pgmap v6350: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:40 smithi146 ceph-mon[103278]: pgmap v6351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:40 smithi078 ceph-mon[91136]: pgmap v6351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:51:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:51:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:40 smithi078 ceph-mon[99959]: pgmap v6351: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:41.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:41.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:41.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:42 smithi146 ceph-mon[103278]: pgmap v6352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:42 smithi078 ceph-mon[91136]: pgmap v6352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:42 smithi078 ceph-mon[99959]: pgmap v6352: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:44.780 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:44 smithi078 ceph-mon[99959]: pgmap v6353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:44.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:44 smithi078 ceph-mon[91136]: pgmap v6353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:44 smithi146 ceph-mon[103278]: pgmap v6353: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:45.446 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 3h - - 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 3h - - 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 18s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 13s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 18s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 13s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 18s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 18s ago 3h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:51:45.987 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 13s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 18s ago 3h 252M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 18s ago 3h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 13s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 18s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 18s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 3h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 13s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 13s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 13s ago 3h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 13s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 13s ago 3h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 13s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:51:45.988 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 13s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:51:46.297 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:51:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:51:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:51:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:51:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:51:46.299 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:51:46.565 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:51:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:51:46.566 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:51:46.877 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:51:46.877 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:51:46.877 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[99959]: pgmap v6354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[99959]: from='client.58443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[99959]: from='client.58449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2699416093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[91136]: pgmap v6354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[91136]: from='client.58443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[91136]: from='client.58449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2699416093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:46 smithi146 ceph-mon[103278]: pgmap v6354: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:46 smithi146 ceph-mon[103278]: from='client.58443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:46 smithi146 ceph-mon[103278]: from='client.58449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2699416093' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:47 smithi146 ceph-mon[103278]: from='client.58455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:47 smithi146 ceph-mon[103278]: from='client.78383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2438614122' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[91136]: from='client.58455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[91136]: from='client.78383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2438614122' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[99959]: from='client.58455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[99959]: from='client.78383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2438614122' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:48 smithi146 ceph-mon[103278]: pgmap v6355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:48 smithi078 ceph-mon[91136]: pgmap v6355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:48 smithi078 ceph-mon[99959]: pgmap v6355: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:50 smithi146 ceph-mon[103278]: pgmap v6356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:51.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:50 smithi078 ceph-mon[91136]: pgmap v6356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:51.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:50 smithi078 ceph-mon[99959]: pgmap v6356: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:51.017 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:51:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:51:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:51.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:51.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:51:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:51:51.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:51:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:51:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:52 smithi146 ceph-mon[103278]: pgmap v6357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[91136]: pgmap v6357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[99959]: pgmap v6357: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:51:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:54 smithi146 ceph-mon[103278]: pgmap v6358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:54 smithi078 ceph-mon[91136]: pgmap v6358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:54 smithi078 ceph-mon[99959]: pgmap v6358: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:51:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:56 smithi146 ceph-mon[103278]: pgmap v6359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:56 smithi078 ceph-mon[91136]: pgmap v6359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:56 smithi078 ceph-mon[99959]: pgmap v6359: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:58 smithi146 ceph-mon[103278]: pgmap v6360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:58 smithi078 ceph-mon[91136]: pgmap v6360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:51:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:58 smithi078 ceph-mon[99959]: pgmap v6360: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:51:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:00 smithi146 ceph-mon[103278]: pgmap v6361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:01.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:00 smithi078 ceph-mon[99959]: pgmap v6361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:01.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:00 smithi078 ceph-mon[91136]: pgmap v6361: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:01.018 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:52:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:52:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:01.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:01.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:01.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:02.974 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:02 smithi078 ceph-mon[99959]: pgmap v6362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:02.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:02 smithi078 ceph-mon[91136]: pgmap v6362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:02 smithi146 ceph-mon[103278]: pgmap v6362: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:04 smithi078 ceph-mon[91136]: pgmap v6363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:04 smithi078 ceph-mon[99959]: pgmap v6363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:04 smithi146 ceph-mon[103278]: pgmap v6363: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:06 smithi146 ceph-mon[103278]: pgmap v6364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:06 smithi078 ceph-mon[91136]: pgmap v6364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:06 smithi078 ceph-mon[99959]: pgmap v6364: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:08 smithi146 ceph-mon[103278]: pgmap v6365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[91136]: pgmap v6365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[99959]: pgmap v6365: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:10 smithi146 ceph-mon[103278]: pgmap v6366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:11.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:10 smithi078 ceph-mon[99959]: pgmap v6366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:11.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:10 smithi078 ceph-mon[91136]: pgmap v6366: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:11.019 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:52:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:52:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:11.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:11.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:11.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:12 smithi146 ceph-mon[103278]: pgmap v6367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[91136]: pgmap v6367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[99959]: pgmap v6367: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:14 smithi146 ceph-mon[103278]: pgmap v6368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:14 smithi078 ceph-mon[91136]: pgmap v6368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:14 smithi078 ceph-mon[99959]: pgmap v6368: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:16 smithi146 ceph-mon[103278]: pgmap v6369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:16 smithi078 ceph-mon[99959]: pgmap v6369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:16 smithi078 ceph-mon[91136]: pgmap v6369: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:17.171 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:52:17.696 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:52:17.696 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 3h - - 2024-09-20T23:52:17.696 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 3h - - 2024-09-20T23:52:17.696 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 9s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:52:17.696 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 5s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:52:17.696 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 9s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 5s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 9s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 9s ago 3h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 5s ago 3h 244M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 9s ago 3h 252M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 9s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 5s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 9s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 9s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 9s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 9s ago 3h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 5s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 5s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 5s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:52:17.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 5s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:52:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 5s ago 3h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:52:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 9s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:52:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 5s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:52:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 9s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:52:17.698 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 5s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:18.010 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:52:18.011 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:52:18.276 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:52:18.276 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:52:18.276 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:52:18.276 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:52:18.277 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:52:18.586 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:52:18.586 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:18.586 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:18.586 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:18 smithi146 ceph-mon[103278]: pgmap v6370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:18 smithi146 ceph-mon[103278]: from='client.58479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:18 smithi146 ceph-mon[103278]: from='client.78401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:18 smithi146 ceph-mon[103278]: from='client.58491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3074572403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/262272721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[91136]: pgmap v6370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[91136]: from='client.58479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[91136]: from='client.78401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[91136]: from='client.58491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3074572403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/262272721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[99959]: pgmap v6370: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[99959]: from='client.58479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[99959]: from='client.78401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[99959]: from='client.58491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3074572403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/262272721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:19 smithi146 ceph-mon[103278]: from='client.78419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:19 smithi078 ceph-mon[99959]: from='client.78419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:19 smithi078 ceph-mon[91136]: from='client.78419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:20 smithi146 ceph-mon[103278]: pgmap v6371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:21.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:20 smithi078 ceph-mon[99959]: pgmap v6371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:21.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:20 smithi078 ceph-mon[91136]: pgmap v6371: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:21.019 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:52:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:52:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:21.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:21.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:21.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:22 smithi078 ceph-mon[99959]: pgmap v6372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:22 smithi078 ceph-mon[91136]: pgmap v6372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:22 smithi146 ceph-mon[103278]: pgmap v6372: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:24.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:24 smithi078 ceph-mon[91136]: pgmap v6373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:24.838 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:24 smithi078 ceph-mon[99959]: pgmap v6373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:24 smithi146 ceph-mon[103278]: pgmap v6373: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:26 smithi078 ceph-mon[91136]: pgmap v6374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:26 smithi078 ceph-mon[99959]: pgmap v6374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:26 smithi146 ceph-mon[103278]: pgmap v6374: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[91136]: pgmap v6375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[99959]: pgmap v6375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:29.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:28 smithi146 ceph-mon[103278]: pgmap v6375: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:29.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:29.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:31.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:30 smithi078 ceph-mon[99959]: pgmap v6376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:31.020 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:52:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:52:31.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:30 smithi078 ceph-mon[91136]: pgmap v6376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:30 smithi146 ceph-mon[103278]: pgmap v6376: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:31.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:31.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:31.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:32.075 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:32 smithi146 ceph-mon[103278]: pgmap v6377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[91136]: pgmap v6377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[99959]: pgmap v6377: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:33 smithi146 ceph-mon[103278]: pgmap v6378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:33 smithi078 ceph-mon[91136]: pgmap v6378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:33 smithi078 ceph-mon[99959]: pgmap v6378: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:36 smithi146 ceph-mon[103278]: pgmap v6379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:36 smithi078 ceph-mon[91136]: pgmap v6379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:36 smithi078 ceph-mon[99959]: pgmap v6379: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:38 smithi146 ceph-mon[103278]: pgmap v6380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:38 smithi078 ceph-mon[91136]: pgmap v6380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:38 smithi078 ceph-mon[99959]: pgmap v6380: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:40.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:40 smithi146 ceph-mon[103278]: pgmap v6381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:40 smithi078 ceph-mon[99959]: pgmap v6381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:40 smithi078 ceph-mon[91136]: pgmap v6381: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:41.020 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:52:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:52:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:41.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:41.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:41.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:42 smithi146 ceph-mon[103278]: pgmap v6382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:42 smithi078 ceph-mon[91136]: pgmap v6382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:42 smithi078 ceph-mon[99959]: pgmap v6382: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:44 smithi146 ceph-mon[103278]: pgmap v6383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:44 smithi078 ceph-mon[91136]: pgmap v6383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:44 smithi078 ceph-mon[99959]: pgmap v6383: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:46 smithi146 ceph-mon[103278]: pgmap v6384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:46 smithi078 ceph-mon[91136]: pgmap v6384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:46 smithi078 ceph-mon[99959]: pgmap v6384: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[91136]: pgmap v6385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[99959]: pgmap v6385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:48 smithi146 ceph-mon[103278]: pgmap v6385: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:48.891 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:52:49.419 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:52:49.419 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 3h - - 2024-09-20T23:52:49.419 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 3h - - 2024-09-20T23:52:49.419 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 1s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:52:49.419 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 16s ago 3h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 1s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 16s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 1s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 1s ago 3h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 16s ago 3h 245M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 1s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 1s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 16s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 1s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 1s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 1s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 1s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 16s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 16s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 16s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 16s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 16s ago 3h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 16s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 16s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:52:49.726 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:52:49.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:52:49.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:52:49.727 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:52:49.994 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:52:49.995 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:52:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[91136]: from='client.78431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[91136]: from='client.78437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[91136]: pgmap v6386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[91136]: from='client.78443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/316897908' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[99959]: from='client.78431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[99959]: from='client.78437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[99959]: pgmap v6386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[99959]: from='client.78443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/316897908' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:50.306 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:52:50.306 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:52:50.306 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:52:50.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:50 smithi146 ceph-mon[103278]: from='client.78431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:50 smithi146 ceph-mon[103278]: from='client.78437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:50 smithi146 ceph-mon[103278]: pgmap v6386: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:50.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:50 smithi146 ceph-mon[103278]: from='client.78443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:50.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/316897908' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:52:51.021 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:52:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:52:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-mon[99959]: from='client.58539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/131540848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-mon[91136]: from='client.58539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/131540848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:51.020Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:51.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:52:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:52:51.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:52:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:52:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:51 smithi146 ceph-mon[103278]: from='client.58539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/131540848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:52:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:52 smithi146 ceph-mon[103278]: pgmap v6387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:52 smithi078 ceph-mon[91136]: pgmap v6387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:52 smithi078 ceph-mon[99959]: pgmap v6387: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:52:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:54 smithi146 ceph-mon[103278]: pgmap v6388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:54 smithi078 ceph-mon[99959]: pgmap v6388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:54 smithi078 ceph-mon[91136]: pgmap v6388: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:52:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:56 smithi146 ceph-mon[103278]: pgmap v6389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:56 smithi078 ceph-mon[99959]: pgmap v6389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:56 smithi078 ceph-mon[91136]: pgmap v6389: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:58 smithi146 ceph-mon[103278]: pgmap v6390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:58 smithi078 ceph-mon[91136]: pgmap v6390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:52:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:58 smithi078 ceph-mon[99959]: pgmap v6390: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:52:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:00 smithi146 ceph-mon[103278]: pgmap v6391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:01.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:00 smithi078 ceph-mon[99959]: pgmap v6391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:01.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:00 smithi078 ceph-mon[91136]: pgmap v6391: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:01.023 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:53:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:53:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:01.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:01.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:01.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:01.692 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:01.974 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:01.975 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:02 smithi078 ceph-mon[91136]: pgmap v6392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:02 smithi078 ceph-mon[99959]: pgmap v6392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:02 smithi146 ceph-mon[103278]: pgmap v6392: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:04.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:04 smithi078 ceph-mon[99959]: pgmap v6393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:04.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:04 smithi078 ceph-mon[91136]: pgmap v6393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:04 smithi146 ceph-mon[103278]: pgmap v6393: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:06.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:06 smithi078 ceph-mon[91136]: pgmap v6394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:06.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:06 smithi078 ceph-mon[99959]: pgmap v6394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:06 smithi146 ceph-mon[103278]: pgmap v6394: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:08 smithi146 ceph-mon[103278]: pgmap v6395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[99959]: pgmap v6395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[91136]: pgmap v6395: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:10 smithi146 ceph-mon[103278]: pgmap v6396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:11.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:10 smithi078 ceph-mon[99959]: pgmap v6396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:11.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:10 smithi078 ceph-mon[91136]: pgmap v6396: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:11.024 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:53:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:53:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:11.022Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:11.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:11.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:12 smithi146 ceph-mon[103278]: pgmap v6397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:12 smithi078 ceph-mon[91136]: pgmap v6397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:12 smithi078 ceph-mon[99959]: pgmap v6397: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:14 smithi146 ceph-mon[103278]: pgmap v6398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:14 smithi078 ceph-mon[91136]: pgmap v6398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:14 smithi078 ceph-mon[99959]: pgmap v6398: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:16 smithi146 ceph-mon[103278]: pgmap v6399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:16 smithi078 ceph-mon[91136]: pgmap v6399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:16 smithi078 ceph-mon[99959]: pgmap v6399: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:18 smithi146 ceph-mon[103278]: pgmap v6400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:18 smithi078 ceph-mon[99959]: pgmap v6400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:18 smithi078 ceph-mon[91136]: pgmap v6400: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:20.600 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:53:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:20 smithi146 ceph-mon[103278]: pgmap v6401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:21.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:20 smithi078 ceph-mon[99959]: pgmap v6401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:21.024 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:53:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:53:21.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:20 smithi078 ceph-mon[91136]: pgmap v6401: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 3h - - 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 3h - - 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 8s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 8s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:53:21.129 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 3h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 8s ago 3h 246M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 8s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 13s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 13s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 13s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 8s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 8s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 8s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 8s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:53:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 8s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:53:21.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:53:21.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 8s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:53:21.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:53:21.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 8s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:53:21.278 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:21.023Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:21.278 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:21.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:21.278 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:21.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:53:21.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:53:21.468 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:53:21.745 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:53:21.746 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:53:21.746 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:53:21.746 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:53:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:21 smithi146 ceph-mon[103278]: from='client.78467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2189722700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-mon[99959]: from='client.78467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2189722700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-mon[91136]: from='client.78467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2189722700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:22.063 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:53:22.063 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:22.063 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:22.063 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:22 smithi146 ceph-mon[103278]: from='client.78473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:22 smithi146 ceph-mon[103278]: from='client.58563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:22 smithi146 ceph-mon[103278]: pgmap v6402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:22 smithi146 ceph-mon[103278]: from='client.58575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1814070137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[91136]: from='client.78473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[91136]: from='client.58563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[91136]: pgmap v6402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[91136]: from='client.58575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1814070137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[99959]: from='client.78473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[99959]: from='client.58563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[99959]: pgmap v6402: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[99959]: from='client.58575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1814070137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:24 smithi146 ceph-mon[103278]: pgmap v6403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:25.049 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:24 smithi078 ceph-mon[99959]: pgmap v6403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:25.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:24 smithi078 ceph-mon[91136]: pgmap v6403: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:26 smithi146 ceph-mon[103278]: pgmap v6404: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:26 smithi078 ceph-mon[99959]: pgmap v6404: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:26 smithi078 ceph-mon[91136]: pgmap v6404: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:28 smithi146 ceph-mon[103278]: pgmap v6405: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[91136]: pgmap v6405: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[99959]: pgmap v6405: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:31.025 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:30 smithi078 ceph-mon[91136]: pgmap v6406: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:31.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:30 smithi078 ceph-mon[99959]: pgmap v6406: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:31.026 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:53:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:53:31.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:30 smithi146 ceph-mon[103278]: pgmap v6406: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:31.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:31.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:31.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:32 smithi078 ceph-mon[91136]: pgmap v6407: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:32 smithi078 ceph-mon[99959]: pgmap v6407: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:32 smithi146 ceph-mon[103278]: pgmap v6407: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:33 smithi146 ceph-mon[103278]: pgmap v6408: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[91136]: pgmap v6408: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:33 smithi078 ceph-mon[99959]: pgmap v6408: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:36 smithi146 ceph-mon[103278]: pgmap v6409: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:36 smithi078 ceph-mon[91136]: pgmap v6409: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:36 smithi078 ceph-mon[99959]: pgmap v6409: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:38 smithi146 ceph-mon[103278]: pgmap v6410: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:38 smithi078 ceph-mon[91136]: pgmap v6410: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:38 smithi078 ceph-mon[99959]: pgmap v6410: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:40 smithi146 ceph-mon[103278]: pgmap v6411: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:40 smithi078 ceph-mon[91136]: pgmap v6411: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:40 smithi078 ceph-mon[99959]: pgmap v6411: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:41.026 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:53:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:53:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:41.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:41.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:41.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:41.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:42.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:42 smithi146 ceph-mon[103278]: pgmap v6412: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:42 smithi078 ceph-mon[99959]: pgmap v6412: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:42 smithi078 ceph-mon[91136]: pgmap v6412: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:44 smithi146 ceph-mon[103278]: pgmap v6413: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:44 smithi078 ceph-mon[91136]: pgmap v6413: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:44 smithi078 ceph-mon[99959]: pgmap v6413: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:46.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:46 smithi146 ceph-mon[103278]: pgmap v6414: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:46 smithi078 ceph-mon[91136]: pgmap v6414: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:46 smithi078 ceph-mon[99959]: pgmap v6414: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:48.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:48 smithi078 ceph-mon[91136]: pgmap v6415: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:48.370 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:48 smithi078 ceph-mon[99959]: pgmap v6415: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:48 smithi146 ceph-mon[103278]: pgmap v6415: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:50 smithi146 ceph-mon[103278]: pgmap v6416: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:50 smithi078 ceph-mon[91136]: pgmap v6416: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:50 smithi078 ceph-mon[99959]: pgmap v6416: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:53:50] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-20T23:53:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:51.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:51.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:53:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:53:51.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:53:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:53:52.355 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:53:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:52 smithi146 ceph-mon[103278]: pgmap v6417: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:52 smithi078 ceph-mon[99959]: pgmap v6417: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:52 smithi078 ceph-mon[91136]: pgmap v6417: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:52.881 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 3h - - 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 3h - - 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 4s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 20s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 4s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 20s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 4s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 4s ago 3h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 20s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 4s ago 3h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 4s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:53:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 20s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 4s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 4s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 4s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 4s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 20s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 20s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 20s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 20s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 20s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 20s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:53:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 20s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:53:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:53:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:53:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:53:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:53:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:53:53.198 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:53:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:53:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:53:53.468 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[99959]: from='client.78503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[99959]: from='client.58593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4021773434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:53.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[91136]: from='client.78503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:53.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[91136]: from='client.58593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:53.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4021773434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:53 smithi146 ceph-mon[103278]: from='client.78503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:53 smithi146 ceph-mon[103278]: from='client.58593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:53:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4021773434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:53:53.774 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:53:53.775 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:53:53.775 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:53:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:54 smithi146 ceph-mon[103278]: from='client.78515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:54 smithi146 ceph-mon[103278]: pgmap v6418: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:54 smithi146 ceph-mon[103278]: from='client.78527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3308747377' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[99959]: from='client.78515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[99959]: pgmap v6418: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[99959]: from='client.78527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3308747377' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[91136]: from='client.78515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[91136]: pgmap v6418: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:53:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[91136]: from='client.78527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3308747377' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:53:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:56 smithi146 ceph-mon[103278]: pgmap v6419: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:56 smithi078 ceph-mon[91136]: pgmap v6419: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:56 smithi078 ceph-mon[99959]: pgmap v6419: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:58 smithi146 ceph-mon[103278]: pgmap v6420: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:53:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:58 smithi078 ceph-mon[99959]: pgmap v6420: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:53:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:53:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:58 smithi078 ceph-mon[91136]: pgmap v6420: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:53:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:53:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:00 smithi146 ceph-mon[103278]: pgmap v6421: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:00 smithi078 ceph-mon[91136]: pgmap v6421: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:00 smithi078 ceph-mon[99959]: pgmap v6421: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:54:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:54:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:01.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:01.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:01.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:02.601 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:02 smithi146 ceph-mon[103278]: pgmap v6422: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:02 smithi078 ceph-mon[91136]: pgmap v6422: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:02 smithi078 ceph-mon[99959]: pgmap v6422: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:04 smithi146 ceph-mon[103278]: pgmap v6423: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:04 smithi078 ceph-mon[91136]: pgmap v6423: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:04 smithi078 ceph-mon[99959]: pgmap v6423: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:06 smithi146 ceph-mon[103278]: pgmap v6424: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:06 smithi078 ceph-mon[91136]: pgmap v6424: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:06 smithi078 ceph-mon[99959]: pgmap v6424: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:08.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:08 smithi078 ceph-mon[91136]: pgmap v6425: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:08.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:08 smithi078 ceph-mon[99959]: pgmap v6425: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:08 smithi146 ceph-mon[103278]: pgmap v6425: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:10 smithi146 ceph-mon[103278]: pgmap v6426: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:10 smithi078 ceph-mon[91136]: pgmap v6426: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:54:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:54:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:10 smithi078 ceph-mon[99959]: pgmap v6426: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:11.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:11.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:11.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:12 smithi146 ceph-mon[103278]: pgmap v6427: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:12 smithi078 ceph-mon[99959]: pgmap v6427: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:12 smithi078 ceph-mon[91136]: pgmap v6427: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:14 smithi146 ceph-mon[103278]: pgmap v6428: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[99959]: pgmap v6428: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[91136]: pgmap v6428: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:16 smithi146 ceph-mon[103278]: pgmap v6429: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:16 smithi078 ceph-mon[99959]: pgmap v6429: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:16 smithi078 ceph-mon[91136]: pgmap v6429: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:18 smithi146 ceph-mon[103278]: pgmap v6430: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:18 smithi078 ceph-mon[91136]: pgmap v6430: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:18 smithi078 ceph-mon[99959]: pgmap v6430: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:20 smithi078 ceph-mon[91136]: pgmap v6431: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:20 smithi078 ceph-mon[99959]: pgmap v6431: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:20 smithi146 ceph-mon[103278]: pgmap v6431: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:21.030 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:54:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:54:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:21.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:21.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:21.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:22.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:22 smithi146 ceph-mon[103278]: pgmap v6432: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:22 smithi078 ceph-mon[91136]: pgmap v6432: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:22 smithi078 ceph-mon[99959]: pgmap v6432: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:24.058 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:54:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:24 smithi146 ceph-mon[103278]: pgmap v6433: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:24 smithi078 ceph-mon[91136]: pgmap v6433: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:24 smithi078 ceph-mon[99959]: pgmap v6433: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 3h - - 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 3h - - 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 16s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 11s ago 3h 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 11s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 3h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:54:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 11s ago 3h 247M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 16s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 16s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 11s ago 3h 14.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 16s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 16s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 16s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 16s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 11s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 11s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 11s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 11s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 11s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 16s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 11s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:54:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 16s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:54:24.585 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 11s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:54:24.892 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:54:24.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:54:24.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:54:24.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:54:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:54:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:54:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:54:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:54:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:54:25.161 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:54:25.472 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:54:25.472 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:25.472 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:25.472 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:25.472 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:25.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[99959]: from='client.58623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[99959]: from='client.78545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[99959]: from='client.58635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1097851904' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:25.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[91136]: from='client.58623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[91136]: from='client.78545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[91136]: from='client.58635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1097851904' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:25 smithi146 ceph-mon[103278]: from='client.58623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:25 smithi146 ceph-mon[103278]: from='client.78545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:25 smithi146 ceph-mon[103278]: from='client.58635 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1097851904' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:26.259 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 20 23:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-20T23:54:25.849119577Z level=info msg="Completed cleanup jobs" duration=13.141952ms 2024-09-20T23:54:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:26 smithi146 ceph-mon[103278]: from='client.78563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:26 smithi146 ceph-mon[103278]: pgmap v6434: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3951910883' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:26 smithi078 ceph-mon[91136]: from='client.78563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:26 smithi078 ceph-mon[91136]: pgmap v6434: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3951910883' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:26 smithi078 ceph-mon[99959]: from='client.78563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:26 smithi078 ceph-mon[99959]: pgmap v6434: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:26.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3951910883' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:28 smithi146 ceph-mon[103278]: pgmap v6435: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:28 smithi078 ceph-mon[91136]: pgmap v6435: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:28 smithi078 ceph-mon[99959]: pgmap v6435: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:30 smithi146 ceph-mon[103278]: pgmap v6436: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:31.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:30 smithi078 ceph-mon[91136]: pgmap v6436: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:31.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:30 smithi078 ceph-mon[99959]: pgmap v6436: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:31.031 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:54:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:54:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:31.029Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:31.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:31.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:32 smithi146 ceph-mon[103278]: pgmap v6437: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:32 smithi078 ceph-mon[91136]: pgmap v6437: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:32 smithi078 ceph-mon[99959]: pgmap v6437: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:34 smithi146 ceph-mon[103278]: pgmap v6438: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[99959]: pgmap v6438: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[91136]: pgmap v6438: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:36 smithi146 ceph-mon[103278]: pgmap v6439: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:36 smithi078 ceph-mon[91136]: pgmap v6439: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:36 smithi078 ceph-mon[99959]: pgmap v6439: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:38 smithi146 ceph-mon[103278]: pgmap v6440: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:38 smithi078 ceph-mon[99959]: pgmap v6440: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:38 smithi078 ceph-mon[91136]: pgmap v6440: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:40 smithi078 ceph-mon[91136]: pgmap v6441: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:40 smithi078 ceph-mon[99959]: pgmap v6441: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:40 smithi146 ceph-mon[103278]: pgmap v6441: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:41.031 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:54:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:54:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:41.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:41.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:41.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:42 smithi146 ceph-mon[103278]: pgmap v6442: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:42 smithi078 ceph-mon[91136]: pgmap v6442: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:42 smithi078 ceph-mon[99959]: pgmap v6442: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:44 smithi146 ceph-mon[103278]: pgmap v6443: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:44 smithi078 ceph-mon[91136]: pgmap v6443: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:44 smithi078 ceph-mon[99959]: pgmap v6443: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:46 smithi146 ceph-mon[103278]: pgmap v6444: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:46 smithi078 ceph-mon[91136]: pgmap v6444: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:46 smithi078 ceph-mon[99959]: pgmap v6444: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:48.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:48 smithi078 ceph-mon[91136]: pgmap v6445: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:48.303 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:48 smithi078 ceph-mon[99959]: pgmap v6445: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:48 smithi146 ceph-mon[103278]: pgmap v6445: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:50.842 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:50 smithi078 ceph-mon[99959]: pgmap v6446: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:50.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:50 smithi078 ceph-mon[91136]: pgmap v6446: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:50.843 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:54:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:54:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:50 smithi146 ceph-mon[103278]: pgmap v6446: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:51.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:51.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:54:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:54:51.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:54:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:52.110 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:54:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:52 smithi078 ceph-mon[91136]: pgmap v6447: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:52 smithi078 ceph-mon[99959]: pgmap v6447: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:52 smithi146 ceph-mon[103278]: pgmap v6447: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:54 smithi146 ceph-mon[103278]: pgmap v6448: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[91136]: pgmap v6448: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[99959]: pgmap v6448: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:54:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:54:55.767 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 3h - - 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 3h - - 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 7s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 2s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 7s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 2s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 7s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 7s ago 3h 384M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 2s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 7s ago 3h 255M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:54:56.292 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 7s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 2s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 7s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 7s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 7s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 7s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 2s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 2s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 2s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 2s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 2s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 2s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:54:56.293 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 2s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:56.600 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:54:56.601 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:54:56.601 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:56 smithi078 ceph-mon[99959]: pgmap v6449: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:56.601 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:56 smithi078 ceph-mon[99959]: from='client.78575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:56.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:56 smithi078 ceph-mon[91136]: pgmap v6449: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:56.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:56 smithi078 ceph-mon[91136]: from='client.78575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:56 smithi146 ceph-mon[103278]: pgmap v6449: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:56 smithi146 ceph-mon[103278]: from='client.78575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:54:56.867 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:54:56.868 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:54:56.868 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:54:57.172 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:54:57.172 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:54:57.172 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:54:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[99959]: from='client.58665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[99959]: from='client.58671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2582655283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/94026483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[91136]: from='client.58665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[91136]: from='client.58671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2582655283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/94026483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:57 smithi146 ceph-mon[103278]: from='client.58665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:57 smithi146 ceph-mon[103278]: from='client.58671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2582655283' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:54:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/94026483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:54:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:58 smithi146 ceph-mon[103278]: from='client.78599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:58 smithi146 ceph-mon[103278]: pgmap v6450: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:54:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:58 smithi078 ceph-mon[91136]: from='client.78599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:58 smithi078 ceph-mon[91136]: pgmap v6450: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:54:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:54:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:58 smithi078 ceph-mon[99959]: from='client.78599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:58 smithi078 ceph-mon[99959]: pgmap v6450: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:54:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:54:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:00 smithi146 ceph-mon[103278]: pgmap v6451: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:00 smithi078 ceph-mon[91136]: pgmap v6451: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:00 smithi078 ceph-mon[99959]: pgmap v6451: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:55:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:55:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:01.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:01.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:01.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:01.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:02 smithi146 ceph-mon[103278]: pgmap v6452: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:02 smithi078 ceph-mon[91136]: pgmap v6452: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:02 smithi078 ceph-mon[99959]: pgmap v6452: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:04 smithi146 ceph-mon[103278]: pgmap v6453: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:04 smithi078 ceph-mon[91136]: pgmap v6453: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:04 smithi078 ceph-mon[99959]: pgmap v6453: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:06 smithi146 ceph-mon[103278]: pgmap v6454: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:06 smithi078 ceph-mon[99959]: pgmap v6454: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:06 smithi078 ceph-mon[91136]: pgmap v6454: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:08 smithi146 ceph-mon[103278]: pgmap v6455: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:09.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:08 smithi078 ceph-mon[91136]: pgmap v6455: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:09.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:08 smithi078 ceph-mon[99959]: pgmap v6455: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:10.240 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[99959]: pgmap v6456: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.241 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:09 smithi078 ceph-mon[91136]: pgmap v6456: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:09 smithi146 ceph-mon[103278]: pgmap v6456: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:55:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:55:11.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:11.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:11.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:11.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:11.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:11.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:11.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:11.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:11 smithi146 ceph-mon[103278]: pgmap v6457: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:11 smithi078 ceph-mon[91136]: pgmap v6457: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:11 smithi078 ceph-mon[99959]: pgmap v6457: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:14 smithi146 ceph-mon[103278]: pgmap v6458: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[91136]: pgmap v6458: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[99959]: pgmap v6458: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:16 smithi146 ceph-mon[103278]: pgmap v6459: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:16 smithi078 ceph-mon[91136]: pgmap v6459: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:16 smithi078 ceph-mon[99959]: pgmap v6459: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:18 smithi146 ceph-mon[103278]: pgmap v6460: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:18 smithi078 ceph-mon[91136]: pgmap v6460: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:18 smithi078 ceph-mon[99959]: pgmap v6460: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:20 smithi078 ceph-mon[91136]: pgmap v6461: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:20 smithi078 ceph-mon[99959]: pgmap v6461: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:20 smithi146 ceph-mon[103278]: pgmap v6461: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:55:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:55:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:21.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:21.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:21.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:22 smithi078 ceph-mon[91136]: pgmap v6462: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:22 smithi078 ceph-mon[99959]: pgmap v6462: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:22 smithi146 ceph-mon[103278]: pgmap v6462: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:24 smithi146 ceph-mon[103278]: pgmap v6463: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:24 smithi078 ceph-mon[91136]: pgmap v6463: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:24 smithi078 ceph-mon[99959]: pgmap v6463: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:26 smithi078 ceph-mon[91136]: pgmap v6464: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:26 smithi078 ceph-mon[99959]: pgmap v6464: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:26 smithi146 ceph-mon[103278]: pgmap v6464: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:27.469 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 3h - - 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 3h - - 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 19s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 14s ago 3h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 19s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:55:27.992 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 14s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 19s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 19s ago 3h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 14s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 19s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 19s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 14s ago 3h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 19s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 19s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 19s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 19s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 14s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 14s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 14s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:55:27.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:55:27.994 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 14s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:55:27.994 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 19s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:55:27.994 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:55:27.994 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 19s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:55:27.994 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:55:28.306 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:55:28.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:55:28.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:55:28.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:55:28.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:55:28.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:55:28.307 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:55:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:28 smithi078 ceph-mon[91136]: pgmap v6465: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:28 smithi078 ceph-mon[91136]: from='client.78611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:28 smithi078 ceph-mon[91136]: from='client.78617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:28 smithi078 ceph-mon[99959]: pgmap v6465: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:28 smithi078 ceph-mon[99959]: from='client.78611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:28 smithi078 ceph-mon[99959]: from='client.78617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:55:28.577 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:55:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:55:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:55:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:55:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:55:28.578 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:55:28.578 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:55:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:28 smithi146 ceph-mon[103278]: pgmap v6465: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:28 smithi146 ceph-mon[103278]: from='client.78611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:28 smithi146 ceph-mon[103278]: from='client.78617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.888 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:55:28.888 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:55:28.888 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:55:28.888 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:55:28.888 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:55:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='client.78623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1553149653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='client.78635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1175559914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='client.78623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1553149653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='client.78635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1175559914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='client.78623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1553149653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:55:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='client.78635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1175559914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:55:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:30.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:30 smithi078 ceph-mon[99959]: pgmap v6466: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:30 smithi078 ceph-mon[91136]: pgmap v6466: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:30 smithi146 ceph-mon[103278]: pgmap v6466: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:30.865 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:55:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:55:31.167 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:31.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:31.167 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:31.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:31.167 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:31.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:31.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:31.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:31.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:32 smithi146 ceph-mon[103278]: pgmap v6467: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:32 smithi078 ceph-mon[91136]: pgmap v6467: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:32 smithi078 ceph-mon[99959]: pgmap v6467: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:34 smithi146 ceph-mon[103278]: pgmap v6468: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[99959]: pgmap v6468: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[91136]: pgmap v6468: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:36 smithi146 ceph-mon[103278]: pgmap v6469: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:36 smithi078 ceph-mon[91136]: pgmap v6469: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:36 smithi078 ceph-mon[99959]: pgmap v6469: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:38 smithi146 ceph-mon[103278]: pgmap v6470: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:38 smithi078 ceph-mon[91136]: pgmap v6470: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:38 smithi078 ceph-mon[99959]: pgmap v6470: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:40 smithi146 ceph-mon[103278]: pgmap v6471: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:40 smithi078 ceph-mon[99959]: pgmap v6471: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:40 smithi078 ceph-mon[91136]: pgmap v6471: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:55:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:55:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:41.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:41.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:41.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:42.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:42 smithi146 ceph-mon[103278]: pgmap v6472: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:42 smithi078 ceph-mon[91136]: pgmap v6472: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:42 smithi078 ceph-mon[99959]: pgmap v6472: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:43.861 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:44 smithi078 ceph-mon[91136]: pgmap v6473: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:44 smithi078 ceph-mon[99959]: pgmap v6473: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:44.867 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:44 smithi146 ceph-mon[103278]: pgmap v6473: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:46 smithi146 ceph-mon[103278]: pgmap v6474: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:46 smithi078 ceph-mon[91136]: pgmap v6474: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:46 smithi078 ceph-mon[99959]: pgmap v6474: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:48 smithi146 ceph-mon[103278]: pgmap v6475: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:48 smithi078 ceph-mon[91136]: pgmap v6475: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:48 smithi078 ceph-mon[99959]: pgmap v6475: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:50.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[99959]: pgmap v6476: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.347 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-mon[91136]: pgmap v6476: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:50 smithi146 ceph-mon[103278]: pgmap v6476: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:55:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-20T23:55:51.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:51.346 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:51.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:51.346 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:51.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:55:51.346 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:55:51.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:55:51.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:55:52.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:52 smithi078 ceph-mon[91136]: pgmap v6477: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:52.239 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:52 smithi078 ceph-mon[99959]: pgmap v6477: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:52 smithi146 ceph-mon[103278]: pgmap v6477: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:54 smithi146 ceph-mon[103278]: pgmap v6478: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[91136]: pgmap v6478: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[99959]: pgmap v6478: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:55:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:56 smithi146 ceph-mon[103278]: pgmap v6479: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:56 smithi078 ceph-mon[91136]: pgmap v6479: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:56 smithi078 ceph-mon[99959]: pgmap v6479: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:58 smithi078 ceph-mon[99959]: pgmap v6480: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:55:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:58 smithi078 ceph-mon[91136]: pgmap v6480: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:55:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:59.184 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:55:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:58 smithi146 ceph-mon[103278]: pgmap v6480: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:55:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:55:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:55:59.707 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:55:59.707 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 3h - - 2024-09-20T23:55:59.707 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 3h - - 2024-09-20T23:55:59.707 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 10s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:55:59.707 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 5s ago 3h 88.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:55:59.707 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 10s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 5s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 10s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 10s ago 3h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 5s ago 3h 248M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 10s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 10s ago 3h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 5s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 10s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 10s ago 3h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 10s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 10s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 5s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 5s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 5s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:55:59.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 5s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:55:59.709 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 5s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:55:59.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 10s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:55:59.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 5s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:55:59.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 10s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:55:59.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 5s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:56:00.018 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:56:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:56:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:56:00.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:56:00.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:56:00.287 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:56:00.287 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:56:00.287 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:56:00.287 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:56:00.287 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:56:00.288 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:56:00.595 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:56:00.595 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:00.596 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:00.596 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:00.596 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:56:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[91136]: from='client.58731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[91136]: pgmap v6481: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[91136]: from='client.58737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[91136]: from='client.58743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1359194721' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2068179574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[99959]: from='client.58731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[99959]: pgmap v6481: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[99959]: from='client.58737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[99959]: from='client.58743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1359194721' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2068179574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:00 smithi146 ceph-mon[103278]: from='client.58731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:00 smithi146 ceph-mon[103278]: pgmap v6481: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:01.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:00 smithi146 ceph-mon[103278]: from='client.58737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:00 smithi146 ceph-mon[103278]: from='client.58743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1359194721' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:01.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2068179574' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:01.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:01.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:01.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-mon[91136]: from='client.58755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:01 smithi078 ceph-mon[99959]: from='client.58755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:02.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:02.084 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:01 smithi146 ceph-mon[103278]: from='client.58755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:02 smithi078 ceph-mon[91136]: pgmap v6482: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:02 smithi078 ceph-mon[99959]: pgmap v6482: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:03.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:02 smithi146 ceph-mon[103278]: pgmap v6482: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:04 smithi078 ceph-mon[91136]: pgmap v6483: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:04 smithi078 ceph-mon[99959]: pgmap v6483: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:05.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:04 smithi146 ceph-mon[103278]: pgmap v6483: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:06 smithi078 ceph-mon[91136]: pgmap v6484: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:06 smithi078 ceph-mon[99959]: pgmap v6484: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:06 smithi146 ceph-mon[103278]: pgmap v6484: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:08 smithi078 ceph-mon[91136]: pgmap v6485: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:08 smithi078 ceph-mon[99959]: pgmap v6485: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:08 smithi146 ceph-mon[103278]: pgmap v6485: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:10 smithi146 ceph-mon[103278]: pgmap v6486: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[99959]: pgmap v6486: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[91136]: pgmap v6486: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:10.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:56:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:56:11.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:11.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:11.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:11.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:11.184 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:11.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:12 smithi078 ceph-mon[99959]: pgmap v6487: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:12 smithi078 ceph-mon[91136]: pgmap v6487: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:12.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:12 smithi146 ceph-mon[103278]: pgmap v6487: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[91136]: pgmap v6488: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[99959]: pgmap v6488: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:14 smithi146 ceph-mon[103278]: pgmap v6488: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:16 smithi078 ceph-mon[99959]: pgmap v6489: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:16 smithi078 ceph-mon[91136]: pgmap v6489: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:16 smithi146 ceph-mon[103278]: pgmap v6489: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:18 smithi146 ceph-mon[103278]: pgmap v6490: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:18 smithi078 ceph-mon[91136]: pgmap v6490: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:18 smithi078 ceph-mon[99959]: pgmap v6490: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:20 smithi078 ceph-mon[91136]: pgmap v6491: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:20 smithi078 ceph-mon[99959]: pgmap v6491: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:20 smithi146 ceph-mon[103278]: pgmap v6491: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:56:20] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:56:21.296 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:21.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:21.296 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:21.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:21.296 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:21.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:22.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:22 smithi078 ceph-mon[99959]: pgmap v6492: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:22.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:22 smithi078 ceph-mon[91136]: pgmap v6492: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:22 smithi146 ceph-mon[103278]: pgmap v6492: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:24.634 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:24 smithi146 ceph-mon[103278]: pgmap v6493: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:24 smithi078 ceph-mon[91136]: pgmap v6493: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:24 smithi078 ceph-mon[99959]: pgmap v6493: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:26 smithi146 ceph-mon[103278]: pgmap v6494: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:26 smithi078 ceph-mon[91136]: pgmap v6494: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:26 smithi078 ceph-mon[99959]: pgmap v6494: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:28 smithi146 ceph-mon[103278]: pgmap v6495: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:28 smithi078 ceph-mon[91136]: pgmap v6495: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:28 smithi078 ceph-mon[99959]: pgmap v6495: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:30 smithi146 ceph-mon[103278]: pgmap v6496: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:56:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:56:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[91136]: pgmap v6496: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[99959]: pgmap v6496: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:30.889 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:56:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:31.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:31.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:31.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:31.422 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 3h - - 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 3h - - 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 2s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 17s ago 3h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 17s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 3h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 17s ago 3h 249M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 2s ago 3h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:56:31.423 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 17s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 2s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 2s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 2s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 17s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 17s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 17s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 17s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 17s ago 3h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 2s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 17s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 2s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:56:31.424 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 17s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:56:31.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:31.672 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:31.672 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:56:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:56:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:56:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:56:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:56:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:56:31.745 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:56:32.016 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:56:32.016 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:56:32.016 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:56:32.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:56:32.329 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:56:32.329 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:32.329 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:56:32.329 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:32.330 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:56:32.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[99959]: from='client.78683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[99959]: from='client.78689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[99959]: pgmap v6497: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[99959]: from='client.58779 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3523870429' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4135847435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[91136]: from='client.78683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[91136]: from='client.78689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[91136]: pgmap v6497: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[91136]: from='client.58779 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3523870429' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:32.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4135847435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:32.678 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:32 smithi146 ceph-mon[103278]: from='client.78683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:32 smithi146 ceph-mon[103278]: from='client.78689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:32 smithi146 ceph-mon[103278]: pgmap v6497: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:32.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:32 smithi146 ceph-mon[103278]: from='client.58779 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:32.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3523870429' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:56:32.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4135847435' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:56:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:33 smithi078 ceph-mon[91136]: from='client.58791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:33 smithi078 ceph-mon[99959]: from='client.58791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:33 smithi146 ceph-mon[103278]: from='client.58791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:34 smithi146 ceph-mon[103278]: pgmap v6498: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[91136]: pgmap v6498: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[99959]: pgmap v6498: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:36 smithi146 ceph-mon[103278]: pgmap v6499: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:36 smithi078 ceph-mon[91136]: pgmap v6499: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:36 smithi078 ceph-mon[99959]: pgmap v6499: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:38 smithi146 ceph-mon[103278]: pgmap v6500: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:38 smithi078 ceph-mon[91136]: pgmap v6500: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:38 smithi078 ceph-mon[99959]: pgmap v6500: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:40 smithi146 ceph-mon[103278]: pgmap v6501: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:40 smithi078 ceph-mon[91136]: pgmap v6501: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:40 smithi078 ceph-mon[99959]: pgmap v6501: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:56:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:56:41.432 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:41.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:41.432 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:41.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:41.432 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:41.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:42.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:42 smithi146 ceph-mon[103278]: pgmap v6502: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:42 smithi078 ceph-mon[91136]: pgmap v6502: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:42 smithi078 ceph-mon[99959]: pgmap v6502: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:44 smithi146 ceph-mon[103278]: pgmap v6503: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:44 smithi078 ceph-mon[91136]: pgmap v6503: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:44 smithi078 ceph-mon[99959]: pgmap v6503: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:46 smithi146 ceph-mon[103278]: pgmap v6504: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:46 smithi078 ceph-mon[91136]: pgmap v6504: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:46 smithi078 ceph-mon[99959]: pgmap v6504: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:48 smithi146 ceph-mon[103278]: pgmap v6505: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:48 smithi078 ceph-mon[91136]: pgmap v6505: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:48 smithi078 ceph-mon[99959]: pgmap v6505: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:50 smithi146 ceph-mon[103278]: pgmap v6506: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:56:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:56:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[91136]: pgmap v6506: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[99959]: pgmap v6506: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:51.040Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:51.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:56:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:56:51.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:56:52.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:52.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:56:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:52 smithi078 ceph-mon[91136]: pgmap v6507: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:52 smithi078 ceph-mon[99959]: pgmap v6507: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:52 smithi146 ceph-mon[103278]: pgmap v6507: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[99959]: pgmap v6508: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[91136]: pgmap v6508: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:54 smithi146 ceph-mon[103278]: pgmap v6508: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:56:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:56:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:56 smithi146 ceph-mon[103278]: pgmap v6509: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:56 smithi078 ceph-mon[91136]: pgmap v6509: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:56 smithi078 ceph-mon[99959]: pgmap v6509: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:58 smithi146 ceph-mon[103278]: pgmap v6510: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:56:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:58 smithi078 ceph-mon[99959]: pgmap v6510: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:56:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:56:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:58 smithi078 ceph-mon[91136]: pgmap v6510: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:56:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:56:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:00 smithi146 ceph-mon[103278]: pgmap v6511: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:00 smithi078 ceph-mon[99959]: pgmap v6511: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:00 smithi078 ceph-mon[91136]: pgmap v6511: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:57:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:57:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:01.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:01.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:01.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:01.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:02.625 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:57:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:02 smithi146 ceph-mon[103278]: pgmap v6512: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:02 smithi078 ceph-mon[91136]: pgmap v6512: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:02 smithi078 ceph-mon[99959]: pgmap v6512: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 3h - - 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 3h - - 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 9s ago 3h 91.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 9s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:57:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 3h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 9s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 9s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 13s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 13s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 13s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 9s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 9s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 9s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 9s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 9s ago 3h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:57:03.151 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 9s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:57:03.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:57:03.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 9s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:57:03.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:57:03.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:57:03.731 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:57:03.732 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:57:03.732 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:57:03.732 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:57:03.732 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:57:03.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:57:03.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:03 smithi078 ceph-mon[99959]: from='client.78719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:03.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1267548226' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:03.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:03 smithi078 ceph-mon[91136]: from='client.78719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:03.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1267548226' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:03 smithi146 ceph-mon[103278]: from='client.78719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1267548226' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:04.042 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:57:04.042 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:57:04.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:57:04.042 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:57:04.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:57:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:04 smithi146 ceph-mon[103278]: from='client.58809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:04 smithi146 ceph-mon[103278]: from='client.58815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:04 smithi146 ceph-mon[103278]: pgmap v6513: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:04 smithi146 ceph-mon[103278]: from='client.78743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3826496207' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[91136]: from='client.58809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[91136]: from='client.58815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[91136]: pgmap v6513: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[91136]: from='client.78743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3826496207' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[99959]: from='client.58809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[99959]: from='client.58815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[99959]: pgmap v6513: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[99959]: from='client.78743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3826496207' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:06 smithi146 ceph-mon[103278]: pgmap v6514: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:06 smithi078 ceph-mon[91136]: pgmap v6514: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:06 smithi078 ceph-mon[99959]: pgmap v6514: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:08 smithi146 ceph-mon[103278]: pgmap v6515: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:08 smithi078 ceph-mon[99959]: pgmap v6515: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:08 smithi078 ceph-mon[91136]: pgmap v6515: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:10 smithi146 ceph-mon[103278]: pgmap v6516: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[91136]: pgmap v6516: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:57:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:57:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[99959]: pgmap v6516: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:11.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:11.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:11.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:12 smithi078 ceph-mon[91136]: pgmap v6517: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:12 smithi078 ceph-mon[99959]: pgmap v6517: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:12 smithi146 ceph-mon[103278]: pgmap v6517: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:13.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:13.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:14 smithi146 ceph-mon[103278]: pgmap v6518: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[99959]: pgmap v6518: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[91136]: pgmap v6518: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:16 smithi146 ceph-mon[103278]: pgmap v6519: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:16 smithi078 ceph-mon[91136]: pgmap v6519: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:16 smithi078 ceph-mon[99959]: pgmap v6519: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:18 smithi146 ceph-mon[103278]: pgmap v6520: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:18 smithi078 ceph-mon[91136]: pgmap v6520: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:18 smithi078 ceph-mon[99959]: pgmap v6520: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:20 smithi146 ceph-mon[103278]: pgmap v6521: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:20 smithi078 ceph-mon[91136]: pgmap v6521: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:20 smithi078 ceph-mon[99959]: pgmap v6521: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:57:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-20T23:57:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:21.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:21.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:21.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:21.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:22 smithi146 ceph-mon[103278]: pgmap v6522: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:22 smithi078 ceph-mon[91136]: pgmap v6522: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:22 smithi078 ceph-mon[99959]: pgmap v6522: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:24 smithi146 ceph-mon[103278]: pgmap v6523: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:24 smithi078 ceph-mon[91136]: pgmap v6523: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:24 smithi078 ceph-mon[99959]: pgmap v6523: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:26 smithi146 ceph-mon[103278]: pgmap v6524: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:26 smithi078 ceph-mon[91136]: pgmap v6524: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:26 smithi078 ceph-mon[99959]: pgmap v6524: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:28 smithi146 ceph-mon[103278]: pgmap v6525: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:28 smithi078 ceph-mon[99959]: pgmap v6525: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:28 smithi078 ceph-mon[91136]: pgmap v6525: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:57:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:57:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[91136]: pgmap v6526: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[99959]: pgmap v6526: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:30 smithi146 ceph-mon[103278]: pgmap v6526: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:31.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:31.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:31.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:32 smithi146 ceph-mon[103278]: pgmap v6527: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:33.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:32 smithi078 ceph-mon[91136]: pgmap v6527: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:33.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:32 smithi078 ceph-mon[99959]: pgmap v6527: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:34.346 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:57:34.894 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 3h - - 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 3h - - 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 5s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 0s ago 3h 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 0s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 5s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 5s ago 3h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 0s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 5s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 5s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:57:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 0s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 5s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 5s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 5s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 5s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 0s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 0s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 0s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 0s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 0s ago 3h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 5s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 0s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 5s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:57:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 0s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:57:35.217 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:57:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:57:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:57:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:57:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:57:35.218 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[99959]: pgmap v6528: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[91136]: pgmap v6528: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:34 smithi146 ceph-mon[103278]: pgmap v6528: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:35.492 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:57:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:57:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:57:35.492 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:57:35.493 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:57:35.809 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:57:35.810 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:57:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:57:35.810 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:57:35.810 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:57:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:35 smithi146 ceph-mon[103278]: from='client.58839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:35 smithi146 ceph-mon[103278]: from='client.78761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2691053856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:36.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1298094697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[91136]: from='client.58839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[91136]: from='client.78761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2691053856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1298094697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[99959]: from='client.58839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[99959]: from='client.78761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2691053856' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:57:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1298094697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:57:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:36 smithi146 ceph-mon[103278]: from='client.78767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:36 smithi146 ceph-mon[103278]: pgmap v6529: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:36 smithi146 ceph-mon[103278]: from='client.58863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:36 smithi078 ceph-mon[91136]: from='client.78767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:36 smithi078 ceph-mon[91136]: pgmap v6529: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:36 smithi078 ceph-mon[91136]: from='client.58863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:36 smithi078 ceph-mon[99959]: from='client.78767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:36 smithi078 ceph-mon[99959]: pgmap v6529: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:36 smithi078 ceph-mon[99959]: from='client.58863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:37 smithi146 ceph-mon[103278]: pgmap v6530: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:37 smithi078 ceph-mon[91136]: pgmap v6530: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:37 smithi078 ceph-mon[99959]: pgmap v6530: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:40 smithi146 ceph-mon[103278]: pgmap v6531: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:40 smithi078 ceph-mon[91136]: pgmap v6531: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:40 smithi078 ceph-mon[99959]: pgmap v6531: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:57:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:57:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:41.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:41.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:41.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:42 smithi146 ceph-mon[103278]: pgmap v6532: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:42 smithi078 ceph-mon[99959]: pgmap v6532: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:42 smithi078 ceph-mon[91136]: pgmap v6532: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:44 smithi146 ceph-mon[103278]: pgmap v6533: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:44 smithi078 ceph-mon[99959]: pgmap v6533: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:44 smithi078 ceph-mon[91136]: pgmap v6533: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:46 smithi146 ceph-mon[103278]: pgmap v6534: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:46 smithi078 ceph-mon[91136]: pgmap v6534: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:46 smithi078 ceph-mon[99959]: pgmap v6534: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:49.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:48 smithi146 ceph-mon[103278]: pgmap v6535: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:48 smithi078 ceph-mon[91136]: pgmap v6535: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:48 smithi078 ceph-mon[99959]: pgmap v6535: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[91136]: pgmap v6536: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:57:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:57:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[99959]: pgmap v6536: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:50 smithi146 ceph-mon[103278]: pgmap v6536: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:51.044Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:51.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:57:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:57:51.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:57:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:57:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:52 smithi078 ceph-mon[91136]: pgmap v6537: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:52 smithi078 ceph-mon[99959]: pgmap v6537: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:52 smithi146 ceph-mon[103278]: pgmap v6537: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:54.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[91136]: pgmap v6538: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[99959]: pgmap v6538: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:54.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:54 smithi146 ceph-mon[103278]: pgmap v6538: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:57:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:57:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:56 smithi078 ceph-mon[91136]: pgmap v6539: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:56 smithi078 ceph-mon[99959]: pgmap v6539: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:56 smithi146 ceph-mon[103278]: pgmap v6539: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:58 smithi146 ceph-mon[103278]: pgmap v6540: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:58 smithi078 ceph-mon[91136]: pgmap v6540: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:57:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:58 smithi078 ceph-mon[99959]: pgmap v6540: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:57:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:00 smithi078 ceph-mon[99959]: pgmap v6541: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:00 smithi078 ceph-mon[91136]: pgmap v6541: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:58:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:58:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:00 smithi146 ceph-mon[103278]: pgmap v6541: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:01.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:01.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:01.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:02.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:02 smithi146 ceph-mon[103278]: pgmap v6542: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:02 smithi078 ceph-mon[99959]: pgmap v6542: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:02 smithi078 ceph-mon[91136]: pgmap v6542: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:04 smithi146 ceph-mon[103278]: pgmap v6543: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:04 smithi078 ceph-mon[99959]: pgmap v6543: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:04 smithi078 ceph-mon[91136]: pgmap v6543: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:06.100 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:58:06.655 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:58:06.693 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 3h - - 2024-09-20T23:58:06.693 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 3h - - 2024-09-20T23:58:06.693 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 16s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:58:06.693 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 12s ago 3h 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 3h 387M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 3h 250M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 16s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 16s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 12s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 16s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 16s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 16s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 16s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 12s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 12s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 12s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:58:06.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 12s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:58:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 3h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:58:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 16s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:58:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 12s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:58:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 16s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:58:06.695 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 12s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:58:06.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:58:06.964 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:58:06.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:06 smithi078 ceph-mon[99959]: pgmap v6544: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:06.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:06 smithi078 ceph-mon[91136]: pgmap v6544: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:07.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:58:07.233 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:58:07.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:58:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:06 smithi146 ceph-mon[103278]: pgmap v6544: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:07.543 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:58:07.543 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:07.543 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:07.544 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:07.544 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:07 smithi146 ceph-mon[103278]: from='client.58875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:07 smithi146 ceph-mon[103278]: from='client.78797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:07 smithi146 ceph-mon[103278]: from='client.58887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1886755297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3075884084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[91136]: from='client.58875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[91136]: from='client.78797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[91136]: from='client.58887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1886755297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3075884084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[99959]: from='client.58875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[99959]: from='client.78797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[99959]: from='client.58887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1886755297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3075884084' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:08 smithi146 ceph-mon[103278]: from='client.78815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:08 smithi146 ceph-mon[103278]: pgmap v6545: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:08 smithi078 ceph-mon[91136]: from='client.78815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:08 smithi078 ceph-mon[91136]: pgmap v6545: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:08 smithi078 ceph-mon[99959]: from='client.78815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:08 smithi078 ceph-mon[99959]: pgmap v6545: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[91136]: pgmap v6546: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:58:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-20T23:58:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[99959]: pgmap v6546: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:10 smithi146 ceph-mon[103278]: pgmap v6546: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:11.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:11.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:11.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:12 smithi146 ceph-mon[103278]: pgmap v6547: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:12 smithi078 ceph-mon[99959]: pgmap v6547: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:12 smithi078 ceph-mon[91136]: pgmap v6547: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:14.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:14.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:14 smithi146 ceph-mon[103278]: pgmap v6548: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[99959]: pgmap v6548: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[91136]: pgmap v6548: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:15 smithi078 ceph-mon[99959]: pgmap v6549: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:15 smithi078 ceph-mon[91136]: pgmap v6549: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:15 smithi146 ceph-mon[103278]: pgmap v6549: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:18 smithi146 ceph-mon[103278]: pgmap v6550: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:18 smithi078 ceph-mon[91136]: pgmap v6550: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:18 smithi078 ceph-mon[99959]: pgmap v6550: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:20 smithi078 ceph-mon[91136]: pgmap v6551: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:20 smithi078 ceph-mon[99959]: pgmap v6551: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:20 smithi146 ceph-mon[103278]: pgmap v6551: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:58:20] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-20T23:58:21.328 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:21.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:21.328 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:21.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:21.328 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:21.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:22 smithi078 ceph-mon[91136]: pgmap v6552: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:22 smithi078 ceph-mon[99959]: pgmap v6552: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:22 smithi146 ceph-mon[103278]: pgmap v6552: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:24 smithi146 ceph-mon[103278]: pgmap v6553: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:24 smithi078 ceph-mon[99959]: pgmap v6553: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:24 smithi078 ceph-mon[91136]: pgmap v6553: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:26 smithi146 ceph-mon[103278]: pgmap v6554: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:26 smithi078 ceph-mon[91136]: pgmap v6554: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:26 smithi078 ceph-mon[99959]: pgmap v6554: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:28.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:28 smithi146 ceph-mon[103278]: pgmap v6555: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:28 smithi078 ceph-mon[99959]: pgmap v6555: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:28 smithi078 ceph-mon[91136]: pgmap v6555: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:29.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[91136]: pgmap v6556: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[99959]: pgmap v6556: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.644 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:30 smithi146 ceph-mon[103278]: pgmap v6556: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:30.644 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.644 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:30.644 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:58:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:58:31.357 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:31.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:31.357 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:31.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:31.357 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:31.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:31.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:31.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:32 smithi146 ceph-mon[103278]: pgmap v6557: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:32 smithi078 ceph-mon[91136]: pgmap v6557: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:32 smithi078 ceph-mon[99959]: pgmap v6557: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:34 smithi146 ceph-mon[103278]: pgmap v6558: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:34 smithi078 ceph-mon[91136]: pgmap v6558: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:34 smithi078 ceph-mon[99959]: pgmap v6558: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:36 smithi078 ceph-mon[91136]: pgmap v6559: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:36 smithi078 ceph-mon[99959]: pgmap v6559: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:36 smithi146 ceph-mon[103278]: pgmap v6559: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:37.834 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:58:38.381 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:58:38.381 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 3h - - 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 3h - - 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 8s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 3s ago 3h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 8s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 3s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 8s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:58:38.382 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 8s ago 3h 388M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 3s ago 3h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 8s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 8s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 3s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 8s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 8s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:58:38.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 8s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 8s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 3s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 3s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 3s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 3s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 3s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 8s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:58:38.384 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 3s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:58:38.385 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 8s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:58:38.385 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 3s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:58:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:58:38.699 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:58:38.976 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:58:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:58:38.977 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:58:38.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:38 smithi078 ceph-mon[91136]: pgmap v6560: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:38.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:38 smithi078 ceph-mon[91136]: from='client.78827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:38.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1598683341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:38.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:38 smithi078 ceph-mon[99959]: pgmap v6560: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:38.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:38 smithi078 ceph-mon[99959]: from='client.78827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:38.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1598683341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:38 smithi146 ceph-mon[103278]: pgmap v6560: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:38 smithi146 ceph-mon[103278]: from='client.78827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1598683341' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:58:39.285 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:58:39.286 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:58:39.286 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:58:40.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:39 smithi078 ceph-mon[99959]: from='client.58917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:40.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:39 smithi078 ceph-mon[99959]: from='client.78839 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:40.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1812203898' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:40.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:39 smithi078 ceph-mon[91136]: from='client.58917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:40.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:39 smithi078 ceph-mon[91136]: from='client.78839 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:40.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1812203898' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:39 smithi146 ceph-mon[103278]: from='client.58917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:39 smithi146 ceph-mon[103278]: from='client.78839 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1812203898' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:58:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:40 smithi078 ceph-mon[91136]: from='client.58935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:40 smithi078 ceph-mon[91136]: pgmap v6561: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:58:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:58:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:40 smithi078 ceph-mon[99959]: from='client.58935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:40 smithi078 ceph-mon[99959]: pgmap v6561: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:40 smithi146 ceph-mon[103278]: from='client.58935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:40 smithi146 ceph-mon[103278]: pgmap v6561: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:41.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:41.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:41.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:42 smithi146 ceph-mon[103278]: pgmap v6562: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:42 smithi078 ceph-mon[91136]: pgmap v6562: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:42 smithi078 ceph-mon[99959]: pgmap v6562: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:44 smithi146 ceph-mon[103278]: pgmap v6563: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:44 smithi078 ceph-mon[91136]: pgmap v6563: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:44 smithi078 ceph-mon[99959]: pgmap v6563: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:46 smithi146 ceph-mon[103278]: pgmap v6564: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:46 smithi078 ceph-mon[91136]: pgmap v6564: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:46 smithi078 ceph-mon[99959]: pgmap v6564: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:49.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:48 smithi146 ceph-mon[103278]: pgmap v6565: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:48 smithi078 ceph-mon[91136]: pgmap v6565: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:48 smithi078 ceph-mon[99959]: pgmap v6565: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:50 smithi146 ceph-mon[103278]: pgmap v6566: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:58:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-20T23:58:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[91136]: pgmap v6566: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[99959]: pgmap v6566: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:51.295 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:51.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:51.295 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:51.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:58:51.295 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:58:51.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:58:52.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:52.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:58:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:52 smithi146 ceph-mon[103278]: pgmap v6567: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:52 smithi078 ceph-mon[91136]: pgmap v6567: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:52 smithi078 ceph-mon[99959]: pgmap v6567: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:53 smithi146 ceph-mon[103278]: pgmap v6568: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:53 smithi078 ceph-mon[91136]: pgmap v6568: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:53 smithi078 ceph-mon[99959]: pgmap v6568: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:58:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:55 smithi146 ceph-mon[103278]: pgmap v6569: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[91136]: pgmap v6569: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:58:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:55 smithi078 ceph-mon[99959]: pgmap v6569: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:58 smithi146 ceph-mon[103278]: pgmap v6570: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:58 smithi078 ceph-mon[91136]: pgmap v6570: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:58 smithi078 ceph-mon[99959]: pgmap v6570: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:58:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:58:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:00 smithi078 ceph-mon[91136]: pgmap v6571: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:00 smithi078 ceph-mon[99959]: pgmap v6571: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:00 smithi146 ceph-mon[103278]: pgmap v6571: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:59:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:59:01.354 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:01.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:01.354 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:01.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:01.354 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:01.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:59:01.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:01.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:02.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:02 smithi146 ceph-mon[103278]: pgmap v6572: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:02 smithi078 ceph-mon[91136]: pgmap v6572: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:02 smithi078 ceph-mon[99959]: pgmap v6572: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:04 smithi146 ceph-mon[103278]: pgmap v6573: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:04 smithi078 ceph-mon[91136]: pgmap v6573: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:04 smithi078 ceph-mon[99959]: pgmap v6573: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:06 smithi146 ceph-mon[103278]: pgmap v6574: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:06 smithi078 ceph-mon[99959]: pgmap v6574: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:06 smithi078 ceph-mon[91136]: pgmap v6574: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:08 smithi146 ceph-mon[103278]: pgmap v6575: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:08 smithi078 ceph-mon[91136]: pgmap v6575: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:08 smithi078 ceph-mon[99959]: pgmap v6575: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:09.587 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 3h - - 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 3h - - 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 19s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 15s ago 3h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 19s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 19s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:59:10.114 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 19s ago 3h 389M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 3h 251M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 19s ago 3h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 19s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 15s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 19s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 19s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 19s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 19s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 15s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 15s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 15s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 19s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 15s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:59:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 19s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:59:10.116 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 15s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:59:10.430 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:59:10.431 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:59:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:10 smithi146 ceph-mon[103278]: pgmap v6576: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:10 smithi146 ceph-mon[103278]: from='client.58947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:10.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:10 smithi078 ceph-mon[91136]: pgmap v6576: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:10.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:10 smithi078 ceph-mon[91136]: from='client.58947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:10.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:10 smithi078 ceph-mon[99959]: pgmap v6576: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:10.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:10 smithi078 ceph-mon[99959]: from='client.58947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:59:10.706 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:59:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:59:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:59:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:59:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:59:10.707 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:59:10.707 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:59:10.707 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:59:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-20T23:59:11.014 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:59:11.014 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:11.014 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:11.014 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:11.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:11.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:11.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='client.78869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='client.58959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4074742842' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='client.58971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/625703826' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='client.78869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='client.58959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4074742842' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='client.58971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/625703826' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='client.78869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='client.58959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4074742842' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='client.58971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/625703826' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:12 smithi146 ceph-mon[103278]: pgmap v6577: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:12 smithi078 ceph-mon[91136]: pgmap v6577: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:12 smithi078 ceph-mon[99959]: pgmap v6577: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:14 smithi146 ceph-mon[103278]: pgmap v6578: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:14 smithi078 ceph-mon[91136]: pgmap v6578: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:14 smithi078 ceph-mon[99959]: pgmap v6578: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[91136]: pgmap v6579: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:16 smithi078 ceph-mon[99959]: pgmap v6579: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:16 smithi146 ceph-mon[103278]: pgmap v6579: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:18 smithi146 ceph-mon[103278]: pgmap v6580: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:18 smithi078 ceph-mon[91136]: pgmap v6580: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:18 smithi078 ceph-mon[99959]: pgmap v6580: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:20 smithi078 ceph-mon[91136]: pgmap v6581: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:20 smithi078 ceph-mon[99959]: pgmap v6581: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:20 smithi146 ceph-mon[103278]: pgmap v6581: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:59:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:59:21.351 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:21.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:21.351 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:21.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:21.351 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:21.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:59:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:22 smithi146 ceph-mon[103278]: pgmap v6582: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:22 smithi078 ceph-mon[91136]: pgmap v6582: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:22 smithi078 ceph-mon[99959]: pgmap v6582: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:24 smithi146 ceph-mon[103278]: pgmap v6583: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:24 smithi078 ceph-mon[91136]: pgmap v6583: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:24 smithi078 ceph-mon[99959]: pgmap v6583: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:26 smithi078 ceph-mon[91136]: pgmap v6584: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:26 smithi078 ceph-mon[99959]: pgmap v6584: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:26 smithi146 ceph-mon[103278]: pgmap v6584: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:28 smithi078 ceph-mon[91136]: pgmap v6585: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:28 smithi078 ceph-mon[99959]: pgmap v6585: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:28 smithi146 ceph-mon[103278]: pgmap v6585: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[91136]: pgmap v6586: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:59:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:59:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[99959]: pgmap v6586: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:30 smithi146 ceph-mon[103278]: pgmap v6586: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:31.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:31.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:31.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:31.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:59:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:32.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:32 smithi078 ceph-mon[91136]: pgmap v6587: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:32 smithi078 ceph-mon[99959]: pgmap v6587: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:33.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:32 smithi146 ceph-mon[103278]: pgmap v6587: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:35.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:34 smithi146 ceph-mon[103278]: pgmap v6588: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:34 smithi078 ceph-mon[91136]: pgmap v6588: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:34 smithi078 ceph-mon[99959]: pgmap v6588: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:36 smithi146 ceph-mon[103278]: pgmap v6589: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[91136]: pgmap v6589: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:36 smithi078 ceph-mon[99959]: pgmap v6589: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:38.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:38 smithi078 ceph-mon[99959]: pgmap v6590: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:38.407 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:38 smithi078 ceph-mon[91136]: pgmap v6590: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:38 smithi146 ceph-mon[103278]: pgmap v6590: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:40 smithi078 ceph-mon[91136]: pgmap v6591: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:40 smithi078 ceph-mon[99959]: pgmap v6591: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:40 smithi146 ceph-mon[103278]: pgmap v6591: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:59:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:59:41.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:41.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:41.313 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:41.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:41.313 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:41.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:59:41.313 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-20T23:59:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 3h - - 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 3h - - 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 11s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 6s ago 3h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 11s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 6s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 11s ago 3h 577M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 11s ago 3h 390M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 6s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 11s ago 3h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-20T23:59:41.847 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 11s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 6s ago 3h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 11s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 11s ago 3h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 11s ago 3h 73.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 11s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 6s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 6s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 6s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 6s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 6s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 11s ago 3h 88.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 6s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 11s ago 3h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-20T23:59:41.848 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 6s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-20T23:59:42.159 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:59:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-20T23:59:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-20T23:59:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-20T23:59:42.160 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:59:42.432 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-20T23:59:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-20T23:59:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-20T23:59:42.432 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-20T23:59:42.433 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-20T23:59:42.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[99959]: pgmap v6592: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:42.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[99959]: from='client.78899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[99959]: from='client.58989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[99959]: from='client.58995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/952341597' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[91136]: pgmap v6592: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[91136]: from='client.78899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[91136]: from='client.58989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[91136]: from='client.58995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/952341597' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:42.739 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-20T23:59:42.739 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:42.739 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-20T23:59:42.739 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:42.740 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-20T23:59:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:42 smithi146 ceph-mon[103278]: pgmap v6592: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:42 smithi146 ceph-mon[103278]: from='client.78899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:42 smithi146 ceph-mon[103278]: from='client.58989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:42 smithi146 ceph-mon[103278]: from='client.58995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/952341597' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-20T23:59:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:43 smithi146 ceph-mon[103278]: from='client.59007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/289357996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:43 smithi078 ceph-mon[91136]: from='client.59007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/289357996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:43 smithi078 ceph-mon[99959]: from='client.59007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/289357996' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-20T23:59:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:44 smithi146 ceph-mon[103278]: pgmap v6593: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:44 smithi078 ceph-mon[99959]: pgmap v6593: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:44 smithi078 ceph-mon[91136]: pgmap v6593: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:46 smithi146 ceph-mon[103278]: pgmap v6594: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:46 smithi078 ceph-mon[91136]: pgmap v6594: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:46 smithi078 ceph-mon[99959]: pgmap v6594: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:48 smithi146 ceph-mon[103278]: pgmap v6595: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:48 smithi078 ceph-mon[91136]: pgmap v6595: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:48 smithi078 ceph-mon[99959]: pgmap v6595: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:50 smithi078 ceph-mon[91136]: pgmap v6596: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:50 smithi078 ceph-mon[99959]: pgmap v6596: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:50 smithi146 ceph-mon[103278]: pgmap v6596: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 20 23:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [20/Sep/2024:23:59:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-20T23:59:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:51.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:51.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-20T23:59:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-20T23:59:51.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-20T23:59:51.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:51.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:51.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:51.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T23:59:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:52 smithi078 ceph-mon[91136]: pgmap v6597: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:52 smithi078 ceph-mon[99959]: pgmap v6597: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:52 smithi146 ceph-mon[103278]: pgmap v6597: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:54 smithi146 ceph-mon[103278]: pgmap v6598: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:54 smithi078 ceph-mon[91136]: pgmap v6598: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:54 smithi078 ceph-mon[99959]: pgmap v6598: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T23:59:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:56 smithi146 ceph-mon[103278]: pgmap v6599: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[99959]: pgmap v6599: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[91136]: pgmap v6599: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-20T23:59:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:58 smithi078 ceph-mon[91136]: pgmap v6600: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:58 smithi078 ceph-mon[99959]: pgmap v6600: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:58 smithi146 ceph-mon[103278]: pgmap v6600: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T23:59:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 20 23:59:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 20 23:59:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-20T23:59:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 20 23:59:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-20T23:59:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:00 smithi078 ceph-mon[99959]: pgmap v6601: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:00 smithi078 ceph-mon[91136]: pgmap v6601: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:00 smithi146 ceph-mon[103278]: pgmap v6601: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:00:00] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:00:01.319 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:01.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:01.319 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:01.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:01.319 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:01.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:02.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:02 smithi146 ceph-mon[103278]: pgmap v6602: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T00:00:02.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:02 smithi078 ceph-mon[91136]: pgmap v6602: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T00:00:02.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:02 smithi078 ceph-mon[99959]: pgmap v6602: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T00:00:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:04 smithi146 ceph-mon[103278]: pgmap v6603: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T00:00:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:04 smithi078 ceph-mon[91136]: pgmap v6603: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T00:00:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:04 smithi078 ceph-mon[99959]: pgmap v6603: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T00:00:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:06 smithi146 ceph-mon[103278]: pgmap v6604: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-21T00:00:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:06 smithi078 ceph-mon[91136]: pgmap v6604: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-21T00:00:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:06 smithi078 ceph-mon[99959]: pgmap v6604: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-21T00:00:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:08 smithi146 ceph-mon[103278]: pgmap v6605: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:08 smithi078 ceph-mon[99959]: pgmap v6605: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:08 smithi078 ceph-mon[91136]: pgmap v6605: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:10.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:10 smithi078 ceph-mon[99959]: pgmap v6606: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-21T00:00:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:10 smithi078 ceph-mon[91136]: pgmap v6606: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-21T00:00:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:10 smithi146 ceph-mon[103278]: pgmap v6606: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 161 op/s 2024-09-21T00:00:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:00:10] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:00:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:11.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:11.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:11.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:12.192 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:12.192 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:13.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:12 smithi146 ceph-mon[103278]: pgmap v6607: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:13.034 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:00:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:12 smithi078 ceph-mon[99959]: pgmap v6607: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:12 smithi078 ceph-mon[91136]: pgmap v6607: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 160 op/s 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 3h - - 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 3h - - 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 2s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 18s ago 3h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 18s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 3h 391M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:00:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 18s ago 3h 253M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 2s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 18s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 3h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 2s ago 3h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 2s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 2s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 18s ago 3h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 18s ago 3h 76.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 18s ago 3h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 18s ago 3h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:00:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 18s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:00:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:00:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 18s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:00:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 2s ago 3h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:00:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 18s ago 3h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:00:13.854 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:13 smithi146 ceph-mon[103278]: from='client.59019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:13.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:13 smithi146 ceph-mon[103278]: from='client.78941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:13.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:13 smithi146 ceph-mon[103278]: pgmap v6608: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:00:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:00:13.884 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:00:14.157 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:00:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:00:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:00:14.158 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:00:14.158 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:00:14.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[99959]: from='client.59019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[99959]: from='client.78941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[99959]: pgmap v6608: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T00:00:14.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:14.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[99959]: from='client.78947 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[91136]: from='client.59019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[91136]: from='client.78941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[91136]: pgmap v6608: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T00:00:14.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:14.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:13 smithi078 ceph-mon[91136]: from='client.78947 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:13 smithi146 ceph-mon[103278]: from='client.78947 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:14.468 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:14.468 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:14.468 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:14.468 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:14.468 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2195530373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:14 smithi146 ceph-mon[103278]: from='client.78959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/840418823' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2195530373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:14 smithi078 ceph-mon[91136]: from='client.78959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/840418823' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2195530373' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:14 smithi078 ceph-mon[99959]: from='client.78959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/840418823' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:16 smithi146 ceph-mon[103278]: pgmap v6609: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:00:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[91136]: pgmap v6609: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[99959]: pgmap v6609: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:18 smithi078 ceph-mon[91136]: pgmap v6610: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T00:00:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:18 smithi078 ceph-mon[99959]: pgmap v6610: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T00:00:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:18 smithi146 ceph-mon[103278]: pgmap v6610: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T00:00:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:20 smithi146 ceph-mon[103278]: pgmap v6611: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:20 smithi078 ceph-mon[91136]: pgmap v6611: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:00:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:00:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:20 smithi078 ceph-mon[99959]: pgmap v6611: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:21.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:21.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:21.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:22.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:22 smithi078 ceph-mon[91136]: pgmap v6612: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:22.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:22 smithi078 ceph-mon[99959]: pgmap v6612: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:22 smithi146 ceph-mon[103278]: pgmap v6612: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:24 smithi146 ceph-mon[103278]: pgmap v6613: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:24 smithi078 ceph-mon[91136]: pgmap v6613: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:24 smithi078 ceph-mon[99959]: pgmap v6613: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:26 smithi146 ceph-mon[103278]: pgmap v6614: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:26 smithi078 ceph-mon[91136]: pgmap v6614: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:26 smithi078 ceph-mon[99959]: pgmap v6614: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:28 smithi146 ceph-mon[103278]: pgmap v6615: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:28 smithi078 ceph-mon[91136]: pgmap v6615: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:28 smithi078 ceph-mon[99959]: pgmap v6615: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:30.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:30 smithi078 ceph-mon[91136]: pgmap v6616: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:30.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:30 smithi078 ceph-mon[99959]: pgmap v6616: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:30.985 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:00:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:00:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:30 smithi146 ceph-mon[103278]: pgmap v6616: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:31.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:31.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:31.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:32.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:32.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:32 smithi146 ceph-mon[103278]: pgmap v6617: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:32 smithi078 ceph-mon[91136]: pgmap v6617: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:32 smithi078 ceph-mon[99959]: pgmap v6617: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:34 smithi146 ceph-mon[103278]: pgmap v6618: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:34 smithi078 ceph-mon[91136]: pgmap v6618: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:34 smithi078 ceph-mon[99959]: pgmap v6618: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:36 smithi146 ceph-mon[103278]: pgmap v6619: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[91136]: pgmap v6619: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[99959]: pgmap v6619: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:38.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:38 smithi078 ceph-mon[91136]: pgmap v6620: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:38.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:38 smithi078 ceph-mon[99959]: pgmap v6620: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:38 smithi146 ceph-mon[103278]: pgmap v6620: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:40 smithi078 ceph-mon[91136]: pgmap v6621: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:00:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:00:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:40 smithi078 ceph-mon[99959]: pgmap v6621: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:40 smithi146 ceph-mon[103278]: pgmap v6621: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:41.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:41.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:41.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:41.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:41.437 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:41.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:42 smithi078 ceph-mon[99959]: pgmap v6622: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:42 smithi078 ceph-mon[91136]: pgmap v6622: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:42 smithi146 ceph-mon[103278]: pgmap v6622: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:44.759 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:00:44.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:44 smithi146 ceph-mon[103278]: pgmap v6623: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:44 smithi078 ceph-mon[91136]: pgmap v6623: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:44 smithi078 ceph-mon[99959]: pgmap v6623: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:45.283 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:00:45.283 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 3h - - 2024-09-21T00:00:45.283 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 3h - - 2024-09-21T00:00:45.283 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 14s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 9s ago 3h 87.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 14s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 9s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 14s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 14s ago 3h 392M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 9s ago 3h 254M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 14s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 14s ago 3h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 9s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 14s ago 3h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 14s ago 3h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 14s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 14s ago 3h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:00:45.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 9s ago 3h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 9s ago 3h 76.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 9s ago 3h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 9s ago 3h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 9s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 14s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 9s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 14s ago 3h 90.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:00:45.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 9s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:00:45.601 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:00:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:00:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:00:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:00:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:00:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:00:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:00:45.603 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:00:45.870 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:45 smithi078 ceph-mon[99959]: from='client.78971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:45.871 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:45 smithi078 ceph-mon[91136]: from='client.78971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:00:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:00:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:00:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:00:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:00:45.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:00:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:45 smithi146 ceph-mon[103278]: from='client.78971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:46.181 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:00:46.182 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:00:46.182 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:00:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:46 smithi146 ceph-mon[103278]: from='client.59061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:46 smithi146 ceph-mon[103278]: from='client.59067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:46 smithi146 ceph-mon[103278]: pgmap v6624: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3281199530' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2171595621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[91136]: from='client.59061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[91136]: from='client.59067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[91136]: pgmap v6624: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3281199530' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2171595621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[99959]: from='client.59061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[99959]: from='client.59067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[99959]: pgmap v6624: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3281199530' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:00:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2171595621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:00:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:47 smithi146 ceph-mon[103278]: from='client.78995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:47 smithi078 ceph-mon[99959]: from='client.78995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:47 smithi078 ceph-mon[91136]: from='client.78995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:00:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:48 smithi078 ceph-mon[91136]: pgmap v6625: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:48 smithi078 ceph-mon[99959]: pgmap v6625: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:48 smithi146 ceph-mon[103278]: pgmap v6625: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:50 smithi146 ceph-mon[103278]: pgmap v6626: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:50 smithi078 ceph-mon[91136]: pgmap v6626: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:50 smithi078 ceph-mon[99959]: pgmap v6626: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:00:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:00:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:00:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:51.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:51.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:00:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:00:51.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:00:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:00:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:52.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:52 smithi146 ceph-mon[103278]: pgmap v6627: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:52 smithi078 ceph-mon[99959]: pgmap v6627: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:52 smithi078 ceph-mon[91136]: pgmap v6627: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:00:54.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:54 smithi146 ceph-mon[103278]: pgmap v6628: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:00:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:54 smithi078 ceph-mon[91136]: pgmap v6628: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:00:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:54 smithi078 ceph-mon[99959]: pgmap v6628: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:00:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:56 smithi146 ceph-mon[103278]: pgmap v6629: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T00:00:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[91136]: pgmap v6629: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T00:00:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[99959]: pgmap v6629: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T00:00:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:00:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:58 smithi146 ceph-mon[103278]: pgmap v6630: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:00:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:00:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:58 smithi078 ceph-mon[91136]: pgmap v6630: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:00:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:00:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:00:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:58 smithi078 ceph-mon[99959]: pgmap v6630: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:00:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:00:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:00.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:00 smithi078 ceph-mon[91136]: pgmap v6631: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:01:00.947 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:01:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:01:00.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:00 smithi078 ceph-mon[99959]: pgmap v6631: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:01:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:00 smithi146 ceph-mon[103278]: pgmap v6631: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:01:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:01.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:01.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:01.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:02 smithi146 ceph-mon[103278]: pgmap v6632: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:01:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:02 smithi078 ceph-mon[91136]: pgmap v6632: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:01:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:02 smithi078 ceph-mon[99959]: pgmap v6632: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:01:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:04 smithi146 ceph-mon[103278]: pgmap v6633: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:01:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:04 smithi078 ceph-mon[91136]: pgmap v6633: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:01:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:04 smithi078 ceph-mon[99959]: pgmap v6633: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:01:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:06 smithi146 ceph-mon[103278]: pgmap v6634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T00:01:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:06 smithi078 ceph-mon[99959]: pgmap v6634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T00:01:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:06 smithi078 ceph-mon[91136]: pgmap v6634: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T00:01:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:08 smithi146 ceph-mon[103278]: pgmap v6635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T00:01:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:08 smithi078 ceph-mon[99959]: pgmap v6635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T00:01:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:08 smithi078 ceph-mon[91136]: pgmap v6635: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-21T00:01:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:10 smithi146 ceph-mon[103278]: pgmap v6636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:01:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:10 smithi078 ceph-mon[91136]: pgmap v6636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:01:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:10 smithi078 ceph-mon[99959]: pgmap v6636: 161 pgs: 161 active+clean; 585 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:01:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:01:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:01:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:11.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:11.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:11.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:12 smithi146 ceph-mon[103278]: pgmap v6637: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:01:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:12 smithi078 ceph-mon[91136]: pgmap v6637: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:01:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:12 smithi078 ceph-mon[99959]: pgmap v6637: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:01:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:14.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:14 smithi146 ceph-mon[103278]: pgmap v6638: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:01:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:14 smithi078 ceph-mon[91136]: pgmap v6638: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:01:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:14 smithi078 ceph-mon[99959]: pgmap v6638: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:01:16.468 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:01:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 3h - - 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 3h - - 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 5s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 1s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 1s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 5s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 5s ago 3h 393M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 1s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 5s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 5s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 1s ago 3h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:01:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 5s ago 3h 80.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 5s ago 3h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 5s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 5s ago 3h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 1s ago 3h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 1s ago 3h 77.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 1s ago 3h 81.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 1s ago 3h 77.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 1s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 1s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 5s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:01:16.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 1s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:01:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:16 smithi146 ceph-mon[103278]: pgmap v6639: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-21T00:01:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[91136]: pgmap v6639: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[99959]: pgmap v6639: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 134 op/s 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:17.302 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:01:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:01:17.572 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:01:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:01:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:01:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:01:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:01:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:01:17.573 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:01:17.883 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:01:17.883 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:17.883 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:17.883 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:17.883 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:17.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:17 smithi078 ceph-mon[99959]: from='client.59091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:17.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:17 smithi078 ceph-mon[99959]: from='client.59097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:17.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/133448026' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:17.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:17 smithi078 ceph-mon[91136]: from='client.59091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:17.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:17 smithi078 ceph-mon[91136]: from='client.59097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:17.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/133448026' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:17 smithi146 ceph-mon[103278]: from='client.59091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:17 smithi146 ceph-mon[103278]: from='client.59097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/133448026' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:18 smithi146 ceph-mon[103278]: from='client.79019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:18 smithi146 ceph-mon[103278]: pgmap v6640: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:01:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:18 smithi146 ceph-mon[103278]: from='client.79031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/900455518' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[91136]: from='client.79019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[91136]: pgmap v6640: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[91136]: from='client.79031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/900455518' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[99959]: from='client.79019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[99959]: pgmap v6640: 161 pgs: 161 active+clean; 585 KiB data, 316 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:01:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[99959]: from='client.79031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/900455518' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:01:20] "GET /metrics HTTP/1.1" 200 38988 "" "Prometheus/2.43.0" 2024-09-21T00:01:21.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:20 smithi078 ceph-mon[99959]: pgmap v6641: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:01:21.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:20 smithi078 ceph-mon[91136]: pgmap v6641: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:01:21.196 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:21.060Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:21.197 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:21.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:21.197 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:21.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:20 smithi146 ceph-mon[103278]: pgmap v6641: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:01:22.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:22.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:22 smithi078 ceph-mon[99959]: pgmap v6642: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:01:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:22 smithi078 ceph-mon[91136]: pgmap v6642: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:01:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:22 smithi146 ceph-mon[103278]: pgmap v6642: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:01:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:24 smithi146 ceph-mon[103278]: pgmap v6643: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:01:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:24 smithi078 ceph-mon[99959]: pgmap v6643: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:01:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:24 smithi078 ceph-mon[91136]: pgmap v6643: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:01:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:26 smithi146 ceph-mon[103278]: pgmap v6644: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:01:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:26 smithi078 ceph-mon[91136]: pgmap v6644: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:01:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:26 smithi078 ceph-mon[99959]: pgmap v6644: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:01:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:28 smithi146 ceph-mon[103278]: pgmap v6645: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-21T00:01:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:28 smithi078 ceph-mon[91136]: pgmap v6645: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-21T00:01:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:28 smithi078 ceph-mon[99959]: pgmap v6645: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 39 op/s 2024-09-21T00:01:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:01:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-21T00:01:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:30 smithi146 ceph-mon[103278]: pgmap v6646: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T00:01:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:30 smithi078 ceph-mon[91136]: pgmap v6646: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T00:01:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:30 smithi078 ceph-mon[99959]: pgmap v6646: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T00:01:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:31.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:31.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:31.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:33.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:32 smithi146 ceph-mon[103278]: pgmap v6647: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T00:01:33.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:32 smithi078 ceph-mon[99959]: pgmap v6647: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T00:01:33.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:32 smithi078 ceph-mon[91136]: pgmap v6647: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T00:01:35.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:34 smithi146 ceph-mon[103278]: pgmap v6648: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:34 smithi078 ceph-mon[91136]: pgmap v6648: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:34 smithi078 ceph-mon[99959]: pgmap v6648: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:36 smithi146 ceph-mon[103278]: pgmap v6649: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[99959]: pgmap v6649: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[91136]: pgmap v6649: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:38 smithi146 ceph-mon[103278]: pgmap v6650: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:38 smithi078 ceph-mon[99959]: pgmap v6650: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:38 smithi078 ceph-mon[91136]: pgmap v6650: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:01:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-21T00:01:41.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:40 smithi078 ceph-mon[99959]: pgmap v6651: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:41.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:40 smithi078 ceph-mon[91136]: pgmap v6651: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:41.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:41.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:41.184 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:41.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:41.184 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:41.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:40 smithi146 ceph-mon[103278]: pgmap v6651: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:42.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:42.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-mon[99959]: pgmap v6652: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:42.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:42.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:41 smithi078 ceph-mon[91136]: pgmap v6652: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:41 smithi146 ceph-mon[103278]: pgmap v6652: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:44 smithi146 ceph-mon[103278]: pgmap v6653: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:44 smithi078 ceph-mon[91136]: pgmap v6653: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:44 smithi078 ceph-mon[99959]: pgmap v6653: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:46 smithi146 ceph-mon[103278]: pgmap v6654: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:46 smithi078 ceph-mon[99959]: pgmap v6654: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:46 smithi078 ceph-mon[91136]: pgmap v6654: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:48.185 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:01:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:48 smithi078 ceph-mon[91136]: pgmap v6655: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:48 smithi078 ceph-mon[99959]: pgmap v6655: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 3h - - 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 3h - - 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 17s ago 3h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 12s ago 3h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 17s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:01:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 17s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 17s ago 3h 394M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 17s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 17s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 12s ago 3h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 17s ago 3h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 17s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 17s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 17s ago 3h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 12s ago 3h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 12s ago 3h 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 12s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 12s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 3h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:01:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 17s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:01:48.717 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 12s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:01:48.717 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 17s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:01:48.717 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 12s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:01:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:48 smithi146 ceph-mon[103278]: pgmap v6655: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:01:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:01:49.028 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:01:49.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:01:49.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:01:49.625 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:01:49.625 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:01:49.625 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:01:49.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[99959]: from='client.79043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[99959]: from='client.59133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[99959]: from='client.59139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/867366140' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[91136]: from='client.79043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[91136]: from='client.59133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[91136]: from='client.59139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/867366140' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:49 smithi146 ceph-mon[103278]: from='client.79043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:49 smithi146 ceph-mon[103278]: from='client.59133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:49 smithi146 ceph-mon[103278]: from='client.59139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/867366140' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:01:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:50 smithi146 ceph-mon[103278]: from='client.79067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:50 smithi146 ceph-mon[103278]: pgmap v6656: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2537719013' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:01:50] "GET /metrics HTTP/1.1" 200 38986 "" "Prometheus/2.43.0" 2024-09-21T00:01:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-mon[91136]: from='client.79067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-mon[91136]: pgmap v6656: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2537719013' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-mon[99959]: from='client.79067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:01:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-mon[99959]: pgmap v6656: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2537719013' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:01:51.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:51.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:51.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:51.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:01:51.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:01:51.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:01:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:01:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:52 smithi146 ceph-mon[103278]: pgmap v6657: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[91136]: pgmap v6657: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[99959]: pgmap v6657: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:54 smithi146 ceph-mon[103278]: pgmap v6658: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:54 smithi078 ceph-mon[91136]: pgmap v6658: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:54 smithi078 ceph-mon[99959]: pgmap v6658: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:01:56.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:56 smithi146 ceph-mon[103278]: pgmap v6659: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:56.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.693 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.694 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[91136]: pgmap v6659: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[99959]: pgmap v6659: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:01:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:58 smithi146 ceph-mon[103278]: pgmap v6660: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:58 smithi078 ceph-mon[91136]: pgmap v6660: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:01:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:58 smithi078 ceph-mon[99959]: pgmap v6660: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:01:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:00 smithi146 ceph-mon[103278]: pgmap v6661: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:00 smithi078 ceph-mon[99959]: pgmap v6661: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:00 smithi078 ceph-mon[91136]: pgmap v6661: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:02:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:02:01.375 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:01.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:01.375 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:01.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:01.375 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:01.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:01.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:01.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:02.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:02 smithi078 ceph-mon[99959]: pgmap v6662: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:02.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:02 smithi078 ceph-mon[91136]: pgmap v6662: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:02 smithi146 ceph-mon[103278]: pgmap v6662: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:04 smithi146 ceph-mon[103278]: pgmap v6663: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:04 smithi078 ceph-mon[91136]: pgmap v6663: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:04 smithi078 ceph-mon[99959]: pgmap v6663: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:06 smithi146 ceph-mon[103278]: pgmap v6664: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:06 smithi078 ceph-mon[91136]: pgmap v6664: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:06 smithi078 ceph-mon[99959]: pgmap v6664: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:08 smithi146 ceph-mon[103278]: pgmap v6665: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:08 smithi078 ceph-mon[91136]: pgmap v6665: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:08 smithi078 ceph-mon[99959]: pgmap v6665: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:10 smithi146 ceph-mon[103278]: pgmap v6666: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:10 smithi078 ceph-mon[91136]: pgmap v6666: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:10 smithi078 ceph-mon[99959]: pgmap v6666: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:02:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:02:11.519 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:11.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:11.519 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:11.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:11.519 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:11.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:12 smithi146 ceph-mon[103278]: pgmap v6667: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[91136]: pgmap v6667: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[99959]: pgmap v6667: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:14 smithi146 ceph-mon[103278]: pgmap v6668: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:14 smithi078 ceph-mon[91136]: pgmap v6668: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:14 smithi078 ceph-mon[99959]: pgmap v6668: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:16 smithi146 ceph-mon[103278]: pgmap v6669: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[91136]: pgmap v6669: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[99959]: pgmap v6669: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:18 smithi146 ceph-mon[103278]: pgmap v6670: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:18 smithi078 ceph-mon[91136]: pgmap v6670: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:18 smithi078 ceph-mon[99959]: pgmap v6670: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:19.918 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 3h - - 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 3h - - 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 8s ago 3h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 4s ago 3h 88.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 8s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 4s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 8s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:02:20.446 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 8s ago 3h 395M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 4s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 8s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 8s ago 3h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 4s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 8s ago 3h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 8s ago 3h 87.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 8s ago 3h 74.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 8s ago 3h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 4s ago 3h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 4s ago 3h 76.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 4s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 4s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 4s ago 3h 73.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 8s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 4s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:02:20.447 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 8s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:02:20.448 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 4s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:02:20.757 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:02:20.757 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:02:20.757 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:02:20.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:02:20.758 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:02:20.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:20 smithi078 ceph-mon[99959]: pgmap v6671: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:20.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:20 smithi078 ceph-mon[91136]: pgmap v6671: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:20.759 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:02:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:02:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:20 smithi146 ceph-mon[103278]: pgmap v6671: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:21.026 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:02:21.026 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:02:21.026 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:02:21.026 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:02:21.026 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:02:21.027 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:02:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:21.065Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:21.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:21.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:21.336 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:02:21.336 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:21.336 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:21.336 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:21.336 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:22.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[99959]: from='client.79079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[99959]: from='client.79085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:22.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[99959]: from='client.79091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/250090299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/180833038' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[91136]: from='client.79079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[91136]: from='client.79085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[91136]: from='client.79091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/250090299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:22.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/180833038' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:21 smithi146 ceph-mon[103278]: from='client.79079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:21 smithi146 ceph-mon[103278]: from='client.79085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:21 smithi146 ceph-mon[103278]: from='client.79091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/250090299' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/180833038' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:22 smithi146 ceph-mon[103278]: from='client.79103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:22 smithi146 ceph-mon[103278]: pgmap v6672: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:22 smithi078 ceph-mon[91136]: from='client.79103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:22 smithi078 ceph-mon[91136]: pgmap v6672: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:22 smithi078 ceph-mon[99959]: from='client.79103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:22 smithi078 ceph-mon[99959]: pgmap v6672: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:24 smithi146 ceph-mon[103278]: pgmap v6673: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:24 smithi078 ceph-mon[91136]: pgmap v6673: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:24 smithi078 ceph-mon[99959]: pgmap v6673: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:26 smithi146 ceph-mon[103278]: pgmap v6674: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:26 smithi078 ceph-mon[99959]: pgmap v6674: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:26 smithi078 ceph-mon[91136]: pgmap v6674: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:28 smithi146 ceph-mon[103278]: pgmap v6675: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:28 smithi078 ceph-mon[99959]: pgmap v6675: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:28 smithi078 ceph-mon[91136]: pgmap v6675: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:30.535 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336253]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:30.535 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336253]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:30.535 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336253]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:30.535 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336253]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:30.535 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336256]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:30.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336256]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:30.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336256]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:30.536 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 21 00:02:30 smithi078 sudo[336256]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:30 smithi146 ceph-mon[103278]: pgmap v6676: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:30 smithi078 ceph-mon[91136]: pgmap v6676: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:30 smithi078 ceph-mon[99959]: pgmap v6676: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:02:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:02:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:31.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:31.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:31.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:32 smithi146 ceph-mon[103278]: pgmap v6677: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[91136]: pgmap v6677: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[99959]: pgmap v6677: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:34 smithi146 ceph-mon[103278]: pgmap v6678: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:34 smithi078 ceph-mon[91136]: pgmap v6678: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:34 smithi078 ceph-mon[99959]: pgmap v6678: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:36 smithi146 ceph-mon[103278]: pgmap v6679: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:36.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[91136]: pgmap v6679: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[99959]: pgmap v6679: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:38 smithi146 ceph-mon[103278]: pgmap v6680: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:02:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:38 smithi078 ceph-mon[99959]: pgmap v6680: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:02:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:38 smithi078 ceph-mon[91136]: pgmap v6680: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:02:40.756 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:02:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:02:41.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:40 smithi146 ceph-mon[103278]: pgmap v6681: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:40 smithi078 ceph-mon[99959]: pgmap v6681: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:40 smithi078 ceph-mon[91136]: pgmap v6681: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:41.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:41.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:41.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:42.970 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:42 smithi078 ceph-mon[91136]: pgmap v6682: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:42.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:42 smithi078 ceph-mon[99959]: pgmap v6682: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:42 smithi146 ceph-mon[103278]: pgmap v6682: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:44.966 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:44 smithi078 ceph-mon[99959]: pgmap v6683: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:44.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:44 smithi078 ceph-mon[91136]: pgmap v6683: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:44 smithi146 ceph-mon[103278]: pgmap v6683: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:45.535 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336413]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336413]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336416]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336416]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336416]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336416]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336401]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.536 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336401]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336401]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336401]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336404]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336404]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336404]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.1.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336404]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336407]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336407]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336407]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.537 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336407]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.538 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336410]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.538 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336410]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.538 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336410]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.538 INFO:journalctl@ceph.osd.2.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336410]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280539]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280539]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280539]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280542]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280542]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280542]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.705 INFO:journalctl@ceph.osd.4.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280542]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280545]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280545]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280545]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280545]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280548]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280548]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280548]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280548]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280557]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.706 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.707 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280557]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.707 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280557]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.707 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280560]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.707 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280560]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.707 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280560]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.707 INFO:journalctl@ceph.osd.7.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280560]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280551]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280551]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280554]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280554]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280554]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:45.708 INFO:journalctl@ceph.osd.6.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280554]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280563]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280563]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280563]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280563]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280566]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280566]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:45 smithi146 sudo[280566]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336420]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336420]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336423]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336423]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336423]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336423]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336426]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336426]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336429]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T00:02:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:45 smithi078 sudo[336429]: pam_unix(sudo:session): session closed for user root 2024-09-21T00:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: pgmap v6684: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:46 smithi146 ceph-mon[103278]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: pgmap v6684: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[99959]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: pgmap v6684: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:46.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T00:02:46.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T00:02:46.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T00:02:46.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T00:02:46.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:46 smithi078 ceph-mon[91136]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T00:02:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:48 smithi146 ceph-mon[103278]: pgmap v6685: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:48 smithi078 ceph-mon[91136]: pgmap v6685: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:48 smithi078 ceph-mon[99959]: pgmap v6685: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:02:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:50 smithi146 ceph-mon[103278]: pgmap v6686: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:50 smithi078 ceph-mon[91136]: pgmap v6686: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:50 smithi078 ceph-mon[99959]: pgmap v6686: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:02:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:02:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:02:51.425 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:51.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:51.425 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:51.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:02:51.425 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:02:51.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:02:51.633 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:02:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 3h - - 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 3h - - 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 0s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 15s ago 3h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 0s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 0s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 0s ago 3h 395M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 3h 258M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 0s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:02:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 0s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 15s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 0s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 0s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 0s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 0s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 3h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 15s ago 3h 76.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 15s ago 3h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 15s ago 3h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 3h 73.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 0s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 15s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 0s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:02:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 15s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:02:52.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:02:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:02:52.492 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:02:52.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:02:52.759 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:02:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[99959]: pgmap v6687: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[99959]: from='client.79118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2029619965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[91136]: pgmap v6687: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[91136]: from='client.79118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2029619965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:53.071 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:02:53.071 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:02:53.071 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:02:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:52 smithi146 ceph-mon[103278]: pgmap v6687: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:52 smithi146 ceph-mon[103278]: from='client.79118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2029619965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:02:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:53 smithi146 ceph-mon[103278]: from='client.79124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:53 smithi146 ceph-mon[103278]: from='client.59214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:53 smithi146 ceph-mon[103278]: from='client.79142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2168698544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[91136]: from='client.79124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[91136]: from='client.59214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[91136]: from='client.79142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2168698544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[99959]: from='client.79124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[99959]: from='client.59214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[99959]: from='client.79142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:02:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2168698544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:02:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:54 smithi146 ceph-mon[103278]: pgmap v6688: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:02:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:54 smithi078 ceph-mon[99959]: pgmap v6688: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:02:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:54 smithi078 ceph-mon[91136]: pgmap v6688: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:02:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:55 smithi146 ceph-mon[103278]: pgmap v6689: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:55 smithi078 ceph-mon[91136]: pgmap v6689: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:55 smithi078 ceph-mon[99959]: pgmap v6689: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:57.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.595 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:02:58.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:58 smithi146 ceph-mon[103278]: pgmap v6690: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:58.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:02:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:58 smithi078 ceph-mon[99959]: pgmap v6690: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:02:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:02:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:58 smithi078 ceph-mon[91136]: pgmap v6690: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:02:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:02:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:00 smithi146 ceph-mon[103278]: pgmap v6691: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:03:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:00 smithi078 ceph-mon[99959]: pgmap v6691: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:03:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:00 smithi078 ceph-mon[91136]: pgmap v6691: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 9 op/s 2024-09-21T00:03:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:03:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:03:01.473 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:01.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:01.473 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:01.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:01.473 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:01.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:02 smithi146 ceph-mon[103278]: pgmap v6692: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:03:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:02 smithi078 ceph-mon[91136]: pgmap v6692: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:03:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:02 smithi078 ceph-mon[99959]: pgmap v6692: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 87 KiB/s wr, 8 op/s 2024-09-21T00:03:04.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:04 smithi078 ceph-mon[91136]: pgmap v6693: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:04.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:04 smithi078 ceph-mon[99959]: pgmap v6693: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:04 smithi146 ceph-mon[103278]: pgmap v6693: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:06 smithi146 ceph-mon[103278]: pgmap v6694: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:06 smithi078 ceph-mon[91136]: pgmap v6694: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:06 smithi078 ceph-mon[99959]: pgmap v6694: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:08 smithi146 ceph-mon[103278]: pgmap v6695: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:08 smithi078 ceph-mon[91136]: pgmap v6695: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:08 smithi078 ceph-mon[99959]: pgmap v6695: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:10 smithi146 ceph-mon[103278]: pgmap v6696: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:10 smithi078 ceph-mon[91136]: pgmap v6696: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:10 smithi078 ceph-mon[99959]: pgmap v6696: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:03:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:03:11.491 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:11.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:11.491 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:11.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:11.491 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:11.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:12 smithi146 ceph-mon[103278]: pgmap v6697: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[91136]: pgmap v6697: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[99959]: pgmap v6697: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:14 smithi146 ceph-mon[103278]: pgmap v6698: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:14 smithi078 ceph-mon[91136]: pgmap v6698: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:14 smithi078 ceph-mon[99959]: pgmap v6698: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:16 smithi146 ceph-mon[103278]: pgmap v6699: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:16 smithi078 ceph-mon[91136]: pgmap v6699: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:16 smithi078 ceph-mon[99959]: pgmap v6699: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:18 smithi146 ceph-mon[103278]: pgmap v6700: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:18 smithi078 ceph-mon[91136]: pgmap v6700: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:18 smithi078 ceph-mon[99959]: pgmap v6700: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:20 smithi078 ceph-mon[91136]: pgmap v6701: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:20 smithi078 ceph-mon[99959]: pgmap v6701: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:03:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:03:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:20 smithi146 ceph-mon[103278]: pgmap v6701: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:21.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:21.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:21.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:21.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:21.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:21.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:22 smithi146 ceph-mon[103278]: pgmap v6702: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:22 smithi078 ceph-mon[91136]: pgmap v6702: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:22 smithi078 ceph-mon[99959]: pgmap v6702: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:23.359 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:03:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:03:23.891 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 3h - - 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 3h - - 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 11s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 7s ago 3h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 11s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 7s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 11s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 11s ago 3h 396M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 7s ago 3h 256M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 11s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 11s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 7s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 11s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 11s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 11s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:03:23.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 11s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 7s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 7s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 7s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 7s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 7s ago 3h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 11s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 7s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 11s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:03:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 7s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:03:24.484 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:03:24.484 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:03:24.484 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:03:24.484 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:03:24.484 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:03:24.484 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:03:24.485 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:03:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:03:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:24.796 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:25.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[99959]: from='client.79154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:25.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[99959]: pgmap v6703: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:25.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[99959]: from='client.59244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:25.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/129889979' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:25.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3795028971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:25.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[91136]: from='client.79154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:25.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[91136]: pgmap v6703: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:25.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[91136]: from='client.59244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:25.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/129889979' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:25.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3795028971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:24 smithi146 ceph-mon[103278]: from='client.79154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:24 smithi146 ceph-mon[103278]: pgmap v6703: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:24 smithi146 ceph-mon[103278]: from='client.59244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/129889979' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3795028971' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:26.165 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:25 smithi078 ceph-mon[99959]: from='client.59250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:26.165 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:25 smithi078 ceph-mon[99959]: from='client.79178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:26.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:25 smithi078 ceph-mon[91136]: from='client.59250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:26.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:25 smithi078 ceph-mon[91136]: from='client.79178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:25 smithi146 ceph-mon[103278]: from='client.59250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:25 smithi146 ceph-mon[103278]: from='client.79178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:26 smithi146 ceph-mon[103278]: pgmap v6704: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:26 smithi078 ceph-mon[91136]: pgmap v6704: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:26 smithi078 ceph-mon[99959]: pgmap v6704: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:28 smithi146 ceph-mon[103278]: pgmap v6705: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:28 smithi078 ceph-mon[91136]: pgmap v6705: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:28 smithi078 ceph-mon[99959]: pgmap v6705: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:03:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:03:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:30 smithi146 ceph-mon[103278]: pgmap v6706: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:30 smithi078 ceph-mon[91136]: pgmap v6706: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:30 smithi078 ceph-mon[99959]: pgmap v6706: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:31.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:31.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:31.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:32.137 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:32.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:32 smithi146 ceph-mon[103278]: pgmap v6707: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[91136]: pgmap v6707: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[99959]: pgmap v6707: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:34 smithi146 ceph-mon[103278]: pgmap v6708: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:34 smithi078 ceph-mon[91136]: pgmap v6708: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:34 smithi078 ceph-mon[99959]: pgmap v6708: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:36 smithi146 ceph-mon[103278]: pgmap v6709: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[91136]: pgmap v6709: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[99959]: pgmap v6709: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:38 smithi146 ceph-mon[103278]: pgmap v6710: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:38 smithi078 ceph-mon[99959]: pgmap v6710: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:38 smithi078 ceph-mon[91136]: pgmap v6710: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:03:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:03:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:40 smithi146 ceph-mon[103278]: pgmap v6711: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:40 smithi078 ceph-mon[91136]: pgmap v6711: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:40 smithi078 ceph-mon[99959]: pgmap v6711: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:41.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:41.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:41.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:42 smithi146 ceph-mon[103278]: pgmap v6712: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:42 smithi078 ceph-mon[91136]: pgmap v6712: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:42 smithi078 ceph-mon[99959]: pgmap v6712: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:45.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:44 smithi078 ceph-mon[99959]: pgmap v6713: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:45.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:44 smithi078 ceph-mon[91136]: pgmap v6713: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:44 smithi146 ceph-mon[103278]: pgmap v6713: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:46 smithi078 ceph-mon[91136]: pgmap v6714: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:46 smithi078 ceph-mon[99959]: pgmap v6714: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:46 smithi146 ceph-mon[103278]: pgmap v6714: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:48 smithi146 ceph-mon[103278]: pgmap v6715: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:48 smithi078 ceph-mon[91136]: pgmap v6715: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:48 smithi078 ceph-mon[99959]: pgmap v6715: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:03:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:03:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:50 smithi146 ceph-mon[103278]: pgmap v6716: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:50 smithi078 ceph-mon[91136]: pgmap v6716: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:50 smithi078 ceph-mon[99959]: pgmap v6716: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:51.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:51.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:03:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:03:51.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:03:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:03:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:52 smithi146 ceph-mon[103278]: pgmap v6717: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[91136]: pgmap v6717: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[99959]: pgmap v6717: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:53 smithi146 ceph-mon[103278]: pgmap v6718: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:53 smithi078 ceph-mon[91136]: pgmap v6718: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:53 smithi078 ceph-mon[99959]: pgmap v6718: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:03:55.094 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:03:55.626 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:03:55.626 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 3h - - 2024-09-21T00:03:55.626 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 3h - - 2024-09-21T00:03:55.626 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 3s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:03:55.626 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 18s ago 3h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:03:55.626 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 3s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 18s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 3s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 3s ago 3h 396M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 18s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 3s ago 3h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 3s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 18s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 3s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 3s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 3s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 3s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 18s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 18s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 18s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 18s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:03:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 18s ago 3h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:03:55.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 3s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:03:55.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 18s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:03:55.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 3s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:03:55.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 18s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:03:55.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:03:56.203 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:03:56.204 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:03:56.515 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:03:56.515 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:03:56.516 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[99959]: from='client.79190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[99959]: from='client.79196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[99959]: pgmap v6719: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[99959]: from='client.79202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/705835071' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[91136]: from='client.79190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[91136]: from='client.79196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[91136]: pgmap v6719: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:56.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[91136]: from='client.79202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/705835071' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:56 smithi146 ceph-mon[103278]: from='client.79190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:56 smithi146 ceph-mon[103278]: from='client.79196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:56 smithi146 ceph-mon[103278]: pgmap v6719: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:56 smithi146 ceph-mon[103278]: from='client.79202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/705835071' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:03:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:57 smithi146 ceph-mon[103278]: from='client.79214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3604496270' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[91136]: from='client.79214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3604496270' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[99959]: from='client.79214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:03:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3604496270' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:03:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:03:59.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:58 smithi146 ceph-mon[103278]: pgmap v6720: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:59.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:58 smithi078 ceph-mon[91136]: pgmap v6720: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:03:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:58 smithi078 ceph-mon[99959]: pgmap v6720: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:03:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:00.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:03:59 smithi146 ceph-mon[103278]: pgmap v6721: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:03:59 smithi078 ceph-mon[91136]: pgmap v6721: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:03:59 smithi078 ceph-mon[99959]: pgmap v6721: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:04:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:04:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:01.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:01.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:01.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:01.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:01.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:01.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:01 smithi146 ceph-mon[103278]: pgmap v6722: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:01 smithi078 ceph-mon[91136]: pgmap v6722: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:01 smithi078 ceph-mon[99959]: pgmap v6722: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:04 smithi146 ceph-mon[103278]: pgmap v6723: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:04 smithi078 ceph-mon[91136]: pgmap v6723: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:04 smithi078 ceph-mon[99959]: pgmap v6723: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:06 smithi146 ceph-mon[103278]: pgmap v6724: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:06.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:06 smithi078 ceph-mon[99959]: pgmap v6724: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:06.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:06 smithi078 ceph-mon[91136]: pgmap v6724: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:08 smithi078 ceph-mon[91136]: pgmap v6725: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:08 smithi078 ceph-mon[99959]: pgmap v6725: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:08 smithi146 ceph-mon[103278]: pgmap v6725: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:10 smithi146 ceph-mon[103278]: pgmap v6726: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:10 smithi078 ceph-mon[91136]: pgmap v6726: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:10 smithi078 ceph-mon[99959]: pgmap v6726: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:04:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:04:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:11.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:11.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:11.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:12 smithi146 ceph-mon[103278]: pgmap v6727: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[99959]: pgmap v6727: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[91136]: pgmap v6727: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:14 smithi146 ceph-mon[103278]: pgmap v6728: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:14 smithi078 ceph-mon[91136]: pgmap v6728: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:14 smithi078 ceph-mon[99959]: pgmap v6728: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:16 smithi146 ceph-mon[103278]: pgmap v6729: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:16 smithi078 ceph-mon[91136]: pgmap v6729: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:16 smithi078 ceph-mon[99959]: pgmap v6729: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:17 smithi146 ceph-mon[103278]: pgmap v6730: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[91136]: pgmap v6730: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:17 smithi078 ceph-mon[99959]: pgmap v6730: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:20 smithi146 ceph-mon[103278]: pgmap v6731: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:20 smithi078 ceph-mon[91136]: pgmap v6731: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:04:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:04:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:20 smithi078 ceph-mon[99959]: pgmap v6731: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:21.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:21.073Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:21.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:21.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:21.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:21.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:22 smithi146 ceph-mon[103278]: pgmap v6732: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:22 smithi078 ceph-mon[99959]: pgmap v6732: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:22 smithi078 ceph-mon[91136]: pgmap v6732: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:24 smithi146 ceph-mon[103278]: pgmap v6733: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:24 smithi078 ceph-mon[91136]: pgmap v6733: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:24 smithi078 ceph-mon[99959]: pgmap v6733: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 00:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T00:04:25.848994891Z level=info msg="Completed cleanup jobs" duration=12.978141ms 2024-09-21T00:04:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:04:27.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:26 smithi078 ceph-mon[99959]: pgmap v6734: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:27.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:26 smithi078 ceph-mon[91136]: pgmap v6734: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:26 smithi146 ceph-mon[103278]: pgmap v6734: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:04:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 3h - - 2024-09-21T00:04:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-21T00:04:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 14s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 14s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 14s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 14s ago 3h 397M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:04:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 14s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 14s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 14s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 14s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 14s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 14s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:04:27.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 10s ago 3h 68.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 14s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 14s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:04:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:04:27.674 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:04:28.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:04:28.285 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:04:28.286 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:27 smithi146 ceph-mon[103278]: from='client.59310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:28.286 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1933890586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:04:28.286 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:04:28.287 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:04:28.287 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:04:28.287 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:04:28.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:04:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:27 smithi078 ceph-mon[99959]: from='client.59310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1933890586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:27 smithi078 ceph-mon[91136]: from='client.59310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1933890586' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:04:28.288 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:04:28.288 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:04:28.288 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:04:28.288 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:04:28.288 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[91136]: from='client.59316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[91136]: from='client.59322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[91136]: pgmap v6735: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/469684522' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[99959]: from='client.59316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[99959]: from='client.59322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[99959]: pgmap v6735: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/469684522' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:28 smithi146 ceph-mon[103278]: from='client.59316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:28 smithi146 ceph-mon[103278]: from='client.59322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:28 smithi146 ceph-mon[103278]: pgmap v6735: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/469684522' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:04:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:29 smithi146 ceph-mon[103278]: from='client.79250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:29 smithi078 ceph-mon[99959]: from='client.79250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:29 smithi078 ceph-mon[91136]: from='client.79250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:30.835 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:04:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:04:31.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:30 smithi078 ceph-mon[99959]: pgmap v6736: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:31.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:30 smithi078 ceph-mon[91136]: pgmap v6736: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:31.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:31.074Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:31.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:31.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:31.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:31.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:30 smithi146 ceph-mon[103278]: pgmap v6736: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:32 smithi146 ceph-mon[103278]: pgmap v6737: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[99959]: pgmap v6737: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:33.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.134 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[91136]: pgmap v6737: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:33.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:33.135 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:34 smithi146 ceph-mon[103278]: pgmap v6738: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:34 smithi078 ceph-mon[91136]: pgmap v6738: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:34 smithi078 ceph-mon[99959]: pgmap v6738: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:37.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:36 smithi146 ceph-mon[103278]: pgmap v6739: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:36 smithi078 ceph-mon[91136]: pgmap v6739: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:36 smithi078 ceph-mon[99959]: pgmap v6739: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:38 smithi146 ceph-mon[103278]: pgmap v6740: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[91136]: pgmap v6740: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:38 smithi078 ceph-mon[99959]: pgmap v6740: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:40.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:40 smithi146 ceph-mon[103278]: pgmap v6741: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:40 smithi078 ceph-mon[91136]: pgmap v6741: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:40 smithi078 ceph-mon[99959]: pgmap v6741: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:04:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:04:41.458 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:41.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:41.458 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:41.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:41.458 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:41.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:41.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:42 smithi146 ceph-mon[103278]: pgmap v6742: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:42 smithi078 ceph-mon[91136]: pgmap v6742: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:42 smithi078 ceph-mon[99959]: pgmap v6742: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:44 smithi146 ceph-mon[103278]: pgmap v6743: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:44 smithi078 ceph-mon[91136]: pgmap v6743: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:44 smithi078 ceph-mon[99959]: pgmap v6743: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:46 smithi146 ceph-mon[103278]: pgmap v6744: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:46 smithi078 ceph-mon[91136]: pgmap v6744: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:46 smithi078 ceph-mon[99959]: pgmap v6744: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:48 smithi146 ceph-mon[103278]: pgmap v6745: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:48 smithi078 ceph-mon[91136]: pgmap v6745: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:48 smithi078 ceph-mon[99959]: pgmap v6745: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:50 smithi146 ceph-mon[103278]: pgmap v6746: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:50 smithi078 ceph-mon[99959]: pgmap v6746: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:50 smithi078 ceph-mon[91136]: pgmap v6746: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:04:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:04:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:51.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:51.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:04:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:04:51.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:04:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:04:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:52 smithi146 ceph-mon[103278]: pgmap v6747: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:52 smithi078 ceph-mon[99959]: pgmap v6747: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:52 smithi078 ceph-mon[91136]: pgmap v6747: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:54 smithi146 ceph-mon[103278]: pgmap v6748: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:54 smithi078 ceph-mon[91136]: pgmap v6748: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:54 smithi078 ceph-mon[99959]: pgmap v6748: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:04:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:56 smithi146 ceph-mon[103278]: pgmap v6749: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:56 smithi078 ceph-mon[91136]: pgmap v6749: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:56 smithi078 ceph-mon[99959]: pgmap v6749: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:58 smithi146 ceph-mon[103278]: pgmap v6750: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:58.562 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[99959]: pgmap v6750: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:04:58.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:58 smithi078 ceph-mon[91136]: pgmap v6750: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:04:58.586 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 3h - - 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 3h - - 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 6s ago 3h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 1s ago 3h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 6s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 1s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 6s ago 3h 578M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:04:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 6s ago 3h 397M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 1s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 6s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 6s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 1s ago 3h 16.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 6s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 6s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 6s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 6s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 1s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 1s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 1s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 1s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 1s ago 3h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 6s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:04:59.117 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 1s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:04:59.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 6s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:04:59.118 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 1s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:04:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:04:59.429 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:04:59.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:59.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:59 smithi078 ceph-mon[91136]: from='client.79262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:59.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:04:59 smithi078 ceph-mon[91136]: from='client.59352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:59.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:59.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:59 smithi078 ceph-mon[99959]: from='client.79262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:59.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:04:59 smithi078 ceph-mon[99959]: from='client.59352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:04:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:59 smithi146 ceph-mon[103278]: from='client.79262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:04:59 smithi146 ceph-mon[103278]: from='client.59352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:04:59.693 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:04:59.694 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:04:59.694 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:05:00.001 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:05:00.001 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:00.001 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:00.001 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:00.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:00 smithi146 ceph-mon[103278]: from='client.79274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:00.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:00 smithi146 ceph-mon[103278]: pgmap v6751: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:00.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2714255983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:00.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:00 smithi146 ceph-mon[103278]: from='client.79286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:00.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2676667781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[91136]: from='client.79274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[91136]: pgmap v6751: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2714255983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[91136]: from='client.79286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2676667781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[99959]: from='client.79274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[99959]: pgmap v6751: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2714255983' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[99959]: from='client.79286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2676667781' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:00.952 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:05:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:05:01.268 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:01.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:01.268 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:01.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:01.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:01.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:01.401 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:02.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:02 smithi146 ceph-mon[103278]: pgmap v6752: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:02 smithi078 ceph-mon[91136]: pgmap v6752: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:02 smithi078 ceph-mon[99959]: pgmap v6752: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:04 smithi146 ceph-mon[103278]: pgmap v6753: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:04 smithi078 ceph-mon[91136]: pgmap v6753: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:04 smithi078 ceph-mon[99959]: pgmap v6753: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:06 smithi146 ceph-mon[103278]: pgmap v6754: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:06 smithi078 ceph-mon[91136]: pgmap v6754: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:06 smithi078 ceph-mon[99959]: pgmap v6754: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:08.653 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:08 smithi078 ceph-mon[99959]: pgmap v6755: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:08.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:08 smithi078 ceph-mon[91136]: pgmap v6755: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:08 smithi146 ceph-mon[103278]: pgmap v6755: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:10 smithi146 ceph-mon[103278]: pgmap v6756: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:10 smithi078 ceph-mon[91136]: pgmap v6756: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:05:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:05:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:10 smithi078 ceph-mon[99959]: pgmap v6756: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:11.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:11.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:11.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:11.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:11.478 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:11.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:12.761 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:12 smithi078 ceph-mon[99959]: pgmap v6757: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:12.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:12 smithi078 ceph-mon[91136]: pgmap v6757: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:12 smithi146 ceph-mon[103278]: pgmap v6757: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:14 smithi146 ceph-mon[103278]: pgmap v6758: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:14 smithi078 ceph-mon[99959]: pgmap v6758: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:14 smithi078 ceph-mon[91136]: pgmap v6758: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:16 smithi078 ceph-mon[91136]: pgmap v6759: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:16 smithi078 ceph-mon[99959]: pgmap v6759: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:16 smithi146 ceph-mon[103278]: pgmap v6759: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:18 smithi146 ceph-mon[103278]: pgmap v6760: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[99959]: pgmap v6760: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:18.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:18 smithi078 ceph-mon[91136]: pgmap v6760: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:20.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:20 smithi146 ceph-mon[103278]: pgmap v6761: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:20 smithi078 ceph-mon[91136]: pgmap v6761: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:20 smithi078 ceph-mon[99959]: pgmap v6761: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:05:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:05:21.474 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:21.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:21.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:21.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:21.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:21.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:21.649 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:22.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:22 smithi146 ceph-mon[103278]: pgmap v6762: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:22 smithi078 ceph-mon[91136]: pgmap v6762: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:22 smithi078 ceph-mon[99959]: pgmap v6762: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:24 smithi146 ceph-mon[103278]: pgmap v6763: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:24 smithi078 ceph-mon[91136]: pgmap v6763: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:24 smithi078 ceph-mon[99959]: pgmap v6763: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:26 smithi146 ceph-mon[103278]: pgmap v6764: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:26 smithi078 ceph-mon[91136]: pgmap v6764: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:26 smithi078 ceph-mon[99959]: pgmap v6764: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:28.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:28 smithi078 ceph-mon[99959]: pgmap v6765: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:28.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:28 smithi078 ceph-mon[91136]: pgmap v6765: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:28.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:28 smithi146 ceph-mon[103278]: pgmap v6765: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:30.298 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:05:30.560 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:30 smithi078 ceph-mon[99959]: pgmap v6766: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:30.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:30 smithi078 ceph-mon[91136]: pgmap v6766: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:30 smithi146 ceph-mon[103278]: pgmap v6766: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:30.835 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:05:30.835 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 3h - - 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 3h - - 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 18s ago 3h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 13s ago 3h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 18s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 13s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 18s ago 3h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 18s ago 3h 397M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 13s ago 3h 257M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 18s ago 3h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 18s ago 3h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 13s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 18s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 18s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:05:30.836 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 13s ago 3h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 13s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 13s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 13s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 13s ago 3h 71.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 13s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:05:30.837 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 13s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:05:30.837 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:05:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:05:31.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:05:31.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:05:31.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:05:31.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:05:31.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:05:31.145 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:05:31.146 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:31.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:31.146 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:31.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:31.146 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:31.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:31.414 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:05:31.414 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:05:31.414 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:05:31.415 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:05:31.724 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:05:31.724 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:31.724 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:05:31.725 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:31.725 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[99959]: from='client.79298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[99959]: from='client.59388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[99959]: from='client.59394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/98596920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[91136]: from='client.79298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[91136]: from='client.59388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:31.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[91136]: from='client.59394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.726 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/98596920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:31 smithi146 ceph-mon[103278]: from='client.79298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:31 smithi146 ceph-mon[103278]: from='client.59388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:31 smithi146 ceph-mon[103278]: from='client.59394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/98596920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:05:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:32 smithi146 ceph-mon[103278]: from='client.59406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:32 smithi146 ceph-mon[103278]: pgmap v6767: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1693809456' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:32 smithi078 ceph-mon[99959]: from='client.59406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:32 smithi078 ceph-mon[99959]: pgmap v6767: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1693809456' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:32 smithi078 ceph-mon[91136]: from='client.59406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:05:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:32 smithi078 ceph-mon[91136]: pgmap v6767: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1693809456' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:05:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:34 smithi146 ceph-mon[103278]: pgmap v6768: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:34 smithi078 ceph-mon[91136]: pgmap v6768: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:34 smithi078 ceph-mon[99959]: pgmap v6768: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:35 smithi146 ceph-mon[103278]: pgmap v6769: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:35 smithi078 ceph-mon[91136]: pgmap v6769: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:35 smithi078 ceph-mon[99959]: pgmap v6769: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:38 smithi146 ceph-mon[103278]: pgmap v6770: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[99959]: pgmap v6770: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[91136]: pgmap v6770: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:40 smithi146 ceph-mon[103278]: pgmap v6771: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:40 smithi078 ceph-mon[99959]: pgmap v6771: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:40 smithi078 ceph-mon[91136]: pgmap v6771: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:05:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:05:41.517 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:41.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:41.517 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:41.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:41.518 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:41.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:42 smithi146 ceph-mon[103278]: pgmap v6772: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:42 smithi078 ceph-mon[91136]: pgmap v6772: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:42 smithi078 ceph-mon[99959]: pgmap v6772: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:43.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:44 smithi078 ceph-mon[91136]: pgmap v6773: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:44 smithi078 ceph-mon[99959]: pgmap v6773: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:44 smithi146 ceph-mon[103278]: pgmap v6773: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:46 smithi078 ceph-mon[91136]: pgmap v6774: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:46 smithi078 ceph-mon[99959]: pgmap v6774: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:46 smithi146 ceph-mon[103278]: pgmap v6774: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:48 smithi078 ceph-mon[91136]: pgmap v6775: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:48 smithi078 ceph-mon[99959]: pgmap v6775: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:48 smithi146 ceph-mon[103278]: pgmap v6775: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:50 smithi146 ceph-mon[103278]: pgmap v6776: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:50 smithi078 ceph-mon[91136]: pgmap v6776: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:05:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:05:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:50 smithi078 ceph-mon[99959]: pgmap v6776: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:51.311 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:51.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:51.311 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:51.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:05:51.311 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:05:51.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:05:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:05:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:52 smithi078 ceph-mon[99959]: pgmap v6777: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:05:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:52 smithi078 ceph-mon[91136]: pgmap v6777: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:05:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:52 smithi146 ceph-mon[103278]: pgmap v6777: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:05:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:53 smithi146 ceph-mon[103278]: pgmap v6778: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[91136]: pgmap v6778: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:53 smithi078 ceph-mon[99959]: pgmap v6778: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:05:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:56 smithi146 ceph-mon[103278]: pgmap v6779: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:56 smithi078 ceph-mon[91136]: pgmap v6779: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:56 smithi078 ceph-mon[99959]: pgmap v6779: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:58 smithi146 ceph-mon[103278]: pgmap v6780: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:05:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[91136]: pgmap v6780: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:05:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[99959]: pgmap v6780: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:05:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:05:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:05:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:00 smithi146 ceph-mon[103278]: pgmap v6781: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:00 smithi078 ceph-mon[91136]: pgmap v6781: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:00 smithi078 ceph-mon[99959]: pgmap v6781: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:06:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:06:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:01.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:01.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:01.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:01.947 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:02.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:02.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:02.018 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 3h - - 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 3h - - 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 9s ago 3h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 4s ago 3h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 9s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 4s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 9s ago 3h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 9s ago 3h 400M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 4s ago 3h 259M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 9s ago 3h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 9s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:06:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 4s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 9s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 9s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 9s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 9s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 4s ago 3h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 4s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 4s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 4s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 4s ago 3h 68.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 9s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 4s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 9s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:06:02.541 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 4s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:06:02.873 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:02 smithi078 ceph-mon[99959]: pgmap v6782: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:02.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:02 smithi078 ceph-mon[91136]: pgmap v6782: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:02.874 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:06:02.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:06:02.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:06:02.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:02.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:06:02.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:06:02.875 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:06:02.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:02 smithi146 ceph-mon[103278]: pgmap v6782: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:03.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:06:03.144 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:06:03.144 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:06:03.144 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:06:03.144 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:06:03.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:06:03.145 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:06:03.457 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:06:03.573 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:06:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:06:03.574 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:06:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[91136]: from='client.59418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[91136]: from='client.59424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[91136]: from='client.59430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4286219053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/189537215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[99959]: from='client.59418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[99959]: from='client.59424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[99959]: from='client.59430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4286219053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:03.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/189537215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:03.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:03 smithi146 ceph-mon[103278]: from='client.59418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:03 smithi146 ceph-mon[103278]: from='client.59424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:03 smithi146 ceph-mon[103278]: from='client.59430 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:03.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4286219053' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:03.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/189537215' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:04 smithi146 ceph-mon[103278]: from='client.79358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:04 smithi146 ceph-mon[103278]: pgmap v6783: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:04 smithi078 ceph-mon[99959]: from='client.79358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:04 smithi078 ceph-mon[99959]: pgmap v6783: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:04 smithi078 ceph-mon[91136]: from='client.79358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:04 smithi078 ceph-mon[91136]: pgmap v6783: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:06 smithi146 ceph-mon[103278]: pgmap v6784: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:06 smithi078 ceph-mon[91136]: pgmap v6784: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:06 smithi078 ceph-mon[99959]: pgmap v6784: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:08 smithi146 ceph-mon[103278]: pgmap v6785: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:08 smithi078 ceph-mon[91136]: pgmap v6785: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:08 smithi078 ceph-mon[99959]: pgmap v6785: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:10 smithi078 ceph-mon[91136]: pgmap v6786: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:06:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:06:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:10 smithi078 ceph-mon[99959]: pgmap v6786: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:10 smithi146 ceph-mon[103278]: pgmap v6786: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:11.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:11.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:11.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:11.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:11.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:12 smithi078 ceph-mon[91136]: pgmap v6787: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:12 smithi078 ceph-mon[99959]: pgmap v6787: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:12 smithi146 ceph-mon[103278]: pgmap v6787: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:14.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:14.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:14 smithi146 ceph-mon[103278]: pgmap v6788: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[91136]: pgmap v6788: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:14 smithi078 ceph-mon[99959]: pgmap v6788: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:16 smithi146 ceph-mon[103278]: pgmap v6789: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:16 smithi078 ceph-mon[91136]: pgmap v6789: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:16 smithi078 ceph-mon[99959]: pgmap v6789: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[91136]: pgmap v6790: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[99959]: pgmap v6790: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:18 smithi146 ceph-mon[103278]: pgmap v6790: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:20 smithi078 ceph-mon[91136]: pgmap v6791: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:20 smithi078 ceph-mon[99959]: pgmap v6791: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:06:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:06:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:20 smithi146 ceph-mon[103278]: pgmap v6791: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:21.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:21.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:21.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:22.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:22.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:22 smithi146 ceph-mon[103278]: pgmap v6792: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:22 smithi078 ceph-mon[91136]: pgmap v6792: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:22 smithi078 ceph-mon[99959]: pgmap v6792: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:24 smithi146 ceph-mon[103278]: pgmap v6793: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:24 smithi078 ceph-mon[91136]: pgmap v6793: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:24 smithi078 ceph-mon[99959]: pgmap v6793: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:26 smithi146 ceph-mon[103278]: pgmap v6794: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:26 smithi078 ceph-mon[91136]: pgmap v6794: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:26 smithi078 ceph-mon[99959]: pgmap v6794: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:28 smithi146 ceph-mon[103278]: pgmap v6795: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:28 smithi078 ceph-mon[91136]: pgmap v6795: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:28 smithi078 ceph-mon[99959]: pgmap v6795: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:30.758 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:06:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:06:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:30 smithi078 ceph-mon[91136]: pgmap v6796: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:30 smithi078 ceph-mon[99959]: pgmap v6796: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:30 smithi146 ceph-mon[103278]: pgmap v6796: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:31.358 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:31.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:31.359 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:31.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:31.359 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:31.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:31.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:31.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:32 smithi146 ceph-mon[103278]: pgmap v6797: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:32 smithi078 ceph-mon[91136]: pgmap v6797: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:32 smithi078 ceph-mon[99959]: pgmap v6797: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:33.744 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:06:34.273 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:06:34.273 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 3h - - 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 3h - - 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 1s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 16s ago 3h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 1s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 16s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 1s ago 3h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 1s ago 3h 401M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 16s ago 3h 260M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 1s ago 3h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 1s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 16s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 1s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:06:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 1s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 1s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 1s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 16s ago 3h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 16s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 16s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 16s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 16s ago 3h 68.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 16s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:06:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 16s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:06:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:34 smithi146 ceph-mon[103278]: pgmap v6798: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:34.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:34 smithi146 ceph-mon[103278]: from='client.59454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[91136]: pgmap v6798: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[91136]: from='client.59454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.573 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.574 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:34.574 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[99959]: pgmap v6798: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:34.574 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:34 smithi078 ceph-mon[99959]: from='client.59454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:34.583 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:06:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:06:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:06:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:06:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:06:34.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:06:34.584 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:06:34.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:06:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:06:34.867 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:06:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:06:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:06:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:06:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:06:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:06:34.869 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:06:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:06:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:06:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:06:34.870 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:06:34.870 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:06:35.164 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:06:35.164 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:06:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:06:35.164 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:06:35.164 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:35 smithi146 ceph-mon[103278]: from='client.59460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:35 smithi146 ceph-mon[103278]: from='client.79382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2837748720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:35 smithi146 ceph-mon[103278]: from='client.79394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2557796131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[91136]: from='client.59460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[91136]: from='client.79382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2837748720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[91136]: from='client.79394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2557796131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[99959]: from='client.59460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[99959]: from='client.79382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2837748720' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[99959]: from='client.79394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:06:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2557796131' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:06:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:36 smithi146 ceph-mon[103278]: pgmap v6799: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:36 smithi078 ceph-mon[91136]: pgmap v6799: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:36 smithi078 ceph-mon[99959]: pgmap v6799: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:38 smithi146 ceph-mon[103278]: pgmap v6800: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[91136]: pgmap v6800: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[99959]: pgmap v6800: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:40 smithi146 ceph-mon[103278]: pgmap v6801: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:40 smithi078 ceph-mon[91136]: pgmap v6801: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:40 smithi078 ceph-mon[99959]: pgmap v6801: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:06:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:06:41.504 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:41.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:41.504 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:41.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:41.504 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:41.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:42 smithi146 ceph-mon[103278]: pgmap v6802: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:42 smithi078 ceph-mon[91136]: pgmap v6802: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:42 smithi078 ceph-mon[99959]: pgmap v6802: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:44.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:44 smithi146 ceph-mon[103278]: pgmap v6803: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:44 smithi078 ceph-mon[91136]: pgmap v6803: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:44 smithi078 ceph-mon[99959]: pgmap v6803: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:46 smithi078 ceph-mon[91136]: pgmap v6804: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:46 smithi078 ceph-mon[99959]: pgmap v6804: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:46.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:46 smithi146 ceph-mon[103278]: pgmap v6804: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:48 smithi078 ceph-mon[91136]: pgmap v6805: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:48 smithi078 ceph-mon[99959]: pgmap v6805: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:48 smithi146 ceph-mon[103278]: pgmap v6805: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:50 smithi078 ceph-mon[99959]: pgmap v6806: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:50 smithi078 ceph-mon[91136]: pgmap v6806: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:06:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:06:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:50 smithi146 ceph-mon[103278]: pgmap v6806: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:51.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:51.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:51.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:51.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:06:51.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:06:51.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:06:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:06:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:52 smithi078 ceph-mon[91136]: pgmap v6807: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:52 smithi078 ceph-mon[99959]: pgmap v6807: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:52 smithi146 ceph-mon[103278]: pgmap v6807: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:54 smithi146 ceph-mon[103278]: pgmap v6808: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[91136]: pgmap v6808: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:54 smithi078 ceph-mon[99959]: pgmap v6808: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:06:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:56 smithi146 ceph-mon[103278]: pgmap v6809: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:56 smithi078 ceph-mon[91136]: pgmap v6809: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:56 smithi078 ceph-mon[99959]: pgmap v6809: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:58 smithi146 ceph-mon[103278]: pgmap v6810: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[91136]: pgmap v6810: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:06:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[99959]: pgmap v6810: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:06:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:06:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:00.769 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:00 smithi146 ceph-mon[103278]: pgmap v6811: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:00 smithi078 ceph-mon[91136]: pgmap v6811: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:00 smithi078 ceph-mon[99959]: pgmap v6811: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:00.787 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:07:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:07:01.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:01.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:01.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:01.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:01.513 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:01.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:02 smithi146 ceph-mon[103278]: pgmap v6812: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:02 smithi078 ceph-mon[91136]: pgmap v6812: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:02 smithi078 ceph-mon[99959]: pgmap v6812: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:04.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:04 smithi146 ceph-mon[103278]: pgmap v6813: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:04 smithi078 ceph-mon[91136]: pgmap v6813: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:04 smithi078 ceph-mon[99959]: pgmap v6813: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:05.460 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 3h - - 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 3h - - 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 12s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 7s ago 3h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 12s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 7s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 12s ago 3h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 12s ago 3h 401M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:07:05.991 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 7s ago 3h 261M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 12s ago 3h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 12s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 7s ago 3h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 12s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 12s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 12s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 12s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 7s ago 3h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 7s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 7s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 7s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 7s ago 3h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 12s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:07:05.992 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 7s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:07:05.993 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 12s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:07:05.993 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 7s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:07:06.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:07:06.304 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:07:06.576 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:07:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:07:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:07:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:07:06.577 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:07:06.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[91136]: from='client.59487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[91136]: pgmap v6814: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[91136]: from='client.59493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/757204588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[99959]: from='client.59487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[99959]: pgmap v6814: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[99959]: from='client.59493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:06.578 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/757204588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:06.885 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:07:06.886 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:06.886 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:06.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:07.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:06 smithi146 ceph-mon[103278]: from='client.59487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:07.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:06 smithi146 ceph-mon[103278]: pgmap v6814: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:07.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:06 smithi146 ceph-mon[103278]: from='client.59493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:07.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/757204588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:07 smithi078 ceph-mon[99959]: from='client.59499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:07 smithi078 ceph-mon[99959]: from='client.79430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1449694490' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:07 smithi078 ceph-mon[91136]: from='client.59499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:07 smithi078 ceph-mon[91136]: from='client.79430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1449694490' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:07 smithi146 ceph-mon[103278]: from='client.59499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:07 smithi146 ceph-mon[103278]: from='client.79430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1449694490' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:08 smithi078 ceph-mon[91136]: pgmap v6815: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:08 smithi078 ceph-mon[99959]: pgmap v6815: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:08 smithi146 ceph-mon[103278]: pgmap v6815: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:10 smithi078 ceph-mon[91136]: pgmap v6816: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:10 smithi078 ceph-mon[99959]: pgmap v6816: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:07:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:07:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:10 smithi146 ceph-mon[103278]: pgmap v6816: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:11.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:11.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:11.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:11.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:11.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:12.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:12 smithi078 ceph-mon[99959]: pgmap v6817: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:12.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:12 smithi078 ceph-mon[91136]: pgmap v6817: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:12 smithi146 ceph-mon[103278]: pgmap v6817: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:13.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:13.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.763 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:13.763 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.763 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.763 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:14 smithi146 ceph-mon[103278]: pgmap v6818: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:14 smithi078 ceph-mon[99959]: pgmap v6818: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:14 smithi078 ceph-mon[91136]: pgmap v6818: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:16 smithi146 ceph-mon[103278]: pgmap v6819: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:16 smithi078 ceph-mon[91136]: pgmap v6819: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:16 smithi078 ceph-mon[99959]: pgmap v6819: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:18 smithi146 ceph-mon[103278]: pgmap v6820: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[91136]: pgmap v6820: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[99959]: pgmap v6820: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:20 smithi146 ceph-mon[103278]: pgmap v6821: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:20 smithi078 ceph-mon[91136]: pgmap v6821: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:20 smithi078 ceph-mon[99959]: pgmap v6821: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:07:20] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:07:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:21.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:21.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:21.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:21.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:21.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:22 smithi146 ceph-mon[103278]: pgmap v6822: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:22 smithi078 ceph-mon[91136]: pgmap v6822: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:22 smithi078 ceph-mon[99959]: pgmap v6822: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:24.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:24 smithi146 ceph-mon[103278]: pgmap v6823: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:24 smithi078 ceph-mon[91136]: pgmap v6823: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:24 smithi078 ceph-mon[99959]: pgmap v6823: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:26.991 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:26 smithi146 ceph-mon[103278]: pgmap v6824: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:26 smithi078 ceph-mon[91136]: pgmap v6824: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:26 smithi078 ceph-mon[99959]: pgmap v6824: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:28 smithi146 ceph-mon[103278]: pgmap v6825: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:28 smithi078 ceph-mon[91136]: pgmap v6825: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:28 smithi078 ceph-mon[99959]: pgmap v6825: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:30 smithi146 ceph-mon[103278]: pgmap v6826: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:30 smithi078 ceph-mon[91136]: pgmap v6826: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:30 smithi078 ceph-mon[99959]: pgmap v6826: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:07:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:07:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:31.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:31.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:31.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:32.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:32 smithi078 ceph-mon[99959]: pgmap v6827: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:32.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:32 smithi078 ceph-mon[91136]: pgmap v6827: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:32.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:32 smithi146 ceph-mon[103278]: pgmap v6827: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[91136]: pgmap v6828: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[99959]: pgmap v6828: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:34.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:34 smithi146 ceph-mon[103278]: pgmap v6828: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:36 smithi146 ceph-mon[103278]: pgmap v6829: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:36 smithi078 ceph-mon[91136]: pgmap v6829: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:36 smithi078 ceph-mon[99959]: pgmap v6829: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:37.177 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:07:37.702 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:07:37.702 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 3h - - 2024-09-21T00:07:37.702 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 3h - - 2024-09-21T00:07:37.702 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 4s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 19s ago 3h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 4s ago 3h 62.4M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 19s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 4s ago 3h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 4s ago 3h 402M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 19s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:07:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 4s ago 3h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 4s ago 3h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 19s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 4s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 4s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 4s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 4s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 19s ago 3h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 19s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:07:37.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 19s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:07:37.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 19s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:07:37.705 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 19s ago 3h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:07:37.705 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:07:37.705 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 19s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:07:37.705 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:07:37.705 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 19s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:07:38.010 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:07:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:07:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:07:38.012 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:07:38.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:07:38.277 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:07:38.586 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:07:38.587 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:38.587 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:07:38.587 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:38.587 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='client.59523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='client.79448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: pgmap v6830: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='client.79454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2323883127' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4164563296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='client.59523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='client.79448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: pgmap v6830: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='client.79454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2323883127' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4164563296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='client.59523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='client.79448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: pgmap v6830: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='client.79454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2323883127' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4164563296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:07:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:39 smithi146 ceph-mon[103278]: from='client.59547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:39 smithi078 ceph-mon[91136]: from='client.59547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:39 smithi078 ceph-mon[99959]: from='client.59547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:07:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:40 smithi146 ceph-mon[103278]: pgmap v6831: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:40 smithi078 ceph-mon[91136]: pgmap v6831: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:40 smithi078 ceph-mon[99959]: pgmap v6831: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:07:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:07:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:41.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:41.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:41.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:42 smithi146 ceph-mon[103278]: pgmap v6832: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:42 smithi078 ceph-mon[91136]: pgmap v6832: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:42 smithi078 ceph-mon[99959]: pgmap v6832: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:43.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:44.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:44 smithi146 ceph-mon[103278]: pgmap v6833: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:45.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:44 smithi078 ceph-mon[91136]: pgmap v6833: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:45.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:44 smithi078 ceph-mon[99959]: pgmap v6833: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:46.776 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:46 smithi146 ceph-mon[103278]: pgmap v6834: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:46 smithi078 ceph-mon[91136]: pgmap v6834: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:46 smithi078 ceph-mon[99959]: pgmap v6834: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:48 smithi146 ceph-mon[103278]: pgmap v6835: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:48 smithi078 ceph-mon[91136]: pgmap v6835: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:48 smithi078 ceph-mon[99959]: pgmap v6835: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:50 smithi146 ceph-mon[103278]: pgmap v6836: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:50 smithi078 ceph-mon[91136]: pgmap v6836: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:50 smithi078 ceph-mon[99959]: pgmap v6836: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:07:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:07:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:51.089Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:51.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:07:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:07:51.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:07:51.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:51.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:07:52.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:52 smithi078 ceph-mon[91136]: pgmap v6837: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:52.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:52 smithi078 ceph-mon[99959]: pgmap v6837: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:52 smithi146 ceph-mon[103278]: pgmap v6837: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:54 smithi146 ceph-mon[103278]: pgmap v6838: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[91136]: pgmap v6838: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[99959]: pgmap v6838: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:07:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:56 smithi146 ceph-mon[103278]: pgmap v6839: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:56 smithi078 ceph-mon[91136]: pgmap v6839: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:56 smithi078 ceph-mon[99959]: pgmap v6839: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:58 smithi146 ceph-mon[103278]: pgmap v6840: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[91136]: pgmap v6840: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[99959]: pgmap v6840: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:07:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:07:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:00 smithi146 ceph-mon[103278]: pgmap v6841: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:00 smithi078 ceph-mon[91136]: pgmap v6841: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:00 smithi078 ceph-mon[99959]: pgmap v6841: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:08:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:08:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:01.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:01.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:01.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:02 smithi146 ceph-mon[103278]: pgmap v6842: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:02 smithi078 ceph-mon[91136]: pgmap v6842: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:02 smithi078 ceph-mon[99959]: pgmap v6842: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:04 smithi146 ceph-mon[103278]: pgmap v6843: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:04 smithi078 ceph-mon[91136]: pgmap v6843: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:04 smithi078 ceph-mon[99959]: pgmap v6843: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:06 smithi078 ceph-mon[91136]: pgmap v6844: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:06 smithi078 ceph-mon[99959]: pgmap v6844: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:07.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:06 smithi146 ceph-mon[103278]: pgmap v6844: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:08.885 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:08:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:08 smithi078 ceph-mon[99959]: pgmap v6845: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:08 smithi078 ceph-mon[91136]: pgmap v6845: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:08 smithi146 ceph-mon[103278]: pgmap v6845: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 3h - - 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 3h - - 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 15s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 10s ago 3h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 15s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 15s ago 3h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 15s ago 3h 403M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:08:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 15s ago 3h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 15s ago 3h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 10s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 15s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 15s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 15s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 15s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 3h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:08:09.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 10s ago 3h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 15s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 15s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:08:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:08:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:08:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:08:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:08:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:09.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:08:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:08:09.992 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:08:09.993 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:08:09.993 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:08:09.993 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:08:09.993 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:08:09.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:09 smithi078 ceph-mon[91136]: from='client.59559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:09.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3535017487' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:09.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:09 smithi078 ceph-mon[99959]: from='client.59559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:09.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3535017487' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:09 smithi146 ceph-mon[103278]: from='client.59559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3535017487' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:10.302 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:08:10.302 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:10.302 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:10.302 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:10.302 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:08:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:08:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[91136]: from='client.59565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[91136]: from='client.59571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[91136]: pgmap v6846: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/837108596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[99959]: from='client.59565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[99959]: from='client.59571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[99959]: pgmap v6846: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/837108596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:10 smithi146 ceph-mon[103278]: from='client.59565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:10 smithi146 ceph-mon[103278]: from='client.59571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:10 smithi146 ceph-mon[103278]: pgmap v6846: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/837108596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:11.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:11.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:11.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:11 smithi146 ceph-mon[103278]: from='client.59583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-mon[91136]: from='client.59583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-mon[99959]: from='client.59583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:12 smithi146 ceph-mon[103278]: pgmap v6847: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:12 smithi078 ceph-mon[91136]: pgmap v6847: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:12 smithi078 ceph-mon[99959]: pgmap v6847: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:14 smithi146 ceph-mon[103278]: pgmap v6848: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[99959]: pgmap v6848: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[91136]: pgmap v6848: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:15.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:16 smithi078 ceph-mon[91136]: pgmap v6849: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:16 smithi078 ceph-mon[99959]: pgmap v6849: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:16 smithi146 ceph-mon[103278]: pgmap v6849: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:18 smithi146 ceph-mon[103278]: pgmap v6850: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[99959]: pgmap v6850: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[91136]: pgmap v6850: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:19.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:20.874 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:08:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:08:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:20 smithi146 ceph-mon[103278]: pgmap v6851: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:20 smithi078 ceph-mon[91136]: pgmap v6851: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:20 smithi078 ceph-mon[99959]: pgmap v6851: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:21.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:21.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:21.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:22 smithi146 ceph-mon[103278]: pgmap v6852: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:22 smithi078 ceph-mon[91136]: pgmap v6852: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:22 smithi078 ceph-mon[99959]: pgmap v6852: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:24 smithi146 ceph-mon[103278]: pgmap v6853: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:24 smithi078 ceph-mon[91136]: pgmap v6853: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:24 smithi078 ceph-mon[99959]: pgmap v6853: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:26 smithi146 ceph-mon[103278]: pgmap v6854: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:26 smithi078 ceph-mon[91136]: pgmap v6854: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:26 smithi078 ceph-mon[99959]: pgmap v6854: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:28 smithi146 ceph-mon[103278]: pgmap v6855: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:28 smithi078 ceph-mon[91136]: pgmap v6855: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:28 smithi078 ceph-mon[99959]: pgmap v6855: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:30.893 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:08:30] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:08:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:30 smithi146 ceph-mon[103278]: pgmap v6856: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:31.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:30 smithi078 ceph-mon[91136]: pgmap v6856: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:31.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:30 smithi078 ceph-mon[99959]: pgmap v6856: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:31.291 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:31.092Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:31.291 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:31.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:31.291 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:31.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:32.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:32 smithi146 ceph-mon[103278]: pgmap v6857: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:32 smithi078 ceph-mon[91136]: pgmap v6857: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:32 smithi078 ceph-mon[99959]: pgmap v6857: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[91136]: pgmap v6858: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[99959]: pgmap v6858: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:34 smithi146 ceph-mon[103278]: pgmap v6858: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:36 smithi146 ceph-mon[103278]: pgmap v6859: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:36 smithi078 ceph-mon[91136]: pgmap v6859: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:36 smithi078 ceph-mon[99959]: pgmap v6859: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:38 smithi146 ceph-mon[103278]: pgmap v6860: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[91136]: pgmap v6860: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[99959]: pgmap v6860: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:39 smithi146 ceph-mon[103278]: pgmap v6861: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:39 smithi078 ceph-mon[91136]: pgmap v6861: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:39 smithi078 ceph-mon[99959]: pgmap v6861: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:40.596 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:08:40.917 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:08:40] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 3h - - 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 3h - - 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 7s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 2s ago 3h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 7s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 2s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 7s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 7s ago 4h 403M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:08:41.131 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 2s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 7s ago 3h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 7s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 2s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 7s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 7s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 7s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 7s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 2s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 2s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 2s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 2s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 2s ago 3h 70.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 2s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:08:41.132 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:08:41.133 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 2s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:08:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:40 smithi146 ceph-mon[103278]: from='client.59595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:40 smithi146 ceph-mon[103278]: from='client.79520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-mon[99959]: from='client.59595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-mon[99959]: from='client.79520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-mon[91136]: from='client.59595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:40 smithi078 ceph-mon[91136]: from='client.79520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:41.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:41.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:41.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:08:41.440 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:08:41.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:08:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:08:41.707 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:08:41.707 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:08:41.708 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:08:42.020 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:08:42.021 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:42.021 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:08:42.021 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:42.021 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:08:42.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[99959]: from='client.79526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:42.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1257703712' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:42.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[99959]: pgmap v6862: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:42.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[99959]: from='client.79538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:42.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[91136]: from='client.79526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:42.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1257703712' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:42.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[91136]: pgmap v6862: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:42.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:41 smithi078 ceph-mon[91136]: from='client.79538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:41 smithi146 ceph-mon[103278]: from='client.79526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1257703712' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:08:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:41 smithi146 ceph-mon[103278]: pgmap v6862: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:41 smithi146 ceph-mon[103278]: from='client.79538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:08:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2375132248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2375132248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2375132248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:08:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:43 smithi146 ceph-mon[103278]: pgmap v6863: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:43 smithi078 ceph-mon[99959]: pgmap v6863: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:43 smithi078 ceph-mon[91136]: pgmap v6863: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:46.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:46 smithi146 ceph-mon[103278]: pgmap v6864: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:46 smithi078 ceph-mon[91136]: pgmap v6864: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:46 smithi078 ceph-mon[99959]: pgmap v6864: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:48.951 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:48 smithi146 ceph-mon[103278]: pgmap v6865: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:48 smithi078 ceph-mon[99959]: pgmap v6865: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:48 smithi078 ceph-mon[91136]: pgmap v6865: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:50 smithi146 ceph-mon[103278]: pgmap v6866: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:50 smithi078 ceph-mon[99959]: pgmap v6866: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:50 smithi078 ceph-mon[91136]: pgmap v6866: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:08:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:08:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:51.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:51.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:08:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:08:51.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:08:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:08:52.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:52 smithi078 ceph-mon[91136]: pgmap v6867: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:52.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:52 smithi078 ceph-mon[99959]: pgmap v6867: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:52 smithi146 ceph-mon[103278]: pgmap v6867: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:54.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[91136]: pgmap v6868: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:54.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:54.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:54.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:54.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[99959]: pgmap v6868: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:54.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:54.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:54.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:54 smithi146 ceph-mon[103278]: pgmap v6868: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:08:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:08:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:56 smithi078 ceph-mon[99959]: pgmap v6869: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:56 smithi078 ceph-mon[91136]: pgmap v6869: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:56 smithi146 ceph-mon[103278]: pgmap v6869: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:58.827 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:58 smithi146 ceph-mon[103278]: pgmap v6870: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:58.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:58 smithi078 ceph-mon[91136]: pgmap v6870: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:08:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:58 smithi078 ceph-mon[99959]: pgmap v6870: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:08:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:08:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:08:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:08:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:00 smithi078 ceph-mon[99959]: pgmap v6871: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:00 smithi078 ceph-mon[91136]: pgmap v6871: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:09:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:09:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:00 smithi146 ceph-mon[103278]: pgmap v6871: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:01.095Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:01.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:01.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:02.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:02 smithi146 ceph-mon[103278]: pgmap v6872: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:02 smithi078 ceph-mon[99959]: pgmap v6872: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:02 smithi078 ceph-mon[91136]: pgmap v6872: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:04 smithi146 ceph-mon[103278]: pgmap v6873: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:04 smithi078 ceph-mon[99959]: pgmap v6873: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:04 smithi078 ceph-mon[91136]: pgmap v6873: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:07.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:06 smithi146 ceph-mon[103278]: pgmap v6874: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:06 smithi078 ceph-mon[99959]: pgmap v6874: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:06 smithi078 ceph-mon[91136]: pgmap v6874: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:08.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:08 smithi146 ceph-mon[103278]: pgmap v6875: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:08 smithi078 ceph-mon[99959]: pgmap v6875: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:08 smithi078 ceph-mon[91136]: pgmap v6875: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:10 smithi078 ceph-mon[91136]: pgmap v6876: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:10 smithi078 ceph-mon[99959]: pgmap v6876: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:09:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:09:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:10 smithi146 ceph-mon[103278]: pgmap v6876: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:11.096Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:11.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:11.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:12.317 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:09:12.839 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:09:12.839 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 3h - - 2024-09-21T00:09:12.839 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 3h - - 2024-09-21T00:09:12.839 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 18s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 14s ago 3h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 18s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 14s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 18s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 18s ago 4h 403M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 14s ago 3h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 18s ago 3h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 18s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 14s ago 3h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 18s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 18s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 14s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:09:12.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 14s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 14s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 14s ago 3h 70.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:09:12.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:09:13.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:09:13.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:13.153 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:09:13.153 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:09:13.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:09:13.153 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:09:13.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:12 smithi078 ceph-mon[99959]: pgmap v6877: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:13.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:12 smithi078 ceph-mon[91136]: pgmap v6877: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:12 smithi146 ceph-mon[103278]: pgmap v6877: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:13.420 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:09:13.420 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:09:13.421 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:09:13.731 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:09:13.731 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:13.731 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:13.731 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:13.731 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[91136]: from='client.59631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[91136]: from='client.59637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[91136]: from='client.59643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2378284632' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2973736331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[99959]: from='client.59631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[99959]: from='client.59637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[99959]: from='client.59643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2378284632' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2973736331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:13 smithi146 ceph-mon[103278]: from='client.59631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:13 smithi146 ceph-mon[103278]: from='client.59637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:13 smithi146 ceph-mon[103278]: from='client.59643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2378284632' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2973736331' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:14 smithi146 ceph-mon[103278]: from='client.79574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:14 smithi146 ceph-mon[103278]: pgmap v6878: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[91136]: from='client.79574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[91136]: pgmap v6878: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[99959]: from='client.79574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[99959]: pgmap v6878: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:16 smithi078 ceph-mon[91136]: pgmap v6879: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:16 smithi078 ceph-mon[99959]: pgmap v6879: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:16 smithi146 ceph-mon[103278]: pgmap v6879: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:18 smithi146 ceph-mon[103278]: pgmap v6880: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:18 smithi078 ceph-mon[91136]: pgmap v6880: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:18 smithi078 ceph-mon[99959]: pgmap v6880: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:19 smithi146 ceph-mon[103278]: pgmap v6881: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[91136]: pgmap v6881: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:19 smithi078 ceph-mon[99959]: pgmap v6881: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:20.954 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:09:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:09:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:21.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:21.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:21.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:21 smithi146 ceph-mon[103278]: pgmap v6882: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:21 smithi078 ceph-mon[91136]: pgmap v6882: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:21 smithi078 ceph-mon[99959]: pgmap v6882: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:24 smithi146 ceph-mon[103278]: pgmap v6883: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:24 smithi078 ceph-mon[91136]: pgmap v6883: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:24 smithi078 ceph-mon[99959]: pgmap v6883: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:26 smithi146 ceph-mon[103278]: pgmap v6884: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:26 smithi078 ceph-mon[91136]: pgmap v6884: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:26 smithi078 ceph-mon[99959]: pgmap v6884: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:28.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:28 smithi146 ceph-mon[103278]: pgmap v6885: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:28.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:28 smithi078 ceph-mon[91136]: pgmap v6885: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:28 smithi078 ceph-mon[99959]: pgmap v6885: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:30 smithi146 ceph-mon[103278]: pgmap v6886: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:30 smithi078 ceph-mon[91136]: pgmap v6886: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:30 smithi078 ceph-mon[99959]: pgmap v6886: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:09:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:09:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:31.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:31.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:31.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:32.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:32 smithi078 ceph-mon[91136]: pgmap v6887: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:32.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:32 smithi078 ceph-mon[99959]: pgmap v6887: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:32 smithi146 ceph-mon[103278]: pgmap v6887: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[91136]: pgmap v6888: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[99959]: pgmap v6888: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:34 smithi146 ceph-mon[103278]: pgmap v6888: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:36.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:36 smithi078 ceph-mon[91136]: pgmap v6889: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:36.829 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:36 smithi078 ceph-mon[99959]: pgmap v6889: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:36 smithi146 ceph-mon[103278]: pgmap v6889: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:38.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:38 smithi078 ceph-mon[91136]: pgmap v6890: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:38.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:38 smithi078 ceph-mon[99959]: pgmap v6890: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:38 smithi146 ceph-mon[103278]: pgmap v6890: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:40 smithi146 ceph-mon[103278]: pgmap v6891: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[91136]: pgmap v6891: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-mon[99959]: pgmap v6891: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:09:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:09:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:41.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:41.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:41.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:42 smithi146 ceph-mon[103278]: pgmap v6892: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:42 smithi078 ceph-mon[99959]: pgmap v6892: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:42 smithi078 ceph-mon[91136]: pgmap v6892: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:44.030 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 4h - - 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 10s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 5s ago 3h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 10s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 5s ago 3h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 10s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 10s ago 4h 404M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 5s ago 3h 263M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:09:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 10s ago 3h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 10s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 5s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 10s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 10s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 10s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 10s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 5s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 5s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 5s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 5s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 5s ago 3h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 5s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 10s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:09:44.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 5s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:09:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:44 smithi146 ceph-mon[103278]: pgmap v6893: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:44 smithi078 ceph-mon[99959]: pgmap v6893: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:44 smithi078 ceph-mon[91136]: pgmap v6893: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:44.871 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:09:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:09:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:09:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:09:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:09:44.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:09:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:09:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:09:45.142 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:09:45.451 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:09:45.451 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:09:45.451 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:09:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:45 smithi146 ceph-mon[103278]: from='client.79586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:45 smithi146 ceph-mon[103278]: from='client.59673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:45 smithi146 ceph-mon[103278]: from='client.59679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1918084024' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[91136]: from='client.79586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[91136]: from='client.59673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[91136]: from='client.59679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1918084024' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[99959]: from='client.79586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[99959]: from='client.59673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[99959]: from='client.59679 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1918084024' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:09:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:46 smithi146 ceph-mon[103278]: from='client.79610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4273345360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:46 smithi146 ceph-mon[103278]: pgmap v6894: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:46 smithi078 ceph-mon[99959]: from='client.79610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4273345360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:46 smithi078 ceph-mon[99959]: pgmap v6894: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:46 smithi078 ceph-mon[91136]: from='client.79610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:09:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4273345360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:09:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:46 smithi078 ceph-mon[91136]: pgmap v6894: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:48.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:48 smithi146 ceph-mon[103278]: pgmap v6895: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:48.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:48 smithi078 ceph-mon[91136]: pgmap v6895: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:48.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:48 smithi078 ceph-mon[99959]: pgmap v6895: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:50.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:50 smithi078 ceph-mon[91136]: pgmap v6896: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:50.814 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:09:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:09:50.814 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:50 smithi078 ceph-mon[99959]: pgmap v6896: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:50 smithi146 ceph-mon[103278]: pgmap v6896: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:51.098Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:51.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:09:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:09:51.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:09:51.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:51.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:09:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:52 smithi146 ceph-mon[103278]: pgmap v6897: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:52 smithi078 ceph-mon[91136]: pgmap v6897: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:52 smithi078 ceph-mon[99959]: pgmap v6897: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[91136]: pgmap v6898: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[99959]: pgmap v6898: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:54 smithi146 ceph-mon[103278]: pgmap v6898: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:09:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:09:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:56 smithi146 ceph-mon[103278]: pgmap v6899: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:56 smithi078 ceph-mon[91136]: pgmap v6899: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:56 smithi078 ceph-mon[99959]: pgmap v6899: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:58.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:58 smithi078 ceph-mon[91136]: pgmap v6900: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:58.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:58.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:58 smithi078 ceph-mon[99959]: pgmap v6900: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:58.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:09:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:58 smithi146 ceph-mon[103278]: pgmap v6900: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:09:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[99959]: pgmap v6901: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T00:09:59.998+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[91136]: pgmap v6901: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:00 smithi146 ceph-mon[103278]: pgmap v6901: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:10:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:10:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:01.099Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:01.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:01.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:02 smithi146 ceph-mon[103278]: pgmap v6902: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:02 smithi078 ceph-mon[91136]: pgmap v6902: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:02 smithi078 ceph-mon[99959]: pgmap v6902: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:04 smithi146 ceph-mon[103278]: pgmap v6903: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:04 smithi078 ceph-mon[91136]: pgmap v6903: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:04 smithi078 ceph-mon[99959]: pgmap v6903: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:06 smithi146 ceph-mon[103278]: pgmap v6904: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:06 smithi078 ceph-mon[91136]: pgmap v6904: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:06 smithi078 ceph-mon[99959]: pgmap v6904: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:08 smithi146 ceph-mon[103278]: pgmap v6905: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:08 smithi078 ceph-mon[91136]: pgmap v6905: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:08 smithi078 ceph-mon[99959]: pgmap v6905: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:10.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:10 smithi146 ceph-mon[103278]: pgmap v6906: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:10.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:10 smithi078 ceph-mon[91136]: pgmap v6906: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:10.826 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:10 smithi078 ceph-mon[99959]: pgmap v6906: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:10.826 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:10:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:10:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:11.099Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:11.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:11.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:12 smithi146 ceph-mon[103278]: pgmap v6907: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:12 smithi078 ceph-mon[91136]: pgmap v6907: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:12 smithi078 ceph-mon[99959]: pgmap v6907: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:14 smithi146 ceph-mon[103278]: pgmap v6908: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:14 smithi078 ceph-mon[91136]: pgmap v6908: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:14 smithi078 ceph-mon[99959]: pgmap v6908: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:15.749 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:10:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 4h - - 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 4h - - 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 1s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 17s ago 3h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 1s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 17s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 1s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 1s ago 4h 406M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 17s ago 4h 264M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:10:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 1s ago 4h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 1s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 17s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 1s ago 3h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 1s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 1s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 1s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 17s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 17s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 17s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 17s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 17s ago 3h 73.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 17s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:10:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 17s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:10:16.582 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:10:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:10:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:10:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:10:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:10:16.583 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:10:16.852 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:10:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:10:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:10:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:10:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:10:16.853 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:10:16.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:16 smithi078 ceph-mon[91136]: pgmap v6909: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:10:16.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:16 smithi078 ceph-mon[91136]: from='client.79622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:16.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4198674097' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:16.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:16 smithi078 ceph-mon[99959]: pgmap v6909: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:10:16.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:16 smithi078 ceph-mon[99959]: from='client.79622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:16.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4198674097' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:16 smithi146 ceph-mon[103278]: pgmap v6909: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:10:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:16 smithi146 ceph-mon[103278]: from='client.79622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4198674097' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:17.158 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:17.158 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:17.159 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:17.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[99959]: from='client.59709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[99959]: from='client.59715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[99959]: from='client.79646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2178392454' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:17.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[91136]: from='client.59709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[91136]: from='client.59715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[91136]: from='client.79646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:17.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2178392454' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:17 smithi146 ceph-mon[103278]: from='client.59709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:17 smithi146 ceph-mon[103278]: from='client.59715 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:17 smithi146 ceph-mon[103278]: from='client.79646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2178392454' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:18 smithi146 ceph-mon[103278]: pgmap v6910: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:18 smithi078 ceph-mon[99959]: pgmap v6910: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:18 smithi078 ceph-mon[91136]: pgmap v6910: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:20 smithi146 ceph-mon[103278]: pgmap v6911: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[91136]: pgmap v6911: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-mon[99959]: pgmap v6911: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:20.909 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:10:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:10:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:21.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:21.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:21.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:22 smithi146 ceph-mon[103278]: pgmap v6912: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:22 smithi078 ceph-mon[99959]: pgmap v6912: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:22 smithi078 ceph-mon[91136]: pgmap v6912: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:24 smithi146 ceph-mon[103278]: pgmap v6913: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:24 smithi078 ceph-mon[99959]: pgmap v6913: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:24 smithi078 ceph-mon[91136]: pgmap v6913: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:26 smithi146 ceph-mon[103278]: pgmap v6914: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:26 smithi078 ceph-mon[99959]: pgmap v6914: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:26 smithi078 ceph-mon[91136]: pgmap v6914: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:29.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:28 smithi146 ceph-mon[103278]: pgmap v6915: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:29.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:28 smithi078 ceph-mon[99959]: pgmap v6915: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:28 smithi078 ceph-mon[91136]: pgmap v6915: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:30.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:10:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:10:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:30 smithi146 ceph-mon[103278]: pgmap v6916: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:30 smithi078 ceph-mon[91136]: pgmap v6916: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:30 smithi078 ceph-mon[99959]: pgmap v6916: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:31.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:31.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:31.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:32 smithi146 ceph-mon[103278]: pgmap v6917: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:32 smithi078 ceph-mon[91136]: pgmap v6917: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:32 smithi078 ceph-mon[99959]: pgmap v6917: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[91136]: pgmap v6918: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[99959]: pgmap v6918: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:34 smithi146 ceph-mon[103278]: pgmap v6918: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:36 smithi146 ceph-mon[103278]: pgmap v6919: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:36 smithi078 ceph-mon[91136]: pgmap v6919: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:36 smithi078 ceph-mon[99959]: pgmap v6919: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:39.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:38 smithi078 ceph-mon[99959]: pgmap v6920: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:39.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:38 smithi078 ceph-mon[91136]: pgmap v6920: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:38 smithi146 ceph-mon[103278]: pgmap v6920: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:40 smithi146 ceph-mon[103278]: pgmap v6921: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[91136]: pgmap v6921: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:10:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:10:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:40 smithi078 ceph-mon[99959]: pgmap v6921: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:41.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:41.101Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:41.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:41.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:41.475 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:41.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:42 smithi146 ceph-mon[103278]: pgmap v6922: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:42 smithi078 ceph-mon[91136]: pgmap v6922: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:42 smithi078 ceph-mon[99959]: pgmap v6922: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:44 smithi146 ceph-mon[103278]: pgmap v6923: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:44 smithi078 ceph-mon[91136]: pgmap v6923: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:44 smithi078 ceph-mon[99959]: pgmap v6923: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:46 smithi146 ceph-mon[103278]: pgmap v6924: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:46 smithi078 ceph-mon[99959]: pgmap v6924: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:46 smithi078 ceph-mon[91136]: pgmap v6924: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:47.453 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 4h - - 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 4h - - 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 3h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 8s ago 3h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 8s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 4h 406M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 8s ago 4h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:10:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 8s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 4h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 13s ago 3h 87.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 13s ago 3h 74.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 13s ago 3h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 8s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 8s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 8s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 8s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 8s ago 3h 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:10:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 8s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:10:47.981 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:10:47.981 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 8s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:10:48.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:10:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:10:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:10:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:10:48.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:10:48.558 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:10:48.559 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:10:48.560 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:10:48.873 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:10:48.874 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:10:48.874 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:10:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:48 smithi146 ceph-mon[103278]: from='client.79658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:48 smithi146 ceph-mon[103278]: pgmap v6925: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:48 smithi146 ceph-mon[103278]: from='client.59745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3082929648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2388613995' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[91136]: from='client.79658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[91136]: pgmap v6925: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[91136]: from='client.59745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3082929648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2388613995' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[99959]: from='client.79658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[99959]: pgmap v6925: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[99959]: from='client.59745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:49.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3082929648' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:10:49.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2388613995' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:10:50.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:49 smithi146 ceph-mon[103278]: from='client.59751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:50.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:49 smithi146 ceph-mon[103278]: from='client.79682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:49 smithi078 ceph-mon[99959]: from='client.59751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:49 smithi078 ceph-mon[99959]: from='client.79682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:49 smithi078 ceph-mon[91136]: from='client.59751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:49 smithi078 ceph-mon[91136]: from='client.79682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:10:50.860 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:10:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:10:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:50 smithi146 ceph-mon[103278]: pgmap v6926: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:50 smithi078 ceph-mon[99959]: pgmap v6926: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:50 smithi078 ceph-mon[91136]: pgmap v6926: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:10:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:51.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:51.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:10:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:10:51.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:10:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:10:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:52 smithi146 ceph-mon[103278]: pgmap v6927: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:52 smithi078 ceph-mon[91136]: pgmap v6927: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:52 smithi078 ceph-mon[99959]: pgmap v6927: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:10:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:54 smithi146 ceph-mon[103278]: pgmap v6928: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:10:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:54 smithi078 ceph-mon[99959]: pgmap v6928: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:10:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:54 smithi078 ceph-mon[91136]: pgmap v6928: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:10:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:10:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:56 smithi146 ceph-mon[103278]: pgmap v6929: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:10:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:56 smithi078 ceph-mon[91136]: pgmap v6929: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:10:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:56 smithi078 ceph-mon[99959]: pgmap v6929: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:10:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:58 smithi146 ceph-mon[103278]: pgmap v6930: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:10:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:10:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:58 smithi078 ceph-mon[91136]: pgmap v6930: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:10:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:10:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:10:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:58 smithi078 ceph-mon[99959]: pgmap v6930: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:10:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:10:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:00.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[91136]: pgmap v6931: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:11:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[99959]: pgmap v6931: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:00.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:00.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:00 smithi146 ceph-mon[103278]: pgmap v6931: 161 pgs: 161 active+clean; 585 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:11:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:01.407 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:01.103Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:01.407 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:01.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:01.407 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:01.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:01.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:01.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:01.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:03.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:02 smithi146 ceph-mon[103278]: pgmap v6932: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:11:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:02 smithi078 ceph-mon[91136]: pgmap v6932: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:11:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:02 smithi078 ceph-mon[99959]: pgmap v6932: 161 pgs: 161 active+clean; 585 KiB data, 324 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:11:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:04 smithi146 ceph-mon[103278]: pgmap v6933: 161 pgs: 161 active+clean; 585 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:11:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:04 smithi078 ceph-mon[91136]: pgmap v6933: 161 pgs: 161 active+clean; 585 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:11:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:04 smithi078 ceph-mon[99959]: pgmap v6933: 161 pgs: 161 active+clean; 585 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:11:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:06 smithi146 ceph-mon[103278]: pgmap v6934: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:11:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:06 smithi078 ceph-mon[91136]: pgmap v6934: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:11:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:06 smithi078 ceph-mon[99959]: pgmap v6934: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:11:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:08 smithi146 ceph-mon[103278]: pgmap v6935: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:11:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:08 smithi078 ceph-mon[99959]: pgmap v6935: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:11:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:08 smithi078 ceph-mon[91136]: pgmap v6935: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:11:10.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:10 smithi078 ceph-mon[91136]: pgmap v6936: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:11:10.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:10 smithi078 ceph-mon[99959]: pgmap v6936: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:11:10.882 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:11:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:10 smithi146 ceph-mon[103278]: pgmap v6936: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:11:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:11.104Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:11.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:11.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:12.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:12 smithi146 ceph-mon[103278]: pgmap v6937: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:11:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:12 smithi078 ceph-mon[91136]: pgmap v6937: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:11:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:12 smithi078 ceph-mon[99959]: pgmap v6937: 161 pgs: 161 active+clean; 585 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:11:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:14 smithi146 ceph-mon[103278]: pgmap v6938: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:11:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:14 smithi078 ceph-mon[99959]: pgmap v6938: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:11:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:14 smithi078 ceph-mon[91136]: pgmap v6938: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:11:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:16 smithi146 ceph-mon[103278]: pgmap v6939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[91136]: pgmap v6939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:16 smithi078 ceph-mon[99959]: pgmap v6939: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-09-21T00:11:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:18 smithi146 ceph-mon[103278]: pgmap v6940: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T00:11:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:18 smithi078 ceph-mon[91136]: pgmap v6940: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T00:11:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:18 smithi078 ceph-mon[99959]: pgmap v6940: 161 pgs: 161 active+clean; 585 KiB data, 295 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T00:11:19.169 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 4h - - 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 4h - - 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 4s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 20s ago 3h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 4s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 20s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 4s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 4s ago 4h 407M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:11:19.700 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 20s ago 4h 262M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 4s ago 4h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 4s ago 3h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 20s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 4s ago 4h 83.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 4s ago 4h 87.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 4s ago 4h 75.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 4s ago 3h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 20s ago 3h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 20s ago 3h 76.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 20s ago 3h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 20s ago 3h 79.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 20s ago 3h 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 20s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:11:19.701 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 20s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:11:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:11:20.011 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:11:20.286 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:11:20.287 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:11:20.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:11:20.605 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:11:20.605 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:20.605 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:11:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='client.79694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='client.59781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: pgmap v6941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='client.79706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:20.890 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/348779585' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:20.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2688400263' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='client.79694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='client.59781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: pgmap v6941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='client.79706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/348779585' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:20.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2688400263' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='client.79694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='client.59781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: pgmap v6941: 161 pgs: 161 active+clean; 585 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='client.79706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/348779585' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2688400263' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:21.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:21.105Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:21.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:21.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:21.199 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:21.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:21.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-mon[99959]: from='client.59799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:21.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:21.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-mon[91136]: from='client.59799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:21.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:21 smithi146 ceph-mon[103278]: from='client.59799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:22.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:22 smithi078 ceph-mon[99959]: pgmap v6942: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:11:22.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:22 smithi078 ceph-mon[91136]: pgmap v6942: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:11:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:22 smithi146 ceph-mon[103278]: pgmap v6942: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:11:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:24 smithi146 ceph-mon[103278]: pgmap v6943: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:11:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:24 smithi078 ceph-mon[91136]: pgmap v6943: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:11:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:24 smithi078 ceph-mon[99959]: pgmap v6943: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:11:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:26 smithi146 ceph-mon[103278]: pgmap v6944: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:11:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:26 smithi078 ceph-mon[91136]: pgmap v6944: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:11:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:26 smithi078 ceph-mon[99959]: pgmap v6944: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:11:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:28 smithi146 ceph-mon[103278]: pgmap v6945: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T00:11:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:28 smithi078 ceph-mon[91136]: pgmap v6945: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T00:11:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:28 smithi078 ceph-mon[99959]: pgmap v6945: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T00:11:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:30.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:30 smithi078 ceph-mon[91136]: pgmap v6946: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T00:11:30.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:30 smithi078 ceph-mon[99959]: pgmap v6946: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T00:11:30.895 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:11:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:11:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:30 smithi146 ceph-mon[103278]: pgmap v6946: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-09-21T00:11:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:31.106Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:31.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:31.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:32.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:32 smithi146 ceph-mon[103278]: pgmap v6947: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T00:11:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:32 smithi078 ceph-mon[99959]: pgmap v6947: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T00:11:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:32 smithi078 ceph-mon[91136]: pgmap v6947: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T00:11:34.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:34 smithi146 ceph-mon[103278]: pgmap v6948: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:34 smithi078 ceph-mon[91136]: pgmap v6948: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:34 smithi078 ceph-mon[99959]: pgmap v6948: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:36 smithi146 ceph-mon[103278]: pgmap v6949: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[91136]: pgmap v6949: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:36 smithi078 ceph-mon[99959]: pgmap v6949: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:38 smithi146 ceph-mon[103278]: pgmap v6950: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:38 smithi078 ceph-mon[91136]: pgmap v6950: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:38 smithi078 ceph-mon[99959]: pgmap v6950: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:40.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[91136]: pgmap v6951: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:40.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:40.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:40.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:40.906 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:11:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:11:40.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[99959]: pgmap v6951: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:40.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:40.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:40.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:41.160 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:41.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:41.160 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:41.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:41.160 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:41.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:40 smithi146 ceph-mon[103278]: pgmap v6951: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:42.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:42.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:42 smithi078 ceph-mon[91136]: pgmap v6952: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:42 smithi078 ceph-mon[99959]: pgmap v6952: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:42 smithi146 ceph-mon[103278]: pgmap v6952: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:44.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:44.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:44 smithi146 ceph-mon[103278]: pgmap v6953: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:44 smithi078 ceph-mon[91136]: pgmap v6953: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:44 smithi078 ceph-mon[99959]: pgmap v6953: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:46 smithi146 ceph-mon[103278]: pgmap v6954: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:46 smithi078 ceph-mon[91136]: pgmap v6954: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:46 smithi078 ceph-mon[99959]: pgmap v6954: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:48 smithi146 ceph-mon[103278]: pgmap v6955: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:48 smithi078 ceph-mon[91136]: pgmap v6955: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:48 smithi078 ceph-mon[99959]: pgmap v6955: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:50.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:11:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:11:50.898 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:11:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:50 smithi146 ceph-mon[103278]: pgmap v6956: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:50 smithi078 ceph-mon[99959]: pgmap v6956: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:50 smithi078 ceph-mon[91136]: pgmap v6956: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:51.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:51.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:11:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:11:51.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:11:51.426 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 4h - - 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 4h - - 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 16s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 11s ago 3h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 11s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 4h 407M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 11s ago 4h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 16s ago 4h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 16s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 11s ago 3h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:11:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 16s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 16s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 16s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 16s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 11s ago 3h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 11s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 11s ago 3h 84.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 11s ago 3h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 11s ago 3h 71.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 11s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 16s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:11:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 11s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:11:51.740 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:11:52.014 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:11:52.014 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:11:52.014 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:11:52.014 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:11:52.014 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:11:52.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:11:52.015 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:11:52.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:11:52.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-mon[99959]: from='client.79730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:52.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:52.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1156507525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-mon[91136]: from='client.79730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1156507525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:51 smithi146 ceph-mon[103278]: from='client.79730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:11:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1156507525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:11:52.327 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:11:52.327 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:11:52.327 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:11:52.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:52 smithi146 ceph-mon[103278]: from='client.79736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:52.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:52 smithi146 ceph-mon[103278]: from='client.59823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:52.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:52 smithi146 ceph-mon[103278]: pgmap v6957: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:52.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/355245011' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[91136]: from='client.79736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[91136]: from='client.59823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[91136]: pgmap v6957: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/355245011' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[99959]: from='client.79736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[99959]: from='client.59823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[99959]: pgmap v6957: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/355245011' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:11:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:53 smithi146 ceph-mon[103278]: from='client.59835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:53 smithi078 ceph-mon[91136]: from='client.59835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:53 smithi078 ceph-mon[99959]: from='client.59835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:11:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:54 smithi146 ceph-mon[103278]: pgmap v6958: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:54 smithi078 ceph-mon[91136]: pgmap v6958: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:54 smithi078 ceph-mon[99959]: pgmap v6958: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:11:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:56 smithi146 ceph-mon[103278]: pgmap v6959: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[91136]: pgmap v6959: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:11:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:56 smithi078 ceph-mon[99959]: pgmap v6959: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:58 smithi146 ceph-mon[103278]: pgmap v6960: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:11:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:58 smithi078 ceph-mon[99959]: pgmap v6960: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:11:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:11:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:58 smithi078 ceph-mon[91136]: pgmap v6960: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:11:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:11:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:00.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[99959]: pgmap v6961: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:00.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:00.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:00.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:00.922 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[91136]: pgmap v6961: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:00.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:00.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:00.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:00.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:12:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:12:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:00 smithi146 ceph-mon[103278]: pgmap v6961: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:01.108Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:01.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:01.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:01.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:01.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:02.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:02 smithi078 ceph-mon[91136]: pgmap v6962: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:12:02.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:02 smithi078 ceph-mon[99959]: pgmap v6962: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:12:02.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:02 smithi146 ceph-mon[103278]: pgmap v6962: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:12:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:04 smithi146 ceph-mon[103278]: pgmap v6963: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:12:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:04 smithi078 ceph-mon[91136]: pgmap v6963: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:12:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:04 smithi078 ceph-mon[99959]: pgmap v6963: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:12:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:06 smithi146 ceph-mon[103278]: pgmap v6964: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:06 smithi078 ceph-mon[99959]: pgmap v6964: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:06 smithi078 ceph-mon[91136]: pgmap v6964: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:08 smithi146 ceph-mon[103278]: pgmap v6965: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:08 smithi078 ceph-mon[91136]: pgmap v6965: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:08 smithi078 ceph-mon[99959]: pgmap v6965: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:10.933 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:10 smithi078 ceph-mon[91136]: pgmap v6966: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:10.933 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:10 smithi078 ceph-mon[99959]: pgmap v6966: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:10.933 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:12:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:12:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:10 smithi146 ceph-mon[103278]: pgmap v6966: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:11.109Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:11.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:11.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:12.762 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:12 smithi146 ceph-mon[103278]: pgmap v6967: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:12:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:12 smithi078 ceph-mon[99959]: pgmap v6967: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:12:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:12 smithi078 ceph-mon[91136]: pgmap v6967: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:12:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:14 smithi146 ceph-mon[103278]: pgmap v6968: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:14 smithi078 ceph-mon[99959]: pgmap v6968: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:14 smithi078 ceph-mon[91136]: pgmap v6968: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:16 smithi146 ceph-mon[103278]: pgmap v6969: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:12:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[91136]: pgmap v6969: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:12:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:16 smithi078 ceph-mon[99959]: pgmap v6969: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T00:12:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:18 smithi146 ceph-mon[103278]: pgmap v6970: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:18 smithi078 ceph-mon[91136]: pgmap v6970: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:18 smithi078 ceph-mon[99959]: pgmap v6970: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:20.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:12:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:12:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[91136]: pgmap v6971: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:20.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[99959]: pgmap v6971: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:20.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:20 smithi146 ceph-mon[103278]: pgmap v6971: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T00:12:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:21.110Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:21.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:21.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:22.629 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:12:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:22 smithi078 ceph-mon[99959]: pgmap v6972: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:22 smithi078 ceph-mon[91136]: pgmap v6972: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:22 smithi146 ceph-mon[103278]: pgmap v6972: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 4h - - 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 4h - - 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 7s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 3s ago 3h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 7s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 3s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:12:23.192 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 7s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 7s ago 4h 409M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 3s ago 4h 265M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 7s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 7s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 3s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 7s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 7s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 7s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 7s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 3s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 3s ago 3h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 3s ago 3h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 3s ago 3h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 3s ago 3h 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:12:23.193 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:12:23.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 3s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:12:23.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:12:23.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 3s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:12:23.529 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:12:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:12:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:12:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:12:23.531 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:12:23.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:23 smithi078 ceph-mon[99959]: from='client.79766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:23.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:23 smithi078 ceph-mon[99959]: from='client.79772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:23.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3092807123' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:23.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:23 smithi078 ceph-mon[91136]: from='client.79766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:23.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:23 smithi078 ceph-mon[91136]: from='client.79772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:23.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3092807123' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:12:23.823 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:12:23.824 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:12:23.824 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:12:23.824 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:12:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:23 smithi146 ceph-mon[103278]: from='client.79766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:23 smithi146 ceph-mon[103278]: from='client.79772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3092807123' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:12:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:24.150 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:24.150 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:24.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[99959]: from='client.59859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:24.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[99959]: pgmap v6973: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:12:24.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[99959]: from='client.59871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:24.854 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1313954296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:24.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[91136]: from='client.59859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:24.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[91136]: pgmap v6973: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:12:24.855 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[91136]: from='client.59871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:24.855 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1313954296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:24 smithi146 ceph-mon[103278]: from='client.59859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:24 smithi146 ceph-mon[103278]: pgmap v6973: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T00:12:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:24 smithi146 ceph-mon[103278]: from='client.59871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1313954296' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:26 smithi146 ceph-mon[103278]: pgmap v6974: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:12:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:26 smithi078 ceph-mon[91136]: pgmap v6974: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:26 smithi078 ceph-mon[99959]: pgmap v6974: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T00:12:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:28 smithi146 ceph-mon[103278]: pgmap v6975: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:28 smithi078 ceph-mon[91136]: pgmap v6975: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:28 smithi078 ceph-mon[99959]: pgmap v6975: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:30.906 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:12:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:12:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:30 smithi146 ceph-mon[103278]: pgmap v6976: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:30 smithi078 ceph-mon[91136]: pgmap v6976: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:30 smithi078 ceph-mon[99959]: pgmap v6976: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:31.111Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:31.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:31.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:32.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:32.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:32 smithi146 ceph-mon[103278]: pgmap v6977: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:32 smithi078 ceph-mon[91136]: pgmap v6977: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:32 smithi078 ceph-mon[99959]: pgmap v6977: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:35.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:34 smithi146 ceph-mon[103278]: pgmap v6978: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:34 smithi078 ceph-mon[91136]: pgmap v6978: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:34 smithi078 ceph-mon[99959]: pgmap v6978: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:36 smithi146 ceph-mon[103278]: pgmap v6979: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[99959]: pgmap v6979: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[91136]: pgmap v6979: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:38 smithi146 ceph-mon[103278]: pgmap v6980: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:38 smithi078 ceph-mon[99959]: pgmap v6980: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:38 smithi078 ceph-mon[91136]: pgmap v6980: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:40.951 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:12:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:12:40.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[91136]: pgmap v6981: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:40.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:40.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:40.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[99959]: pgmap v6981: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:40.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:40 smithi146 ceph-mon[103278]: pgmap v6981: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:41.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:41.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:41.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:42.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:42.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:42 smithi078 ceph-mon[99959]: pgmap v6982: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:42.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:42.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:42 smithi078 ceph-mon[91136]: pgmap v6982: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:42 smithi146 ceph-mon[103278]: pgmap v6982: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:44.929 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:44 smithi078 ceph-mon[99959]: pgmap v6983: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:44.930 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:44 smithi078 ceph-mon[91136]: pgmap v6983: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:44 smithi146 ceph-mon[103278]: pgmap v6983: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:46 smithi146 ceph-mon[103278]: pgmap v6984: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:46 smithi078 ceph-mon[91136]: pgmap v6984: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:46 smithi078 ceph-mon[99959]: pgmap v6984: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:48 smithi146 ceph-mon[103278]: pgmap v6985: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:48 smithi078 ceph-mon[91136]: pgmap v6985: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:48 smithi078 ceph-mon[99959]: pgmap v6985: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:50.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:50 smithi078 ceph-mon[91136]: pgmap v6986: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:50.963 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:12:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:12:50.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:50 smithi078 ceph-mon[99959]: pgmap v6986: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:50 smithi146 ceph-mon[103278]: pgmap v6986: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:51.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:51.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:12:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:12:51.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:12:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:52 smithi078 ceph-mon[91136]: pgmap v6987: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:52 smithi078 ceph-mon[99959]: pgmap v6987: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:12:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:52 smithi146 ceph-mon[103278]: pgmap v6987: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:12:54.443 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:12:54.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:54 smithi146 ceph-mon[103278]: pgmap v6988: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:54 smithi078 ceph-mon[91136]: pgmap v6988: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:54 smithi078 ceph-mon[99959]: pgmap v6988: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:12:54.968 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 4h - - 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 4h - - 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 19s ago 3h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 14s ago 3h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 19s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 14s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 19s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 19s ago 4h 409M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 14s ago 4h 266M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 19s ago 4h 274M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 19s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:12:54.969 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 14s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 19s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 19s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 19s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 19s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 14s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 14s ago 4h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 14s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 3h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 14s ago 3h 68.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 19s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 19s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:12:54.970 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:12:55.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:12:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:12:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:12:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:12:55.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:12:55.276 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:12:55.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:12:55.541 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:12:55.541 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:12:55.541 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:12:55.542 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:12:55.867 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:12:55.867 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:12:55.867 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:12:55.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:55 smithi078 ceph-mon[99959]: from='client.59883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:55.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:55 smithi078 ceph-mon[99959]: from='client.79808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:55.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/167168881' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:55.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:55 smithi078 ceph-mon[91136]: from='client.59883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:55.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:55 smithi078 ceph-mon[91136]: from='client.79808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:55.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/167168881' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:55.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:55 smithi146 ceph-mon[103278]: from='client.59883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:55.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:55 smithi146 ceph-mon[103278]: from='client.79808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:55.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/167168881' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: from='client.79814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: from='client.59907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: pgmap v6989: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1659580483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: from='client.79814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: from='client.59907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: pgmap v6989: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1659580483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: from='client.79814 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: from='client.59907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:12:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: pgmap v6989: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:12:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1659580483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:12:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:58 smithi146 ceph-mon[103278]: pgmap v6990: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:12:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:58 smithi078 ceph-mon[91136]: pgmap v6990: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:12:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:12:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:58 smithi078 ceph-mon[99959]: pgmap v6990: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:12:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:12:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:00.964 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:13:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:13:00.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[99959]: pgmap v6991: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:00.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:00.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:00.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:00.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[91136]: pgmap v6991: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:00.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:00.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:00.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:00 smithi146 ceph-mon[103278]: pgmap v6991: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:01.113Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:01.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:01.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:02 smithi146 ceph-mon[103278]: pgmap v6992: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:13:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:02 smithi078 ceph-mon[91136]: pgmap v6992: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:13:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:02 smithi078 ceph-mon[99959]: pgmap v6992: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:13:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:04 smithi146 ceph-mon[103278]: pgmap v6993: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:04 smithi078 ceph-mon[91136]: pgmap v6993: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:04 smithi078 ceph-mon[99959]: pgmap v6993: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:06 smithi146 ceph-mon[103278]: pgmap v6994: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:06 smithi078 ceph-mon[91136]: pgmap v6994: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:06 smithi078 ceph-mon[99959]: pgmap v6994: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:08 smithi146 ceph-mon[103278]: pgmap v6995: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:08 smithi078 ceph-mon[99959]: pgmap v6995: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:08 smithi078 ceph-mon[91136]: pgmap v6995: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:10.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:10 smithi078 ceph-mon[91136]: pgmap v6996: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:10.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:10 smithi078 ceph-mon[99959]: pgmap v6996: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:10.969 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:13:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:13:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:10 smithi146 ceph-mon[103278]: pgmap v6996: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:11.114Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:11.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:11.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:12 smithi146 ceph-mon[103278]: pgmap v6997: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:12 smithi078 ceph-mon[91136]: pgmap v6997: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:12 smithi078 ceph-mon[99959]: pgmap v6997: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:14 smithi146 ceph-mon[103278]: pgmap v6998: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:14 smithi078 ceph-mon[91136]: pgmap v6998: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:14 smithi078 ceph-mon[99959]: pgmap v6998: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:16 smithi146 ceph-mon[103278]: pgmap v6999: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[91136]: pgmap v6999: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[99959]: pgmap v6999: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:18 smithi146 ceph-mon[103278]: pgmap v7000: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:18 smithi078 ceph-mon[99959]: pgmap v7000: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:18 smithi078 ceph-mon[91136]: pgmap v7000: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:20.870 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:13:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:13:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:20 smithi146 ceph-mon[103278]: pgmap v7001: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[99959]: pgmap v7001: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:21.115Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:21.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:21.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[91136]: pgmap v7001: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:22 smithi146 ceph-mon[103278]: pgmap v7002: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:22 smithi078 ceph-mon[91136]: pgmap v7002: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:22 smithi078 ceph-mon[99959]: pgmap v7002: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:24 smithi078 ceph-mon[91136]: pgmap v7003: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:24 smithi078 ceph-mon[99959]: pgmap v7003: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:24 smithi146 ceph-mon[103278]: pgmap v7003: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:26.166 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:13:26.730 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:13:26.730 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 4h - - 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 4h - - 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 10s ago 3h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 6s ago 3h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 10s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 6s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 10s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 10s ago 4h 410M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 6s ago 4h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 10s ago 4h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 10s ago 3h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 6s ago 3h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 10s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 10s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 10s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 10s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:13:26.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 6s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 6s ago 4h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 6s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 6s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 6s ago 3h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 10s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 6s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 10s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:13:26.732 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 6s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:13:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:26 smithi078 ceph-mon[99959]: pgmap v7004: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:26 smithi078 ceph-mon[91136]: pgmap v7004: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:27.046 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:13:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:13:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:13:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:13:27.048 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:13:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:26 smithi146 ceph-mon[103278]: pgmap v7004: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:13:27.312 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:13:27.313 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:13:27.313 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:13:27.619 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:13:27.619 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:27.619 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:27.619 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:27 smithi146 ceph-mon[103278]: from='client.79838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:27 smithi146 ceph-mon[103278]: from='client.59925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:27 smithi146 ceph-mon[103278]: from='client.59931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2325850829' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3644049868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[99959]: from='client.79838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[99959]: from='client.59925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[99959]: from='client.59931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2325850829' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3644049868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[91136]: from='client.79838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[91136]: from='client.59925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[91136]: from='client.59931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2325850829' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:13:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3644049868' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:13:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:28 smithi146 ceph-mon[103278]: from='client.79862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:28 smithi146 ceph-mon[103278]: pgmap v7005: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:28 smithi078 ceph-mon[99959]: from='client.79862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:28 smithi078 ceph-mon[99959]: pgmap v7005: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:28 smithi078 ceph-mon[91136]: from='client.79862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:28 smithi078 ceph-mon[91136]: pgmap v7005: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:30.893 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:13:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:13:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:30 smithi146 ceph-mon[103278]: pgmap v7006: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:30 smithi078 ceph-mon[91136]: pgmap v7006: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:30 smithi078 ceph-mon[99959]: pgmap v7006: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:31.116Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:31.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:31.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:32 smithi146 ceph-mon[103278]: pgmap v7007: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:32 smithi078 ceph-mon[91136]: pgmap v7007: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:32 smithi078 ceph-mon[99959]: pgmap v7007: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:34 smithi146 ceph-mon[103278]: pgmap v7008: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:34 smithi078 ceph-mon[99959]: pgmap v7008: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:34 smithi078 ceph-mon[91136]: pgmap v7008: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:37.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:36 smithi146 ceph-mon[103278]: pgmap v7009: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:37.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[91136]: pgmap v7009: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[99959]: pgmap v7009: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:38 smithi146 ceph-mon[103278]: pgmap v7010: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:38 smithi078 ceph-mon[91136]: pgmap v7010: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:38 smithi078 ceph-mon[99959]: pgmap v7010: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:40.912 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:13:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:13:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:40 smithi146 ceph-mon[103278]: pgmap v7011: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[99959]: pgmap v7011: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[91136]: pgmap v7011: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:41.117Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:41.117Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:41.118Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:42 smithi146 ceph-mon[103278]: pgmap v7012: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:13:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:42 smithi078 ceph-mon[91136]: pgmap v7012: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:13:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:42 smithi078 ceph-mon[99959]: pgmap v7012: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:13:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:45.205 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:44 smithi078 ceph-mon[99959]: pgmap v7013: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:45.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:44 smithi078 ceph-mon[91136]: pgmap v7013: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:44 smithi146 ceph-mon[103278]: pgmap v7013: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:47.045 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:46 smithi078 ceph-mon[99959]: pgmap v7014: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:13:47.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:46 smithi078 ceph-mon[91136]: pgmap v7014: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:13:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:46 smithi146 ceph-mon[103278]: pgmap v7014: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:13:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:48 smithi146 ceph-mon[103278]: pgmap v7015: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:48 smithi078 ceph-mon[91136]: pgmap v7015: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:48 smithi078 ceph-mon[99959]: pgmap v7015: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:50.946 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:13:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:13:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:50 smithi146 ceph-mon[103278]: pgmap v7016: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:50 smithi078 ceph-mon[91136]: pgmap v7016: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:50 smithi078 ceph-mon[99959]: pgmap v7016: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:51.118Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:51.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:13:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:13:51.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:13:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:51 smithi146 ceph-mon[103278]: pgmap v7017: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-mon[91136]: pgmap v7017: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:13:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:51 smithi078 ceph-mon[99959]: pgmap v7017: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:54 smithi146 ceph-mon[103278]: pgmap v7018: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:54 smithi078 ceph-mon[99959]: pgmap v7018: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:54 smithi078 ceph-mon[91136]: pgmap v7018: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:13:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:56 smithi146 ceph-mon[103278]: pgmap v7019: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[91136]: pgmap v7019: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[99959]: pgmap v7019: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:13:57.919 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:13:58.451 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:13:58.451 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 4h - - 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 4h - - 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 2s ago 3h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (3h) 17s ago 3h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 17s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 4h 411M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 17s ago 4h 268M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 2s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 17s ago 3h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:13:58.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 2s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 2s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 2s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 17s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 17s ago 4h 77.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 17s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 17s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 17s ago 4h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 2s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 17s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 2s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:13:58.453 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 17s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:13:58.763 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:13:58.763 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:13:58.763 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:13:58.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:58.763 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:13:58.763 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:13:58.764 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:13:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:58 smithi146 ceph-mon[103278]: pgmap v7020: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:58 smithi146 ceph-mon[103278]: from='client.79874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:13:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:13:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:13:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:13:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:13:59.032 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:13:59.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:58 smithi078 ceph-mon[99959]: pgmap v7020: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:59.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:58 smithi078 ceph-mon[99959]: from='client.79874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:59.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:59.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:58 smithi078 ceph-mon[91136]: pgmap v7020: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:13:59.032 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:58 smithi078 ceph-mon[91136]: from='client.79874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:13:59.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:13:59.343 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:13:59.343 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:59.343 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:13:59.343 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:13:59.343 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:59 smithi146 ceph-mon[103278]: from='client.79880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:59 smithi146 ceph-mon[103278]: from='client.59967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4250178693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:13:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1331587940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[91136]: from='client.79880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[91136]: from='client.59967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4250178693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1331587940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[99959]: from='client.79880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[99959]: from='client.59967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4250178693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:13:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1331587940' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:00 smithi146 ceph-mon[103278]: from='client.79898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:00 smithi146 ceph-mon[103278]: pgmap v7021: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:01.021 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:14:00] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-21T00:14:01.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:00 smithi078 ceph-mon[99959]: from='client.79898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:01.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:00 smithi078 ceph-mon[99959]: pgmap v7021: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:01.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:00 smithi078 ceph-mon[91136]: from='client.79898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:01.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:00 smithi078 ceph-mon[91136]: pgmap v7021: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:01.118Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:01.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:01.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:02 smithi078 ceph-mon[99959]: pgmap v7022: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:02 smithi078 ceph-mon[91136]: pgmap v7022: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:02 smithi146 ceph-mon[103278]: pgmap v7022: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:04 smithi078 ceph-mon[91136]: pgmap v7023: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:04 smithi078 ceph-mon[99959]: pgmap v7023: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:05.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:04 smithi146 ceph-mon[103278]: pgmap v7023: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:06 smithi078 ceph-mon[91136]: pgmap v7024: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:06 smithi078 ceph-mon[99959]: pgmap v7024: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:07.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:06 smithi146 ceph-mon[103278]: pgmap v7024: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:08 smithi078 ceph-mon[91136]: pgmap v7025: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:08 smithi078 ceph-mon[99959]: pgmap v7025: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:08 smithi146 ceph-mon[103278]: pgmap v7025: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:11.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:10 smithi078 ceph-mon[91136]: pgmap v7026: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:11.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:10 smithi078 ceph-mon[99959]: pgmap v7026: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:11.031 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:14:10] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-21T00:14:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:10 smithi146 ceph-mon[103278]: pgmap v7026: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:11.119Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:11.119Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:11.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:12 smithi146 ceph-mon[103278]: pgmap v7027: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:12 smithi078 ceph-mon[91136]: pgmap v7027: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:12 smithi078 ceph-mon[99959]: pgmap v7027: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:14 smithi146 ceph-mon[103278]: pgmap v7028: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:14 smithi078 ceph-mon[99959]: pgmap v7028: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:14 smithi078 ceph-mon[91136]: pgmap v7028: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:16 smithi146 ceph-mon[103278]: pgmap v7029: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[91136]: pgmap v7029: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[99959]: pgmap v7029: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:19.162 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:18 smithi146 ceph-mon[103278]: pgmap v7030: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:18 smithi078 ceph-mon[99959]: pgmap v7030: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:18 smithi078 ceph-mon[91136]: pgmap v7030: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:20 smithi078 ceph-mon[91136]: pgmap v7031: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:20 smithi078 ceph-mon[99959]: pgmap v7031: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:14:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:14:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:20 smithi146 ceph-mon[103278]: pgmap v7031: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:21.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:21.120Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:21.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:21.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:21.497 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:21.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:22.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:22 smithi146 ceph-mon[103278]: pgmap v7032: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:22 smithi078 ceph-mon[91136]: pgmap v7032: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:22 smithi078 ceph-mon[99959]: pgmap v7032: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:24 smithi146 ceph-mon[103278]: pgmap v7033: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:24 smithi078 ceph-mon[91136]: pgmap v7033: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:24 smithi078 ceph-mon[99959]: pgmap v7033: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 00:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T00:14:25.847218458Z level=info msg="Completed cleanup jobs" duration=12.125631ms 2024-09-21T00:14:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:26 smithi078 ceph-mon[91136]: pgmap v7034: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:26 smithi078 ceph-mon[99959]: pgmap v7034: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:26 smithi146 ceph-mon[103278]: pgmap v7034: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:28 smithi146 ceph-mon[103278]: pgmap v7035: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:28 smithi078 ceph-mon[91136]: pgmap v7035: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:28 smithi078 ceph-mon[99959]: pgmap v7035: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:29.629 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 4h - - 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 4h - - 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 13s ago 4h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 9s ago 4h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 13s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 9s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 13s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:14:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 13s ago 4h 412M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 9s ago 4h 267M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 13s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (3h) 13s ago 3h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (3h) 9s ago 3h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 13s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 13s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 13s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 13s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 9s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 9s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 9s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 9s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 9s ago 4h 73.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:14:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:14:30.159 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 9s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:14:30.159 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:14:30.159 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 9s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:14:30.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:14:30.467 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:14:30.739 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:14:30.740 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:14:30.741 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:14:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[91136]: pgmap v7036: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[91136]: from='client.59991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[91136]: from='client.79916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4239047270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[99959]: pgmap v7036: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[99959]: from='client.59991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[99959]: from='client.79916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4239047270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:31.050 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:14:31.050 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:14:31.050 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:14:31.050 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:31.050 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:30 smithi146 ceph-mon[103278]: pgmap v7036: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:30 smithi146 ceph-mon[103278]: from='client.59991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:30 smithi146 ceph-mon[103278]: from='client.79916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4239047270' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:14:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:31.120Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:31.120Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:31.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:31 smithi146 ceph-mon[103278]: from='client.79922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:31 smithi146 ceph-mon[103278]: from='client.60015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1908633406' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-mon[99959]: from='client.79922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-mon[99959]: from='client.60015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1908633406' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-mon[91136]: from='client.79922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-mon[91136]: from='client.60015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:14:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1908633406' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:14:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:32 smithi146 ceph-mon[103278]: pgmap v7037: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:32 smithi078 ceph-mon[91136]: pgmap v7037: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:32 smithi078 ceph-mon[99959]: pgmap v7037: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:34 smithi146 ceph-mon[103278]: pgmap v7038: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:34 smithi078 ceph-mon[91136]: pgmap v7038: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:34 smithi078 ceph-mon[99959]: pgmap v7038: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:37.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:36 smithi146 ceph-mon[103278]: pgmap v7039: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:37.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[91136]: pgmap v7039: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[99959]: pgmap v7039: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:38 smithi146 ceph-mon[103278]: pgmap v7040: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:38 smithi078 ceph-mon[91136]: pgmap v7040: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:38 smithi078 ceph-mon[99959]: pgmap v7040: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:40.940 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:14:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:14:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:40 smithi146 ceph-mon[103278]: pgmap v7041: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:40 smithi078 ceph-mon[91136]: pgmap v7041: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:40 smithi078 ceph-mon[99959]: pgmap v7041: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:41.121Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:41.121Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:41.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[99959]: pgmap v7042: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:42 smithi078 ceph-mon[91136]: pgmap v7042: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:42 smithi146 ceph-mon[103278]: pgmap v7042: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:44 smithi146 ceph-mon[103278]: pgmap v7043: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:44.780 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:44 smithi078 ceph-mon[99959]: pgmap v7043: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:44.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:44 smithi078 ceph-mon[91136]: pgmap v7043: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:46 smithi146 ceph-mon[103278]: pgmap v7044: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:46 smithi078 ceph-mon[99959]: pgmap v7044: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:46 smithi078 ceph-mon[91136]: pgmap v7044: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:48.883 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:48 smithi078 ceph-mon[99959]: pgmap v7045: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:48.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:48 smithi078 ceph-mon[91136]: pgmap v7045: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:48 smithi146 ceph-mon[103278]: pgmap v7045: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:50 smithi146 ceph-mon[103278]: pgmap v7046: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:50 smithi078 ceph-mon[91136]: pgmap v7046: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:14:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:14:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:50 smithi078 ceph-mon[99959]: pgmap v7046: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:51.492 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:51.121Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:51.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:51.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:14:51.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:14:51.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:14:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:52 smithi146 ceph-mon[103278]: pgmap v7047: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:52 smithi078 ceph-mon[91136]: pgmap v7047: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:14:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:52 smithi078 ceph-mon[99959]: pgmap v7047: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:54 smithi146 ceph-mon[103278]: pgmap v7048: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:54 smithi078 ceph-mon[91136]: pgmap v7048: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:54 smithi078 ceph-mon[99959]: pgmap v7048: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:14:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[91136]: pgmap v7049: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[99959]: pgmap v7049: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:56.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:56 smithi146 ceph-mon[103278]: pgmap v7049: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:14:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:58 smithi146 ceph-mon[103278]: pgmap v7050: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:58 smithi078 ceph-mon[99959]: pgmap v7050: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:14:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:58 smithi078 ceph-mon[91136]: pgmap v7050: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:14:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:00 smithi146 ceph-mon[103278]: pgmap v7051: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:00 smithi078 ceph-mon[91136]: pgmap v7051: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:00 smithi078 ceph-mon[99959]: pgmap v7051: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:15:00] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:15:01.347 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:01.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:01.347 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:01.122Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:01.347 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:01.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:01.347 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:15:01.878 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:15:01.878 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 4h - - 2024-09-21T00:15:01.878 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 4h - - 2024-09-21T00:15:01.878 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (3h) 5s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:15:01.878 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 0s ago 4h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:15:01.878 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 0s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 5s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 5s ago 4h 412M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 0s ago 4h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 5s ago 4h 277M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 5s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 0s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 5s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 5s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 5s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 5s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 0s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 0s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:15:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 0s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:15:01.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 0s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:15:01.880 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 0s ago 4h 73.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:15:01.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 5s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:15:01.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 0s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:15:01.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 5s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:15:01.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 0s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:15:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:15:02.193 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:15:02.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:15:02.468 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:15:02.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:02.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='client.60027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='client.60033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: pgmap v7052: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[99959]: from='client.60039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='client.60027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='client.60033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: pgmap v7052: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:02.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:02 smithi078 ceph-mon[91136]: from='client.60039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='client.60027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='client.60033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: pgmap v7052: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:02 smithi146 ceph-mon[103278]: from='client.60039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:02.774 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:15:02.774 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:15:02.775 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:15:02.775 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:15:02.775 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:15:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1813354333' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:03 smithi146 ceph-mon[103278]: from='client.60051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3790112278' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1813354333' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:03 smithi078 ceph-mon[99959]: from='client.60051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3790112278' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1813354333' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:03 smithi078 ceph-mon[91136]: from='client.60051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3790112278' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:04 smithi146 ceph-mon[103278]: pgmap v7053: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:04 smithi078 ceph-mon[91136]: pgmap v7053: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:04 smithi078 ceph-mon[99959]: pgmap v7053: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:06 smithi146 ceph-mon[103278]: pgmap v7054: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:06 smithi078 ceph-mon[91136]: pgmap v7054: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:06 smithi078 ceph-mon[99959]: pgmap v7054: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:09.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:08 smithi078 ceph-mon[99959]: pgmap v7055: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:09.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:08 smithi078 ceph-mon[91136]: pgmap v7055: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:08 smithi146 ceph-mon[103278]: pgmap v7055: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:10 smithi078 ceph-mon[91136]: pgmap v7056: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:15:10] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:15:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:10 smithi078 ceph-mon[99959]: pgmap v7056: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:10 smithi146 ceph-mon[103278]: pgmap v7056: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:11.122Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:11.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:11.123Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:12 smithi146 ceph-mon[103278]: pgmap v7057: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:12 smithi078 ceph-mon[99959]: pgmap v7057: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:12 smithi078 ceph-mon[91136]: pgmap v7057: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:14 smithi146 ceph-mon[103278]: pgmap v7058: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:14 smithi078 ceph-mon[99959]: pgmap v7058: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:14 smithi078 ceph-mon[91136]: pgmap v7058: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:16 smithi146 ceph-mon[103278]: pgmap v7059: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[91136]: pgmap v7059: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[99959]: pgmap v7059: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:18.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:18 smithi146 ceph-mon[103278]: pgmap v7060: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:18 smithi078 ceph-mon[91136]: pgmap v7060: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:18 smithi078 ceph-mon[99959]: pgmap v7060: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:20 smithi146 ceph-mon[103278]: pgmap v7061: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:20 smithi078 ceph-mon[91136]: pgmap v7061: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:20 smithi078 ceph-mon[99959]: pgmap v7061: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:15:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:15:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:21.124Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:21.124Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:21.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[91136]: pgmap v7062: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:22 smithi078 ceph-mon[99959]: pgmap v7062: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:22 smithi146 ceph-mon[103278]: pgmap v7062: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:24 smithi146 ceph-mon[103278]: pgmap v7063: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:24 smithi078 ceph-mon[99959]: pgmap v7063: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:24 smithi078 ceph-mon[91136]: pgmap v7063: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:26 smithi146 ceph-mon[103278]: pgmap v7064: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:26 smithi078 ceph-mon[99959]: pgmap v7064: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:26 smithi078 ceph-mon[91136]: pgmap v7064: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:28 smithi146 ceph-mon[103278]: pgmap v7065: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:28 smithi078 ceph-mon[91136]: pgmap v7065: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:28 smithi078 ceph-mon[99959]: pgmap v7065: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:30 smithi146 ceph-mon[103278]: pgmap v7066: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:30 smithi078 ceph-mon[91136]: pgmap v7066: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:15:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:15:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:30 smithi078 ceph-mon[99959]: pgmap v7066: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:31.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:31.125Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:31.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:31.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:31.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:31.125Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:32 smithi146 ceph-mon[103278]: pgmap v7067: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:32 smithi078 ceph-mon[99959]: pgmap v7067: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:32 smithi078 ceph-mon[91136]: pgmap v7067: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:33.066 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:15:33.595 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:15:33.595 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 4h - - 2024-09-21T00:15:33.595 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 4h - - 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 16s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 12s ago 4h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 16s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 16s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 16s ago 4h 413M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 4h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 16s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 16s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 12s ago 4h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 16s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 16s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 16s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:15:33.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 16s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 12s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 12s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 12s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 12s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 4h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 16s ago 3h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 12s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 16s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:15:33.597 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 12s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:15:33.911 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:15:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:15:33.911 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:15:33.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:15:33.913 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:15:34.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:15:34.180 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:15:34.493 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:15:34.493 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:15:34.493 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:15:34.494 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:15:34.494 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:15:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[91136]: from='client.60063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[91136]: from='client.79988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[91136]: from='client.60075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[91136]: pgmap v7068: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2917121885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2389143123' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[99959]: from='client.60063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[99959]: from='client.79988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[99959]: from='client.60075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[99959]: pgmap v7068: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2917121885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2389143123' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:34 smithi146 ceph-mon[103278]: from='client.60063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:34 smithi146 ceph-mon[103278]: from='client.79988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:34 smithi146 ceph-mon[103278]: from='client.60075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:34 smithi146 ceph-mon[103278]: pgmap v7068: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2917121885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:15:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2389143123' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:15:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:35 smithi146 ceph-mon[103278]: from='client.80006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:35 smithi078 ceph-mon[91136]: from='client.80006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:35 smithi078 ceph-mon[99959]: from='client.80006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:15:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:36 smithi146 ceph-mon[103278]: pgmap v7069: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:36 smithi078 ceph-mon[91136]: pgmap v7069: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:36 smithi078 ceph-mon[99959]: pgmap v7069: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:38.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:39.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:38 smithi146 ceph-mon[103278]: pgmap v7070: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:38 smithi078 ceph-mon[91136]: pgmap v7070: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:38 smithi078 ceph-mon[99959]: pgmap v7070: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:40 smithi078 ceph-mon[99959]: pgmap v7071: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:40 smithi078 ceph-mon[91136]: pgmap v7071: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:15:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:15:41.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:40 smithi146 ceph-mon[103278]: pgmap v7071: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:41.125Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:41.126Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:41.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:42 smithi146 ceph-mon[103278]: pgmap v7072: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[91136]: pgmap v7072: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:42 smithi078 ceph-mon[99959]: pgmap v7072: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:44 smithi146 ceph-mon[103278]: pgmap v7073: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:44 smithi078 ceph-mon[91136]: pgmap v7073: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:44 smithi078 ceph-mon[99959]: pgmap v7073: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:46 smithi146 ceph-mon[103278]: pgmap v7074: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:46 smithi078 ceph-mon[91136]: pgmap v7074: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:46 smithi078 ceph-mon[99959]: pgmap v7074: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:48 smithi146 ceph-mon[103278]: pgmap v7075: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:48 smithi078 ceph-mon[91136]: pgmap v7075: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:48 smithi078 ceph-mon[99959]: pgmap v7075: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:50.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:50 smithi078 ceph-mon[91136]: pgmap v7076: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:50.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:15:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:15:50.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:50 smithi078 ceph-mon[99959]: pgmap v7076: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:50 smithi146 ceph-mon[103278]: pgmap v7076: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:51.233 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:51.126Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:51.234 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:51.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:15:51.234 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:15:51.127Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:15:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:52 smithi146 ceph-mon[103278]: pgmap v7077: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:52 smithi078 ceph-mon[91136]: pgmap v7077: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:15:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:52 smithi078 ceph-mon[99959]: pgmap v7077: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:54 smithi146 ceph-mon[103278]: pgmap v7078: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:54 smithi078 ceph-mon[91136]: pgmap v7078: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:54 smithi078 ceph-mon[99959]: pgmap v7078: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:15:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:56 smithi146 ceph-mon[103278]: pgmap v7079: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:56 smithi078 ceph-mon[91136]: pgmap v7079: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:56 smithi078 ceph-mon[99959]: pgmap v7079: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:15:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:58 smithi146 ceph-mon[103278]: pgmap v7080: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:59.257 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:15:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:58 smithi078 ceph-mon[91136]: pgmap v7080: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:15:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:15:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:58 smithi078 ceph-mon[99959]: pgmap v7080: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:15:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:15:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:00.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:16:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:16:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:00 smithi146 ceph-mon[103278]: pgmap v7081: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:00 smithi078 ceph-mon[91136]: pgmap v7081: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:00 smithi078 ceph-mon[99959]: pgmap v7081: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:01.127Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:01.128Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:01.128Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:02 smithi146 ceph-mon[103278]: pgmap v7082: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[91136]: pgmap v7082: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:02 smithi078 ceph-mon[99959]: pgmap v7082: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:04.790 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:16:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:04 smithi146 ceph-mon[103278]: pgmap v7083: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:04 smithi078 ceph-mon[91136]: pgmap v7083: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:04 smithi078 ceph-mon[99959]: pgmap v7083: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:05.317 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 4h - - 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 4h - - 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 8s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 3s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 8s ago 3h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 3s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 8s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 8s ago 4h 414M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 3s ago 4h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 8s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:16:05.318 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 8s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 3s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 8s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 8s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 8s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 8s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 3s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 3s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 3s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 3s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 3s ago 4h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 8s ago 4h 89.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 8s ago 3h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:16:05.319 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 3s ago 3h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:16:05.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:16:05.628 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:16:05.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:05 smithi078 ceph-mon[91136]: from='client.60099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:05.920 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/595533042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:05.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:05 smithi078 ceph-mon[99959]: from='client.60099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:05.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/595533042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:05.921 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:16:05.921 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:16:05.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:16:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:16:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:16:05.923 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:16:05.923 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:16:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:05 smithi146 ceph-mon[103278]: from='client.60099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/595533042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:06.234 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:16:06.234 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:06.234 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:06.234 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:06 smithi146 ceph-mon[103278]: from='client.80024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:06 smithi146 ceph-mon[103278]: from='client.80030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:06 smithi146 ceph-mon[103278]: pgmap v7084: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:06 smithi146 ceph-mon[103278]: from='client.60123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3745932223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[91136]: from='client.80024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[91136]: from='client.80030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[91136]: pgmap v7084: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[91136]: from='client.60123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3745932223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[99959]: from='client.80024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[99959]: from='client.80030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[99959]: pgmap v7084: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[99959]: from='client.60123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3745932223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:08 smithi146 ceph-mon[103278]: pgmap v7085: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:08 smithi078 ceph-mon[91136]: pgmap v7085: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:08 smithi078 ceph-mon[99959]: pgmap v7085: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:10.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:10 smithi078 ceph-mon[91136]: pgmap v7086: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:10.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:10 smithi078 ceph-mon[99959]: pgmap v7086: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:10.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:16:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:16:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:10 smithi146 ceph-mon[103278]: pgmap v7086: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:11.128Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:11.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:11.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:12 smithi146 ceph-mon[103278]: pgmap v7087: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:12 smithi078 ceph-mon[99959]: pgmap v7087: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:12 smithi078 ceph-mon[91136]: pgmap v7087: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:14 smithi146 ceph-mon[103278]: pgmap v7088: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:14 smithi078 ceph-mon[99959]: pgmap v7088: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:14 smithi078 ceph-mon[91136]: pgmap v7088: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:16 smithi146 ceph-mon[103278]: pgmap v7089: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:16 smithi078 ceph-mon[99959]: pgmap v7089: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:16 smithi078 ceph-mon[91136]: pgmap v7089: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:18 smithi146 ceph-mon[103278]: pgmap v7090: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[99959]: pgmap v7090: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:18 smithi078 ceph-mon[91136]: pgmap v7090: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:20.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:20 smithi146 ceph-mon[103278]: pgmap v7091: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:20 smithi078 ceph-mon[91136]: pgmap v7091: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:20 smithi078 ceph-mon[99959]: pgmap v7091: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:16:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:16:21.488 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:21.129Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:21.488 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:21.129Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:21.488 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:21.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:22 smithi146 ceph-mon[103278]: pgmap v7092: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[91136]: pgmap v7092: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[99959]: pgmap v7092: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:24 smithi146 ceph-mon[103278]: pgmap v7093: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:24 smithi078 ceph-mon[91136]: pgmap v7093: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:24 smithi078 ceph-mon[99959]: pgmap v7093: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:26 smithi146 ceph-mon[103278]: pgmap v7094: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:26 smithi078 ceph-mon[91136]: pgmap v7094: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:26 smithi078 ceph-mon[99959]: pgmap v7094: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:28 smithi146 ceph-mon[103278]: pgmap v7095: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:28 smithi078 ceph-mon[91136]: pgmap v7095: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:28 smithi078 ceph-mon[99959]: pgmap v7095: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:30.946 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:16:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:16:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:30 smithi146 ceph-mon[103278]: pgmap v7096: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:30 smithi078 ceph-mon[91136]: pgmap v7096: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:30 smithi078 ceph-mon[99959]: pgmap v7096: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:31.129Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:31.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:31.130Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:33.235 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:33.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:32 smithi146 ceph-mon[103278]: pgmap v7097: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:32 smithi078 ceph-mon[99959]: pgmap v7097: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:32 smithi078 ceph-mon[91136]: pgmap v7097: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:34 smithi146 ceph-mon[103278]: pgmap v7098: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:34 smithi078 ceph-mon[91136]: pgmap v7098: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:34 smithi078 ceph-mon[99959]: pgmap v7098: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:36.531 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 20s ago 4h - - 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 4h - - 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 20s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 15s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 20s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 20s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 20s ago 4h 415M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:16:37.055 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 4h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 20s ago 4h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 20s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 15s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 20s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 20s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 20s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 20s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 15s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 15s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 15s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 4h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 20s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:16:37.056 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 20s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:16:37.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:16:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:36 smithi146 ceph-mon[103278]: pgmap v7099: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:37.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:36 smithi078 ceph-mon[91136]: pgmap v7099: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:36 smithi078 ceph-mon[99959]: pgmap v7099: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:37.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:16:37.380 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:16:37.380 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:16:37.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:37.380 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:16:37.380 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:16:37.381 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:16:37.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:16:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:16:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:16:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:16:37.647 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:16:37.648 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:16:37.957 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:16:37.958 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:37.958 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:16:37.958 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:37.958 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:16:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:37 smithi146 ceph-mon[103278]: from='client.60135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:37 smithi146 ceph-mon[103278]: from='client.60141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2895782063' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[91136]: from='client.60135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[91136]: from='client.60141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2895782063' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[99959]: from='client.60135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[99959]: from='client.60141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2895782063' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:16:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:38 smithi146 ceph-mon[103278]: from='client.80066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:38 smithi146 ceph-mon[103278]: from='client.60159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:38 smithi146 ceph-mon[103278]: pgmap v7100: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2856347066' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[91136]: from='client.80066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[91136]: from='client.60159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[91136]: pgmap v7100: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2856347066' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[99959]: from='client.80066 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[99959]: from='client.60159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[99959]: pgmap v7100: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2856347066' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:16:40.962 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:16:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:16:41.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:40 smithi146 ceph-mon[103278]: pgmap v7101: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:40 smithi078 ceph-mon[91136]: pgmap v7101: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:40 smithi078 ceph-mon[99959]: pgmap v7101: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:41.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:41.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:41.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:42 smithi146 ceph-mon[103278]: pgmap v7102: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[91136]: pgmap v7102: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[99959]: pgmap v7102: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:44 smithi078 ceph-mon[91136]: pgmap v7103: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:44 smithi078 ceph-mon[99959]: pgmap v7103: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:44 smithi146 ceph-mon[103278]: pgmap v7103: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:46 smithi146 ceph-mon[103278]: pgmap v7104: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:46 smithi078 ceph-mon[91136]: pgmap v7104: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:46 smithi078 ceph-mon[99959]: pgmap v7104: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:48 smithi146 ceph-mon[103278]: pgmap v7105: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:48 smithi078 ceph-mon[91136]: pgmap v7105: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:48 smithi078 ceph-mon[99959]: pgmap v7105: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:50 smithi078 ceph-mon[91136]: pgmap v7106: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:50 smithi078 ceph-mon[99959]: pgmap v7106: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:16:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:16:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:50 smithi146 ceph-mon[103278]: pgmap v7106: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:51.486 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:51.130Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:51.486 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:51.131Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:16:51.487 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:16:51.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:16:53.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:53.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:52 smithi078 ceph-mon[99959]: pgmap v7107: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:53.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:53.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:52 smithi078 ceph-mon[91136]: pgmap v7107: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:16:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:52 smithi146 ceph-mon[103278]: pgmap v7107: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:54 smithi146 ceph-mon[103278]: pgmap v7108: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:54 smithi078 ceph-mon[91136]: pgmap v7108: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:54 smithi078 ceph-mon[99959]: pgmap v7108: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:16:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:56 smithi146 ceph-mon[103278]: pgmap v7109: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:56 smithi078 ceph-mon[91136]: pgmap v7109: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:56 smithi078 ceph-mon[99959]: pgmap v7109: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:58 smithi146 ceph-mon[103278]: pgmap v7110: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[91136]: pgmap v7110: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:58 smithi078 ceph-mon[99959]: pgmap v7110: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:16:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:16:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:00.585 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:00 smithi146 ceph-mon[103278]: pgmap v7111: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:00 smithi078 ceph-mon[91136]: pgmap v7111: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:00.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:00 smithi078 ceph-mon[99959]: pgmap v7111: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:17:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:17:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:01.131Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:01.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:01.132Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:02 smithi146 ceph-mon[103278]: pgmap v7112: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[91136]: pgmap v7112: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[99959]: pgmap v7112: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:04 smithi146 ceph-mon[103278]: pgmap v7113: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:04 smithi078 ceph-mon[91136]: pgmap v7113: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:04 smithi078 ceph-mon[99959]: pgmap v7113: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:06 smithi146 ceph-mon[103278]: pgmap v7114: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:06 smithi078 ceph-mon[91136]: pgmap v7114: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:06 smithi078 ceph-mon[99959]: pgmap v7114: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:08.252 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:17:08.781 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:17:08.781 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 4h - - 2024-09-21T00:17:08.781 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 4h - - 2024-09-21T00:17:08.781 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 11s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:17:08.781 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 6s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 11s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 6s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 11s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 11s ago 4h 416M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 6s ago 4h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 11s ago 4h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 11s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 6s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 11s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 11s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 11s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:17:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 11s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 6s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 6s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 6s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 6s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 6s ago 4h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 11s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 11s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:17:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:17:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:08 smithi078 ceph-mon[91136]: pgmap v7115: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:08 smithi078 ceph-mon[99959]: pgmap v7115: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:17:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:17:09.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:17:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:08 smithi146 ceph-mon[103278]: pgmap v7115: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:17:09.363 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:17:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:17:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:17:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:17:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:17:09.364 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:17:09.364 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:17:09.672 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:17:09.672 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:09.672 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:09 smithi146 ceph-mon[103278]: from='client.60171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:09 smithi146 ceph-mon[103278]: from='client.80096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:09 smithi146 ceph-mon[103278]: from='client.60183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2859563279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1025789168' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[91136]: from='client.60171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[91136]: from='client.80096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[91136]: from='client.60183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2859563279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1025789168' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[99959]: from='client.60171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[99959]: from='client.80096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[99959]: from='client.60183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2859563279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1025789168' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:10.973 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:17:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:17:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:10 smithi146 ceph-mon[103278]: from='client.60195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:10 smithi146 ceph-mon[103278]: pgmap v7116: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:10 smithi078 ceph-mon[91136]: from='client.60195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:10 smithi078 ceph-mon[91136]: pgmap v7116: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:10 smithi078 ceph-mon[99959]: from='client.60195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:10 smithi078 ceph-mon[99959]: pgmap v7116: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:11.132Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:11.133Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:11.134Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:11 smithi146 ceph-mon[103278]: pgmap v7117: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-mon[91136]: pgmap v7117: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:11 smithi078 ceph-mon[99959]: pgmap v7117: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:14 smithi146 ceph-mon[103278]: pgmap v7118: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:14.774 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:14 smithi078 ceph-mon[91136]: pgmap v7118: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:14.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:14 smithi078 ceph-mon[99959]: pgmap v7118: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:16.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:16 smithi078 ceph-mon[99959]: pgmap v7119: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:16.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:16 smithi078 ceph-mon[91136]: pgmap v7119: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:16 smithi146 ceph-mon[103278]: pgmap v7119: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:18 smithi146 ceph-mon[103278]: pgmap v7120: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[91136]: pgmap v7120: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:18 smithi078 ceph-mon[99959]: pgmap v7120: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:20.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:20 smithi146 ceph-mon[103278]: pgmap v7121: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:20 smithi078 ceph-mon[99959]: pgmap v7121: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:20 smithi078 ceph-mon[91136]: pgmap v7121: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:17:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:17:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:21.134Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:21.134Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:21.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:22 smithi146 ceph-mon[103278]: pgmap v7122: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[99959]: pgmap v7122: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[91136]: pgmap v7122: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:24 smithi146 ceph-mon[103278]: pgmap v7123: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:24 smithi078 ceph-mon[91136]: pgmap v7123: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:24 smithi078 ceph-mon[99959]: pgmap v7123: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:26 smithi146 ceph-mon[103278]: pgmap v7124: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:26 smithi078 ceph-mon[91136]: pgmap v7124: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:26 smithi078 ceph-mon[99959]: pgmap v7124: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:28 smithi146 ceph-mon[103278]: pgmap v7125: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:28 smithi078 ceph-mon[91136]: pgmap v7125: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:28 smithi078 ceph-mon[99959]: pgmap v7125: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:30 smithi146 ceph-mon[103278]: pgmap v7126: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:30 smithi078 ceph-mon[91136]: pgmap v7126: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:30 smithi078 ceph-mon[99959]: pgmap v7126: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:17:30] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:17:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:31.134Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:31.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:31.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:32 smithi146 ceph-mon[103278]: pgmap v7127: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:32 smithi078 ceph-mon[99959]: pgmap v7127: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:32 smithi078 ceph-mon[91136]: pgmap v7127: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:34.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:34 smithi078 ceph-mon[91136]: pgmap v7128: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:34.822 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:34 smithi078 ceph-mon[99959]: pgmap v7128: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:34 smithi146 ceph-mon[103278]: pgmap v7128: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:36 smithi146 ceph-mon[103278]: pgmap v7129: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:36 smithi078 ceph-mon[91136]: pgmap v7129: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:36 smithi078 ceph-mon[99959]: pgmap v7129: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:38 smithi146 ceph-mon[103278]: pgmap v7130: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[99959]: pgmap v7130: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:38 smithi078 ceph-mon[91136]: pgmap v7130: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:39.966 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:17:40.492 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 4h - - 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 4h - - 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 2s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 18s ago 4h 91.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 2s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 18s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 2s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 2s ago 4h 416M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 18s ago 4h 269M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 2s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 2s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:17:40.493 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 18s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 2s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 2s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 2s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 2s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 18s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 18s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 18s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 18s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 18s ago 4h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 2s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 2s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:17:40.494 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:17:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:17:40] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:17:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:40 smithi078 ceph-mon[91136]: pgmap v7131: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:40 smithi078 ceph-mon[91136]: from='client.60207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:40 smithi078 ceph-mon[99959]: pgmap v7131: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:40 smithi078 ceph-mon[99959]: from='client.60207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:17:40.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:17:40.810 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:17:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:40 smithi146 ceph-mon[103278]: pgmap v7131: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:40 smithi146 ceph-mon[103278]: from='client.60207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:41.075 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:17:41.075 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:17:41.075 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:17:41.075 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:17:41.076 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:17:41.386 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:17:41.386 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:41.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:17:41.386 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:41.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:17:41.387 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:41.135Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:41.387 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:41.135Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:41.387 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:41.136Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:41 smithi146 ceph-mon[103278]: from='client.60213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:41 smithi146 ceph-mon[103278]: from='client.80138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3603101068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1753313553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[91136]: from='client.60213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[91136]: from='client.80138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3603101068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1753313553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[99959]: from='client.60213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[99959]: from='client.80138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3603101068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:17:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1753313553' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:42 smithi146 ceph-mon[103278]: from='client.60231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:42 smithi146 ceph-mon[103278]: pgmap v7132: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[91136]: from='client.60231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[91136]: pgmap v7132: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[99959]: from='client.60231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:17:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[99959]: pgmap v7132: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:44 smithi146 ceph-mon[103278]: pgmap v7133: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:44 smithi078 ceph-mon[91136]: pgmap v7133: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:44 smithi078 ceph-mon[99959]: pgmap v7133: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:46 smithi146 ceph-mon[103278]: pgmap v7134: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:46 smithi078 ceph-mon[91136]: pgmap v7134: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:46 smithi078 ceph-mon[99959]: pgmap v7134: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:48 smithi146 ceph-mon[103278]: pgmap v7135: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:48 smithi078 ceph-mon[91136]: pgmap v7135: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:48 smithi078 ceph-mon[99959]: pgmap v7135: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:50.968 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:17:50] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:17:51.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:50 smithi078 ceph-mon[91136]: pgmap v7136: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:51.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:50 smithi078 ceph-mon[99959]: pgmap v7136: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:51.244 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:51.136Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:51.244 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:51.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:17:51.244 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:17:51.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:17:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:50 smithi146 ceph-mon[103278]: pgmap v7136: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:52 smithi146 ceph-mon[103278]: pgmap v7137: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:52 smithi078 ceph-mon[99959]: pgmap v7137: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:17:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:52 smithi078 ceph-mon[91136]: pgmap v7137: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:54 smithi146 ceph-mon[103278]: pgmap v7138: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:54 smithi078 ceph-mon[91136]: pgmap v7138: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:54 smithi078 ceph-mon[99959]: pgmap v7138: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:17:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:56 smithi146 ceph-mon[103278]: pgmap v7139: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:56 smithi078 ceph-mon[91136]: pgmap v7139: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:56 smithi078 ceph-mon[99959]: pgmap v7139: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:58 smithi146 ceph-mon[103278]: pgmap v7140: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[99959]: pgmap v7140: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[91136]: pgmap v7140: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:17:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:17:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:00 smithi146 ceph-mon[103278]: pgmap v7141: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:00 smithi078 ceph-mon[91136]: pgmap v7141: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:00 smithi078 ceph-mon[99959]: pgmap v7141: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:18:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:18:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:01.137Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:01.137Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:01.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:02.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:02.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:02 smithi146 ceph-mon[103278]: pgmap v7142: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:02.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:02.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:02.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[91136]: pgmap v7142: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[99959]: pgmap v7142: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:04 smithi146 ceph-mon[103278]: pgmap v7143: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:04 smithi078 ceph-mon[91136]: pgmap v7143: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:04 smithi078 ceph-mon[99959]: pgmap v7143: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:06 smithi146 ceph-mon[103278]: pgmap v7144: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:06 smithi078 ceph-mon[91136]: pgmap v7144: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:06 smithi078 ceph-mon[99959]: pgmap v7144: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:08 smithi146 ceph-mon[103278]: pgmap v7145: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:08 smithi078 ceph-mon[91136]: pgmap v7145: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:08 smithi078 ceph-mon[99959]: pgmap v7145: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:10 smithi146 ceph-mon[103278]: pgmap v7146: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:10 smithi078 ceph-mon[91136]: pgmap v7146: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:10 smithi078 ceph-mon[99959]: pgmap v7146: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:11.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:18:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:18:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:11.137Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:11.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:11.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:11.691 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 4h - - 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 4h - - 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 14s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 9s ago 4h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 14s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:18:12.220 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 9s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 14s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 14s ago 4h 418M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 9s ago 4h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 14s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 14s ago 4h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 9s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 14s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 14s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 14s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 14s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 9s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 9s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 9s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:18:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 9s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:18:12.222 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 9s ago 4h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:18:12.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 14s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:18:12.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:18:12.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 14s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:18:12.222 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:12.535 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:18:12.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:18:12.803 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:18:12.804 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[99959]: from='client.60243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[99959]: pgmap v7147: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[99959]: from='client.80168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2637123282' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[91136]: from='client.60243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:12.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[91136]: pgmap v7147: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:12.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[91136]: from='client.80168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:12.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2637123282' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:12 smithi146 ceph-mon[103278]: from='client.60243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:12 smithi146 ceph-mon[103278]: pgmap v7147: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:12 smithi146 ceph-mon[103278]: from='client.80168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2637123282' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:13.110 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:18:13.110 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:13.110 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:13.110 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:13.111 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:13 smithi146 ceph-mon[103278]: from='client.80174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:13 smithi146 ceph-mon[103278]: from='client.80186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1384959052' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[91136]: from='client.80174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[91136]: from='client.80186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1384959052' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[99959]: from='client.80174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[99959]: from='client.80186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1384959052' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:14 smithi146 ceph-mon[103278]: pgmap v7148: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:14 smithi078 ceph-mon[91136]: pgmap v7148: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:14 smithi078 ceph-mon[99959]: pgmap v7148: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:16 smithi146 ceph-mon[103278]: pgmap v7149: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:16 smithi078 ceph-mon[91136]: pgmap v7149: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:16 smithi078 ceph-mon[99959]: pgmap v7149: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:18 smithi146 ceph-mon[103278]: pgmap v7150: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[99959]: pgmap v7150: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[91136]: pgmap v7150: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:20 smithi078 ceph-mon[91136]: pgmap v7151: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:20 smithi078 ceph-mon[99959]: pgmap v7151: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:18:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:18:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:20 smithi146 ceph-mon[103278]: pgmap v7151: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:21.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:21.138Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:21.499 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:21.138Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:21.499 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:21.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:22 smithi146 ceph-mon[103278]: pgmap v7152: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[91136]: pgmap v7152: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[99959]: pgmap v7152: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:24 smithi146 ceph-mon[103278]: pgmap v7153: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:24 smithi078 ceph-mon[99959]: pgmap v7153: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:24 smithi078 ceph-mon[91136]: pgmap v7153: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:26 smithi146 ceph-mon[103278]: pgmap v7154: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:26 smithi078 ceph-mon[99959]: pgmap v7154: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:26 smithi078 ceph-mon[91136]: pgmap v7154: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:28 smithi146 ceph-mon[103278]: pgmap v7155: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:28 smithi078 ceph-mon[99959]: pgmap v7155: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:28 smithi078 ceph-mon[91136]: pgmap v7155: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:30 smithi078 ceph-mon[99959]: pgmap v7156: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:30 smithi078 ceph-mon[91136]: pgmap v7156: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:18:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:18:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:30 smithi146 ceph-mon[103278]: pgmap v7156: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:31.139Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:31.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:31.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:32 smithi146 ceph-mon[103278]: pgmap v7157: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:32 smithi078 ceph-mon[91136]: pgmap v7157: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:32 smithi078 ceph-mon[99959]: pgmap v7157: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:34 smithi146 ceph-mon[103278]: pgmap v7158: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:34 smithi078 ceph-mon[91136]: pgmap v7158: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:34 smithi078 ceph-mon[99959]: pgmap v7158: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:37.056 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:36 smithi078 ceph-mon[91136]: pgmap v7159: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:37.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:36 smithi078 ceph-mon[99959]: pgmap v7159: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:36 smithi146 ceph-mon[103278]: pgmap v7159: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[91136]: pgmap v7160: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[99959]: pgmap v7160: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:38 smithi146 ceph-mon[103278]: pgmap v7160: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:40.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:18:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:18:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:40 smithi146 ceph-mon[103278]: pgmap v7161: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:40 smithi078 ceph-mon[99959]: pgmap v7161: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:41.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:40 smithi078 ceph-mon[91136]: pgmap v7161: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:41.139Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:41.139Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:41.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:43.220 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:43.220 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:42 smithi146 ceph-mon[103278]: pgmap v7162: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:43.220 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.220 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.220 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[99959]: pgmap v7162: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[91136]: pgmap v7162: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:43.409 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:18:43.934 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:18:43.934 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 4h - - 2024-09-21T00:18:43.934 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 4h - - 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 1s ago 4h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 5s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 1s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 6s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 6s ago 4h 416M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 1s ago 4h 270M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 6s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 1s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 6s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 6s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 6s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 6s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 1s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 1s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:18:43.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 1s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:18:43.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 1s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:18:43.936 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 1s ago 4h 69.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:18:43.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 6s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:18:43.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:18:43.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 5s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:18:43.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:18:44.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:18:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:18:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:18:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:18:44.250 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:18:44.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:44.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:44.516 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:18:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:18:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:18:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:18:44.517 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:18:44.824 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:18:44.825 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:18:44.825 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:44 smithi146 ceph-mon[103278]: from='client.80198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:44 smithi146 ceph-mon[103278]: from='client.80204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:44 smithi146 ceph-mon[103278]: pgmap v7163: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:44 smithi146 ceph-mon[103278]: from='client.60291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/169393901' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1163654126' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[91136]: from='client.80198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[91136]: from='client.80204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[91136]: pgmap v7163: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[91136]: from='client.60291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/169393901' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1163654126' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[99959]: from='client.80198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[99959]: from='client.80204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[99959]: pgmap v7163: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[99959]: from='client.60291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/169393901' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:18:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1163654126' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:18:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:45 smithi146 ceph-mon[103278]: from='client.80222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:45 smithi078 ceph-mon[91136]: from='client.80222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:45 smithi078 ceph-mon[99959]: from='client.80222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:18:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:46 smithi146 ceph-mon[103278]: pgmap v7164: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:46 smithi078 ceph-mon[91136]: pgmap v7164: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:46 smithi078 ceph-mon[99959]: pgmap v7164: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:48 smithi078 ceph-mon[91136]: pgmap v7165: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:48 smithi078 ceph-mon[99959]: pgmap v7165: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:48 smithi146 ceph-mon[103278]: pgmap v7165: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:50.961 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:18:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:18:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:50 smithi146 ceph-mon[103278]: pgmap v7166: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:50 smithi078 ceph-mon[91136]: pgmap v7166: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:50 smithi078 ceph-mon[99959]: pgmap v7166: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:51.139Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:51.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:18:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:18:51.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:18:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:52 smithi146 ceph-mon[103278]: pgmap v7167: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:52 smithi078 ceph-mon[99959]: pgmap v7167: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:18:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:52 smithi078 ceph-mon[91136]: pgmap v7167: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:54 smithi146 ceph-mon[103278]: pgmap v7168: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:54 smithi078 ceph-mon[91136]: pgmap v7168: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:54 smithi078 ceph-mon[99959]: pgmap v7168: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:18:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:56 smithi146 ceph-mon[103278]: pgmap v7169: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:56 smithi078 ceph-mon[91136]: pgmap v7169: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:56 smithi078 ceph-mon[99959]: pgmap v7169: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:58 smithi146 ceph-mon[103278]: pgmap v7170: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:18:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[99959]: pgmap v7170: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:18:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[91136]: pgmap v7170: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:18:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:18:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:18:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:00.977 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:19:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:19:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:00 smithi146 ceph-mon[103278]: pgmap v7171: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:00 smithi078 ceph-mon[91136]: pgmap v7171: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:00 smithi078 ceph-mon[99959]: pgmap v7171: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:01.140Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:01.140Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:01.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:02 smithi146 ceph-mon[103278]: pgmap v7172: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[99959]: pgmap v7172: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[91136]: pgmap v7172: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:04.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:04 smithi146 ceph-mon[103278]: pgmap v7173: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:04 smithi078 ceph-mon[91136]: pgmap v7173: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:04 smithi078 ceph-mon[99959]: pgmap v7173: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:06 smithi146 ceph-mon[103278]: pgmap v7174: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:06 smithi078 ceph-mon[99959]: pgmap v7174: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:06 smithi078 ceph-mon[91136]: pgmap v7174: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:08 smithi146 ceph-mon[103278]: pgmap v7175: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:08 smithi078 ceph-mon[91136]: pgmap v7175: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:08 smithi078 ceph-mon[99959]: pgmap v7175: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:10 smithi146 ceph-mon[103278]: pgmap v7176: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:10 smithi078 ceph-mon[91136]: pgmap v7176: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:10 smithi078 ceph-mon[99959]: pgmap v7176: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:19:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:19:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:11.140Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:11.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:11.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:12 smithi146 ceph-mon[103278]: pgmap v7177: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:12 smithi078 ceph-mon[91136]: pgmap v7177: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:12 smithi078 ceph-mon[99959]: pgmap v7177: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:14 smithi078 ceph-mon[91136]: pgmap v7178: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:14 smithi078 ceph-mon[99959]: pgmap v7178: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:15.122 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:19:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:14 smithi146 ceph-mon[103278]: pgmap v7178: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:15.647 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 4h - - 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 4h - - 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 17s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 12s ago 4h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 17s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 12s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 17s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 17s ago 4h 417M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 12s ago 4h 271M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 17s ago 4h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 17s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 12s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 17s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:19:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 17s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 17s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 17s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 12s ago 4h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 12s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 12s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 12s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 12s ago 4h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 17s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 17s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:19:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:19:15.959 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:19:15.959 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:19:15.959 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:19:15.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:15.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:19:15.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:19:15.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:15.960 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:19:15.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:19:15.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:15.961 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:19:15.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:19:15.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:15.961 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:19:15.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:19:15.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:19:15.962 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:19:16.232 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:19:16.233 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:19:16.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:19:16.549 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:19:16.549 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:16.549 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:16.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[91136]: from='client.80234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[91136]: from='client.60321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[91136]: from='client.80246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[91136]: pgmap v7179: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/268859203' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3578192875' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[99959]: from='client.80234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[99959]: from='client.60321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[99959]: from='client.80246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[99959]: pgmap v7179: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/268859203' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:16.956 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3578192875' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:16 smithi146 ceph-mon[103278]: from='client.80234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:16 smithi146 ceph-mon[103278]: from='client.60321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:16 smithi146 ceph-mon[103278]: from='client.80246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:16 smithi146 ceph-mon[103278]: pgmap v7179: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/268859203' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3578192875' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:18.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:17 smithi078 ceph-mon[91136]: from='client.60339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:18.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:17 smithi078 ceph-mon[99959]: from='client.60339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:17 smithi146 ceph-mon[103278]: from='client.60339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[91136]: pgmap v7180: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[99959]: pgmap v7180: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:18 smithi146 ceph-mon[103278]: pgmap v7180: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:20 smithi078 ceph-mon[91136]: pgmap v7181: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:20 smithi078 ceph-mon[99959]: pgmap v7181: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:19:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:19:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:20 smithi146 ceph-mon[103278]: pgmap v7181: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:21.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:21.141Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:21.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:21.141Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:21.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:21.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:22 smithi078 ceph-mon[99959]: pgmap v7182: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:22 smithi078 ceph-mon[91136]: pgmap v7182: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:22 smithi146 ceph-mon[103278]: pgmap v7182: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:25.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:24 smithi146 ceph-mon[103278]: pgmap v7183: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:24 smithi078 ceph-mon[91136]: pgmap v7183: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:24 smithi078 ceph-mon[99959]: pgmap v7183: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:26 smithi146 ceph-mon[103278]: pgmap v7184: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:26 smithi078 ceph-mon[91136]: pgmap v7184: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:26 smithi078 ceph-mon[99959]: pgmap v7184: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:28 smithi146 ceph-mon[103278]: pgmap v7185: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:28 smithi078 ceph-mon[91136]: pgmap v7185: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:28 smithi078 ceph-mon[99959]: pgmap v7185: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:30.880 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:19:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:19:31.142 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:30 smithi078 ceph-mon[99959]: pgmap v7186: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:31.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:30 smithi078 ceph-mon[91136]: pgmap v7186: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:30 smithi146 ceph-mon[103278]: pgmap v7186: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:31.141Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:31.142Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:31.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:32 smithi146 ceph-mon[103278]: pgmap v7187: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:32 smithi078 ceph-mon[91136]: pgmap v7187: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:32 smithi078 ceph-mon[99959]: pgmap v7187: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:34 smithi146 ceph-mon[103278]: pgmap v7188: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:34 smithi078 ceph-mon[91136]: pgmap v7188: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:34 smithi078 ceph-mon[99959]: pgmap v7188: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:36 smithi146 ceph-mon[103278]: pgmap v7189: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:36 smithi078 ceph-mon[91136]: pgmap v7189: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:36 smithi078 ceph-mon[99959]: pgmap v7189: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:39.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[91136]: pgmap v7190: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:39.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[99959]: pgmap v7190: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:39.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.129 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:38 smithi146 ceph-mon[103278]: pgmap v7190: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:40.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:19:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:19:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:40 smithi146 ceph-mon[103278]: pgmap v7191: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:41.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:40 smithi078 ceph-mon[91136]: pgmap v7191: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:40 smithi078 ceph-mon[99959]: pgmap v7191: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:41.142Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:41.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:41.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:42 smithi146 ceph-mon[103278]: pgmap v7192: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:42 smithi078 ceph-mon[91136]: pgmap v7192: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:42 smithi078 ceph-mon[99959]: pgmap v7192: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:44.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.244 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:45.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:44 smithi146 ceph-mon[103278]: pgmap v7193: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:44 smithi078 ceph-mon[91136]: pgmap v7193: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:44 smithi078 ceph-mon[99959]: pgmap v7193: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:46.845 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:19:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:46 smithi146 ceph-mon[103278]: pgmap v7194: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:47.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:46 smithi078 ceph-mon[91136]: pgmap v7194: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:46 smithi078 ceph-mon[99959]: pgmap v7194: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:47.374 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:19:47.374 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 4h - - 2024-09-21T00:19:47.374 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 4h - - 2024-09-21T00:19:47.374 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 9s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 4s ago 4h 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 9s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 4s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 9s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 9s ago 4h 418M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 4s ago 4h 272M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 9s ago 4h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 9s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 4s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 9s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 9s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 9s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 9s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 4s ago 4h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 4s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:19:47.375 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 4s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:19:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 4s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:19:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 4s ago 4h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:19:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 9s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:19:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:19:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 9s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:19:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:19:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:19:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:19:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:19:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:19:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:19:47.683 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:19:47.950 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:19:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:19:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:19:47.951 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:19:48.259 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:19:48.259 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:48.260 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:19:48.260 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:48.260 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:47 smithi078 ceph-mon[91136]: from='client.60351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:47 smithi078 ceph-mon[91136]: from='client.60357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:47 smithi078 ceph-mon[91136]: from='client.60363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3586203204' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:47 smithi078 ceph-mon[91136]: pgmap v7195: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:47 smithi078 ceph-mon[91136]: from='client.80294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:48 smithi078 ceph-mon[99959]: from='client.60351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:48 smithi078 ceph-mon[99959]: from='client.60357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:48 smithi078 ceph-mon[99959]: from='client.60363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3586203204' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:48.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:48 smithi078 ceph-mon[99959]: pgmap v7195: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:48.261 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:48 smithi078 ceph-mon[99959]: from='client.80294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:47 smithi146 ceph-mon[103278]: from='client.60351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:47 smithi146 ceph-mon[103278]: from='client.60357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:47 smithi146 ceph-mon[103278]: from='client.60363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3586203204' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:19:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:47 smithi146 ceph-mon[103278]: pgmap v7195: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:47 smithi146 ceph-mon[103278]: from='client.80294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:19:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1347103183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1347103183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1347103183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:19:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:50 smithi146 ceph-mon[103278]: pgmap v7196: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:50 smithi078 ceph-mon[91136]: pgmap v7196: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:50 smithi078 ceph-mon[99959]: pgmap v7196: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:19:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:19:51.342 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:51.143Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:51.342 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:51.143Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:19:51.342 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:19:51.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:19:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:52 smithi078 ceph-mon[91136]: pgmap v7197: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:52 smithi078 ceph-mon[99959]: pgmap v7197: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:19:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:52 smithi146 ceph-mon[103278]: pgmap v7197: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:54 smithi078 ceph-mon[91136]: pgmap v7198: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:54 smithi078 ceph-mon[99959]: pgmap v7198: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:54 smithi146 ceph-mon[103278]: pgmap v7198: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:19:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:56 smithi078 ceph-mon[91136]: pgmap v7199: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:56 smithi078 ceph-mon[99959]: pgmap v7199: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:56 smithi146 ceph-mon[103278]: pgmap v7199: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[91136]: pgmap v7200: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[99959]: pgmap v7200: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:58 smithi146 ceph-mon[103278]: pgmap v7200: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:19:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:00.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T00:19:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:00.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:00 smithi078 ceph-mon[91136]: pgmap v7201: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:00.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:00.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:20:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:20:00.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:00 smithi078 ceph-mon[99959]: pgmap v7201: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:00.887 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:00 smithi146 ceph-mon[103278]: pgmap v7201: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:01.144Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:01.144Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:01.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:02 smithi078 ceph-mon[91136]: pgmap v7202: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:02 smithi078 ceph-mon[99959]: pgmap v7202: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:03.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:03.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:02 smithi146 ceph-mon[103278]: pgmap v7202: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:04 smithi146 ceph-mon[103278]: pgmap v7203: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[91136]: pgmap v7203: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:04 smithi078 ceph-mon[99959]: pgmap v7203: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:06.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:06 smithi146 ceph-mon[103278]: pgmap v7204: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:06 smithi078 ceph-mon[91136]: pgmap v7204: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:06 smithi078 ceph-mon[99959]: pgmap v7204: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:08 smithi078 ceph-mon[91136]: pgmap v7205: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:08 smithi078 ceph-mon[99959]: pgmap v7205: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:08 smithi146 ceph-mon[103278]: pgmap v7205: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:10.991 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:20:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:20:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:10 smithi146 ceph-mon[103278]: pgmap v7206: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:10 smithi078 ceph-mon[91136]: pgmap v7206: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:10 smithi078 ceph-mon[99959]: pgmap v7206: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:11.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:11.145Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:11.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:11.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:12 smithi146 ceph-mon[103278]: pgmap v7207: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:12 smithi078 ceph-mon[91136]: pgmap v7207: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:12 smithi078 ceph-mon[99959]: pgmap v7207: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:14.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:14.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:13 smithi146 ceph-mon[103278]: pgmap v7208: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:13 smithi078 ceph-mon[91136]: pgmap v7208: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:13 smithi078 ceph-mon[99959]: pgmap v7208: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:16 smithi078 ceph-mon[99959]: pgmap v7209: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:16 smithi078 ceph-mon[91136]: pgmap v7209: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:16 smithi146 ceph-mon[103278]: pgmap v7209: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:18.560 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[99959]: pgmap v7210: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[91136]: pgmap v7210: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:20:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 4h - - 2024-09-21T00:20:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 4h - - 2024-09-21T00:20:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 0s ago 4h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:20:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 15s ago 4h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:20:19.100 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 0s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 15s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 0s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 0s ago 4h 419M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 15s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 0s ago 4h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 0s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 15s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 0s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 0s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:20:19.101 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 0s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 0s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 15s ago 4h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 15s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 15s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 15s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 15s ago 4h 68.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 0s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:20:19.102 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:20:19.103 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 0s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:20:19.103 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:20:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:18 smithi146 ceph-mon[103278]: pgmap v7210: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:20:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:20:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:20:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:20:19.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:20:19.420 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:20:19.695 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:20:19.695 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:20:19.695 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:20:19.695 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:20:19.696 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:20:20.010 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:20.010 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:20.010 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:20.010 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:20.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:19 smithi078 ceph-mon[91136]: from='client.60387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:19 smithi078 ceph-mon[91136]: from='client.60393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2323338450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:20.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:19 smithi078 ceph-mon[99959]: from='client.60387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:19 smithi078 ceph-mon[99959]: from='client.60393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2323338450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:19 smithi146 ceph-mon[103278]: from='client.60387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:19 smithi146 ceph-mon[103278]: from='client.60393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2323338450' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[99959]: from='client.60399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[99959]: from='client.60411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[99959]: pgmap v7211: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/819800478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[91136]: from='client.60399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[91136]: from='client.60411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[91136]: pgmap v7211: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:20.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/819800478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:20.909 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:20:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:20:21.192 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:21.145Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:21.193 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:21.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:21.193 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:21.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:20 smithi146 ceph-mon[103278]: from='client.60399 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:20 smithi146 ceph-mon[103278]: from='client.60411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:20 smithi146 ceph-mon[103278]: pgmap v7211: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/819800478' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:22 smithi078 ceph-mon[99959]: pgmap v7212: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:22 smithi078 ceph-mon[91136]: pgmap v7212: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:23.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:23.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:22 smithi146 ceph-mon[103278]: pgmap v7212: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:24 smithi146 ceph-mon[103278]: pgmap v7213: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[91136]: pgmap v7213: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:24 smithi078 ceph-mon[99959]: pgmap v7213: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:26 smithi146 ceph-mon[103278]: pgmap v7214: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:26 smithi078 ceph-mon[91136]: pgmap v7214: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:26 smithi078 ceph-mon[99959]: pgmap v7214: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:28 smithi146 ceph-mon[103278]: pgmap v7215: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:28.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:28 smithi078 ceph-mon[91136]: pgmap v7215: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:28 smithi078 ceph-mon[99959]: pgmap v7215: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:30.995 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:20:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:20:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:30 smithi146 ceph-mon[103278]: pgmap v7216: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:30 smithi078 ceph-mon[91136]: pgmap v7216: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:30 smithi078 ceph-mon[99959]: pgmap v7216: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:31.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:31.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:31.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:32 smithi146 ceph-mon[103278]: pgmap v7217: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:32 smithi078 ceph-mon[99959]: pgmap v7217: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:32 smithi078 ceph-mon[91136]: pgmap v7217: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:34 smithi078 ceph-mon[91136]: pgmap v7218: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:34 smithi078 ceph-mon[99959]: pgmap v7218: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:34 smithi146 ceph-mon[103278]: pgmap v7218: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:36 smithi078 ceph-mon[91136]: pgmap v7219: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:36 smithi078 ceph-mon[99959]: pgmap v7219: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:36 smithi146 ceph-mon[103278]: pgmap v7219: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[91136]: pgmap v7220: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[99959]: pgmap v7220: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:38 smithi146 ceph-mon[103278]: pgmap v7220: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:40.900 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:40 smithi078 ceph-mon[99959]: pgmap v7221: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:40.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:40 smithi078 ceph-mon[91136]: pgmap v7221: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:40.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:20:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:20:41.250 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:41.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:41.251 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:41.146Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:41.251 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:41.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:40 smithi146 ceph-mon[103278]: pgmap v7221: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:42 smithi078 ceph-mon[91136]: pgmap v7222: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:42 smithi078 ceph-mon[99959]: pgmap v7222: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:43.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:42 smithi146 ceph-mon[103278]: pgmap v7222: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:44 smithi146 ceph-mon[103278]: pgmap v7223: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[91136]: pgmap v7223: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:20:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:44 smithi078 ceph-mon[99959]: pgmap v7223: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:47.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:46 smithi146 ceph-mon[103278]: pgmap v7224: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:46 smithi078 ceph-mon[91136]: pgmap v7224: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:46 smithi078 ceph-mon[99959]: pgmap v7224: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:49.003 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:48 smithi146 ceph-mon[103278]: pgmap v7225: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:48 smithi078 ceph-mon[91136]: pgmap v7225: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:48 smithi078 ceph-mon[99959]: pgmap v7225: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:50.295 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:20:50.821 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:20:50.821 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 4h - - 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 4h - - 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 12s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 7s ago 4h 88.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 12s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 7s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (3h) 12s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 12s ago 4h 420M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 7s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 12s ago 4h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 12s ago 4h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 7s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 12s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:20:50.822 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 12s ago 4h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 12s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 12s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 7s ago 4h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 7s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 7s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 7s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 7s ago 4h 68.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 12s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 12s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:20:50.823 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:20:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:50 smithi078 ceph-mon[91136]: pgmap v7226: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:20:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:20:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:50 smithi078 ceph-mon[99959]: pgmap v7226: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:51.131 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:20:51.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:20:51.132 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:20:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:50 smithi146 ceph-mon[103278]: pgmap v7226: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:20:51.389 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:51.146Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:51.389 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:51.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:20:51.389 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:20:51.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:20:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:20:51.402 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:20:51.714 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:20:51.714 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:20:51.714 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:20:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[91136]: from='client.80342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[91136]: from='client.60429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[91136]: from='client.80354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2978039317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3971409962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[99959]: from='client.80342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[99959]: from='client.60429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[99959]: from='client.80354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2978039317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3971409962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:51 smithi146 ceph-mon[103278]: from='client.80342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:51 smithi146 ceph-mon[103278]: from='client.60429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:51 smithi146 ceph-mon[103278]: from='client.80354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2978039317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:20:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3971409962' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:52 smithi146 ceph-mon[103278]: from='client.80366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:52 smithi146 ceph-mon[103278]: pgmap v7227: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:52 smithi078 ceph-mon[91136]: from='client.80366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:52 smithi078 ceph-mon[91136]: pgmap v7227: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:20:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:52 smithi078 ceph-mon[99959]: from='client.80366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:20:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:52 smithi078 ceph-mon[99959]: pgmap v7227: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:20:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:54 smithi146 ceph-mon[103278]: pgmap v7228: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:20:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:54 smithi078 ceph-mon[91136]: pgmap v7228: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:20:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:54 smithi078 ceph-mon[99959]: pgmap v7228: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:20:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:56 smithi146 ceph-mon[103278]: pgmap v7229: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:20:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:56 smithi078 ceph-mon[91136]: pgmap v7229: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:20:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:56 smithi078 ceph-mon[99959]: pgmap v7229: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:20:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:58 smithi146 ceph-mon[103278]: pgmap v7230: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:20:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:58 smithi078 ceph-mon[91136]: pgmap v7230: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:20:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:20:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:58 smithi078 ceph-mon[99959]: pgmap v7230: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:20:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:21:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:21:01.084 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:00 smithi078 ceph-mon[99959]: pgmap v7231: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:21:01.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:00 smithi078 ceph-mon[91136]: pgmap v7231: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:21:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:00 smithi146 ceph-mon[103278]: pgmap v7231: 161 pgs: 161 active+clean; 585 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:21:01.394 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:01.147Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:01.394 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:01.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:01.394 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:01.147Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:02 smithi078 ceph-mon[91136]: pgmap v7232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:21:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:02 smithi078 ceph-mon[99959]: pgmap v7232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:21:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:02 smithi146 ceph-mon[103278]: pgmap v7232: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:21:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:04 smithi146 ceph-mon[103278]: pgmap v7233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[99959]: pgmap v7233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:04 smithi078 ceph-mon[91136]: pgmap v7233: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:21:07.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:06 smithi146 ceph-mon[103278]: pgmap v7234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:21:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:06 smithi078 ceph-mon[91136]: pgmap v7234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:21:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:06 smithi078 ceph-mon[99959]: pgmap v7234: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T00:21:09.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:08 smithi146 ceph-mon[103278]: pgmap v7235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:21:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:08 smithi078 ceph-mon[99959]: pgmap v7235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:21:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:08 smithi078 ceph-mon[91136]: pgmap v7235: 161 pgs: 161 active+clean; 585 KiB data, 287 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:21:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:10 smithi078 ceph-mon[91136]: pgmap v7236: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:21:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:10 smithi078 ceph-mon[99959]: pgmap v7236: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:21:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:21:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:21:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:10 smithi146 ceph-mon[103278]: pgmap v7236: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T00:21:11.399 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:11.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:11.399 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:11.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:11.399 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:11.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:13.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:13.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:12 smithi146 ceph-mon[103278]: pgmap v7237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:21:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:12 smithi078 ceph-mon[91136]: pgmap v7237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:21:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:12 smithi078 ceph-mon[99959]: pgmap v7237: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 157 op/s 2024-09-21T00:21:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:14 smithi146 ceph-mon[103278]: pgmap v7238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:21:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:14 smithi078 ceph-mon[91136]: pgmap v7238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:21:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:14 smithi078 ceph-mon[99959]: pgmap v7238: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:21:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:16 smithi146 ceph-mon[103278]: pgmap v7239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:21:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:16 smithi078 ceph-mon[91136]: pgmap v7239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:21:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:16 smithi078 ceph-mon[99959]: pgmap v7239: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:21:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:18 smithi146 ceph-mon[103278]: pgmap v7240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-21T00:21:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:18 smithi078 ceph-mon[91136]: pgmap v7240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-21T00:21:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:18 smithi078 ceph-mon[99959]: pgmap v7240: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-21T00:21:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:20.958 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:21:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:21:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:20 smithi146 ceph-mon[103278]: pgmap v7241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T00:21:21.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:20 smithi078 ceph-mon[91136]: pgmap v7241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T00:21:21.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:20 smithi078 ceph-mon[99959]: pgmap v7241: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T00:21:21.270 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:21.148Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:21.270 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:21.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:21.270 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:21.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:22.014 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:21:22.575 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:21:22.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 4h - - 2024-09-21T00:21:22.575 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 4h - - 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 3s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (3h) 3s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 3s ago 4h 421M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 19s ago 4h 273M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 3s ago 4h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 3s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 3s ago 4h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 3s ago 4h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 3s ago 4h 75.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 3s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:21:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 19s ago 4h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 19s ago 4h 77.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 19s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 19s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (3h) 19s ago 4h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 3s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 3s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:21:22.577 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:21:22.893 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:21:22.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:21:22.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:22.895 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:21:22.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:21:22.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:21:22.895 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:21:23.177 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:21:23.178 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:21:23.178 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:21:23.178 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:21:23.178 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:21:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[91136]: pgmap v7242: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:21:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[91136]: from='client.80378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4134463614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:23.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:23.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[99959]: pgmap v7242: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:21:23.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[99959]: from='client.80378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:23.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4134463614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:22 smithi146 ceph-mon[103278]: pgmap v7242: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T00:21:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:22 smithi146 ceph-mon[103278]: from='client.80378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4134463614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:23.492 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:21:23.493 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:23.493 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:23.493 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:23.493 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:23 smithi146 ceph-mon[103278]: from='client.60465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:23 smithi146 ceph-mon[103278]: from='client.80390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2908411023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[91136]: from='client.60465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[91136]: from='client.80390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2908411023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[99959]: from='client.60465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[99959]: from='client.80390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2908411023' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:25 smithi146 ceph-mon[103278]: from='client.60483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:25 smithi146 ceph-mon[103278]: pgmap v7243: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:21:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:25 smithi078 ceph-mon[99959]: from='client.60483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:25 smithi078 ceph-mon[99959]: pgmap v7243: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:21:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:25 smithi078 ceph-mon[91136]: from='client.60483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:25 smithi078 ceph-mon[91136]: pgmap v7243: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T00:21:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:26 smithi146 ceph-mon[103278]: pgmap v7244: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:21:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:26 smithi078 ceph-mon[91136]: pgmap v7244: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:21:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:26 smithi078 ceph-mon[99959]: pgmap v7244: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:21:28.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:28 smithi146 ceph-mon[103278]: pgmap v7245: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:21:28.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:28 smithi078 ceph-mon[91136]: pgmap v7245: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:21:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:28 smithi078 ceph-mon[99959]: pgmap v7245: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T00:21:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:30 smithi078 ceph-mon[91136]: pgmap v7246: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:21:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:30 smithi078 ceph-mon[99959]: pgmap v7246: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:21:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:21:30] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:21:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:30 smithi146 ceph-mon[103278]: pgmap v7246: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:21:31.418 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:31.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:31.418 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:31.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:31.418 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:31.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:32 smithi078 ceph-mon[91136]: pgmap v7247: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:21:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:32 smithi078 ceph-mon[99959]: pgmap v7247: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:21:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:32 smithi146 ceph-mon[103278]: pgmap v7247: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:21:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:34 smithi146 ceph-mon[103278]: pgmap v7248: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:34 smithi078 ceph-mon[91136]: pgmap v7248: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:34 smithi078 ceph-mon[99959]: pgmap v7248: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:36 smithi146 ceph-mon[103278]: pgmap v7249: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:36 smithi078 ceph-mon[99959]: pgmap v7249: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:36 smithi078 ceph-mon[91136]: pgmap v7249: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:39.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:38 smithi078 ceph-mon[91136]: pgmap v7250: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:39.089 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:38 smithi078 ceph-mon[99959]: pgmap v7250: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:38 smithi146 ceph-mon[103278]: pgmap v7250: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:40 smithi146 ceph-mon[103278]: pgmap v7251: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[91136]: pgmap v7251: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-mon[99959]: pgmap v7251: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:21:40] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:21:41.423 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:41.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:41.423 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:41.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:41.423 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:41.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:43.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:43.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:42 smithi146 ceph-mon[103278]: pgmap v7252: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:43.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:43.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:42 smithi078 ceph-mon[91136]: pgmap v7252: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:43.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:43.255 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:42 smithi078 ceph-mon[99959]: pgmap v7252: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:21:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:44 smithi078 ceph-mon[91136]: pgmap v7253: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:44 smithi078 ceph-mon[99959]: pgmap v7253: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:44 smithi146 ceph-mon[103278]: pgmap v7253: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:46 smithi146 ceph-mon[103278]: pgmap v7254: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:46 smithi078 ceph-mon[91136]: pgmap v7254: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:46 smithi078 ceph-mon[99959]: pgmap v7254: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:49.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:48 smithi146 ceph-mon[103278]: pgmap v7255: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:48 smithi078 ceph-mon[91136]: pgmap v7255: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:48 smithi078 ceph-mon[99959]: pgmap v7255: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:50 smithi078 ceph-mon[91136]: pgmap v7256: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:50 smithi078 ceph-mon[99959]: pgmap v7256: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:21:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:21:51.220 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:50 smithi146 ceph-mon[103278]: pgmap v7256: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:51.434 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:51.150Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:51.434 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:51.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:21:51.434 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:21:51.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:21:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:52 smithi146 ceph-mon[103278]: pgmap v7257: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:52 smithi078 ceph-mon[99959]: pgmap v7257: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:21:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:52 smithi078 ceph-mon[91136]: pgmap v7257: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:53.784 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:21:54.309 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:21:54.309 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 4h - - 2024-09-21T00:21:54.309 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 4h - - 2024-09-21T00:21:54.309 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 15s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 10s ago 4h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 15s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 10s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 15s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 15s ago 4h 421M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 10s ago 4h 275M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 15s ago 4h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 15s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 10s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 15s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 15s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 15s ago 4h 75.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:21:54.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 15s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 10s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 10s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 10s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 10s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 4h 72.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 15s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 15s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:21:54.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:21:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:21:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:21:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:21:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:21:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:21:54.621 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:21:54.893 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:21:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:21:54.895 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:21:54.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:54 smithi078 ceph-mon[99959]: pgmap v7258: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:54.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:54 smithi078 ceph-mon[99959]: from='client.60495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:54.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1347545360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:54.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:54 smithi078 ceph-mon[91136]: pgmap v7258: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:54.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:54 smithi078 ceph-mon[91136]: from='client.60495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:54.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1347545360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:55.201 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:21:55.201 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:55.201 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:21:55.201 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:55.201 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:21:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:54 smithi146 ceph-mon[103278]: pgmap v7258: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:21:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:54 smithi146 ceph-mon[103278]: from='client.60495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1347545360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:21:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:55 smithi146 ceph-mon[103278]: from='client.60501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:55 smithi146 ceph-mon[103278]: from='client.80426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:55 smithi146 ceph-mon[103278]: from='client.80438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/512628149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[91136]: from='client.60501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[91136]: from='client.80426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[91136]: from='client.80438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/512628149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[99959]: from='client.60501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[99959]: from='client.80426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[99959]: from='client.80438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:21:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/512628149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:21:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:56 smithi146 ceph-mon[103278]: pgmap v7259: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:56 smithi078 ceph-mon[91136]: pgmap v7259: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:56 smithi078 ceph-mon[99959]: pgmap v7259: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:59.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:58 smithi078 ceph-mon[99959]: pgmap v7260: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:59.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:21:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:59.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:58 smithi078 ceph-mon[91136]: pgmap v7260: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:59.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:21:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:21:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:58 smithi146 ceph-mon[103278]: pgmap v7260: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:21:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:21:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:00 smithi146 ceph-mon[103278]: pgmap v7261: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[91136]: pgmap v7261: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-mon[99959]: pgmap v7261: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:22:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:22:01.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:01.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:01.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:01.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:01.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:01.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:03.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:03.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:02 smithi078 ceph-mon[91136]: pgmap v7262: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:03.083 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:03.083 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:02 smithi078 ceph-mon[99959]: pgmap v7262: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:02 smithi146 ceph-mon[103278]: pgmap v7262: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:05.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[99959]: pgmap v7263: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:05.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[91136]: pgmap v7263: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:04 smithi146 ceph-mon[103278]: pgmap v7263: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:06 smithi146 ceph-mon[103278]: pgmap v7264: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:06 smithi078 ceph-mon[91136]: pgmap v7264: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:06 smithi078 ceph-mon[99959]: pgmap v7264: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:09.144 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:08 smithi146 ceph-mon[103278]: pgmap v7265: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:08 smithi078 ceph-mon[91136]: pgmap v7265: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:08 smithi078 ceph-mon[99959]: pgmap v7265: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:22:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:22:11.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:10 smithi146 ceph-mon[103278]: pgmap v7266: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:11.152 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:10 smithi078 ceph-mon[99959]: pgmap v7266: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:11.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:10 smithi078 ceph-mon[91136]: pgmap v7266: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:11.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:11.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:11.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:11.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:11.450 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:11.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:12 smithi146 ceph-mon[103278]: pgmap v7267: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:12 smithi078 ceph-mon[91136]: pgmap v7267: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:12 smithi078 ceph-mon[99959]: pgmap v7267: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:14 smithi146 ceph-mon[103278]: pgmap v7268: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:14 smithi078 ceph-mon[91136]: pgmap v7268: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:14 smithi078 ceph-mon[99959]: pgmap v7268: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:16 smithi146 ceph-mon[103278]: pgmap v7269: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:16 smithi078 ceph-mon[91136]: pgmap v7269: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:16 smithi078 ceph-mon[99959]: pgmap v7269: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:18 smithi146 ceph-mon[103278]: pgmap v7270: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:18 smithi078 ceph-mon[99959]: pgmap v7270: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:18 smithi078 ceph-mon[91136]: pgmap v7270: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[91136]: pgmap v7271: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-mon[99959]: pgmap v7271: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:20 smithi146 ceph-mon[103278]: pgmap v7271: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:22:20] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:22:21.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:21.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:21.461 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:21.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:21.461 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:21.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:22 smithi146 ceph-mon[103278]: pgmap v7272: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:23.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:23.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:22 smithi078 ceph-mon[91136]: pgmap v7272: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:23.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:23.605 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:22 smithi078 ceph-mon[99959]: pgmap v7272: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[99959]: pgmap v7273: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[91136]: pgmap v7273: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:24 smithi146 ceph-mon[103278]: pgmap v7273: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:25.501 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 4h - - 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 4h - - 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 2s ago 4h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 2s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 6s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (3h) 6s ago 4h 423M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:22:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 2s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 6s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 2s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 6s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 6s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 6s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 6s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 2s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 2s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 2s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 2s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 2s ago 4h 73.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 6s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:22:26.055 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:22:26.056 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 6s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:22:26.056 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:22:26.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:22:26.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:26.366 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:22:26.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:22:26.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:22:26.366 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:22:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:22:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:22:26.635 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:22:26.946 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:22:26.946 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:26.946 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[91136]: from='client.80450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[91136]: pgmap v7274: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[91136]: from='client.80456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1354018279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[99959]: from='client.80450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[99959]: pgmap v7274: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[99959]: from='client.80456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:26.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1354018279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:26 smithi146 ceph-mon[103278]: from='client.80450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:26 smithi146 ceph-mon[103278]: pgmap v7274: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:26 smithi146 ceph-mon[103278]: from='client.80456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1354018279' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:27 smithi146 ceph-mon[103278]: from='client.60543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:27 smithi146 ceph-mon[103278]: from='client.80474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3180825737' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:27 smithi078 ceph-mon[91136]: from='client.60543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:27 smithi078 ceph-mon[91136]: from='client.80474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3180825737' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:27 smithi078 ceph-mon[99959]: from='client.60543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:27 smithi078 ceph-mon[99959]: from='client.80474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3180825737' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:28 smithi146 ceph-mon[103278]: pgmap v7275: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:28 smithi078 ceph-mon[99959]: pgmap v7275: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:28 smithi078 ceph-mon[91136]: pgmap v7275: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:30 smithi078 ceph-mon[91136]: pgmap v7276: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:30 smithi078 ceph-mon[99959]: pgmap v7276: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:22:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:22:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:30 smithi146 ceph-mon[103278]: pgmap v7276: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:31.472 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:31.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:31.472 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:31.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:31.472 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:31.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:32 smithi146 ceph-mon[103278]: pgmap v7277: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:32 smithi078 ceph-mon[99959]: pgmap v7277: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:32 smithi078 ceph-mon[91136]: pgmap v7277: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:34 smithi146 ceph-mon[103278]: pgmap v7278: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:34 smithi078 ceph-mon[91136]: pgmap v7278: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:34 smithi078 ceph-mon[99959]: pgmap v7278: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:36 smithi146 ceph-mon[103278]: pgmap v7279: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:36 smithi078 ceph-mon[91136]: pgmap v7279: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:36 smithi078 ceph-mon[99959]: pgmap v7279: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:38 smithi146 ceph-mon[103278]: pgmap v7280: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:38 smithi078 ceph-mon[91136]: pgmap v7280: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:38 smithi078 ceph-mon[99959]: pgmap v7280: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:40 smithi146 ceph-mon[103278]: pgmap v7281: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[91136]: pgmap v7281: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:22:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:22:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:40 smithi078 ceph-mon[99959]: pgmap v7281: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:41.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:41.153Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:41.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:41.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:41.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:41.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:42 smithi146 ceph-mon[103278]: pgmap v7282: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:42 smithi078 ceph-mon[91136]: pgmap v7282: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:42 smithi078 ceph-mon[99959]: pgmap v7282: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:44.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[99959]: pgmap v7283: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[91136]: pgmap v7283: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:44 smithi146 ceph-mon[103278]: pgmap v7283: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:22:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:46 smithi078 ceph-mon[91136]: pgmap v7284: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:46 smithi078 ceph-mon[99959]: pgmap v7284: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:46 smithi146 ceph-mon[103278]: pgmap v7284: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:48 smithi146 ceph-mon[103278]: pgmap v7285: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:48 smithi078 ceph-mon[91136]: pgmap v7285: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:48 smithi078 ceph-mon[99959]: pgmap v7285: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:50.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:50 smithi078 ceph-mon[91136]: pgmap v7286: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:50.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:22:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-21T00:22:50.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:50 smithi078 ceph-mon[99959]: pgmap v7286: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:51.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:50 smithi146 ceph-mon[103278]: pgmap v7286: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:51.154Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:51.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:22:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:22:51.155Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:22:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:52 smithi146 ceph-mon[103278]: pgmap v7287: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:52 smithi078 ceph-mon[99959]: pgmap v7287: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:22:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:52 smithi078 ceph-mon[91136]: pgmap v7287: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:54 smithi146 ceph-mon[103278]: pgmap v7288: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:54 smithi078 ceph-mon[91136]: pgmap v7288: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:54 smithi078 ceph-mon[99959]: pgmap v7288: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:22:57.114 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:56 smithi078 ceph-mon[99959]: pgmap v7289: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:57.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:56 smithi078 ceph-mon[91136]: pgmap v7289: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:57.245 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:22:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:56 smithi146 ceph-mon[103278]: pgmap v7289: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 4h - - 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 4h - - 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 18s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 13s ago 4h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 18s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (3h) 13s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 18s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:22:57.773 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 18s ago 4h 423M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 13s ago 4h 277M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 18s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 18s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 13s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 18s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 18s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 13s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 13s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 13s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 13s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 13s ago 4h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:22:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:22:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:22:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:22:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:22:58.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:22:58.087 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:22:58.356 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:22:58.356 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:22:58.356 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:22:58.357 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:22:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:22:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:58.667 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:22:58.667 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:58.667 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: from='client.60567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: from='client.80492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: from='client.80498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: pgmap v7290: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1872106253' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3203679975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: from='client.60567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: from='client.80492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: from='client.80498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: pgmap v7290: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1872106253' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3203679975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: from='client.60567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: from='client.80492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: from='client.80498 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: pgmap v7290: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1872106253' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:22:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3203679975' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:59 smithi146 ceph-mon[103278]: from='client.80510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:22:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[91136]: from='client.80510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[99959]: from='client.80510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:22:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:00.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:23:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:23:01.156 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:00 smithi078 ceph-mon[99959]: pgmap v7291: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:01.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:00 smithi078 ceph-mon[91136]: pgmap v7291: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:00 smithi146 ceph-mon[103278]: pgmap v7291: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:01.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:01.155Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:01.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:01.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:01.491 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:01.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:02 smithi146 ceph-mon[103278]: pgmap v7292: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:02 smithi078 ceph-mon[91136]: pgmap v7292: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:02 smithi078 ceph-mon[99959]: pgmap v7292: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:05.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[99959]: pgmap v7293: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:05.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[91136]: pgmap v7293: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:05.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:04 smithi146 ceph-mon[103278]: pgmap v7293: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:06 smithi146 ceph-mon[103278]: pgmap v7294: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:06 smithi078 ceph-mon[99959]: pgmap v7294: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:06 smithi078 ceph-mon[91136]: pgmap v7294: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:08 smithi146 ceph-mon[103278]: pgmap v7295: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:08 smithi078 ceph-mon[91136]: pgmap v7295: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:08 smithi078 ceph-mon[99959]: pgmap v7295: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:10.888 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:23:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:23:11.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:10 smithi146 ceph-mon[103278]: pgmap v7296: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:11.156 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:10 smithi078 ceph-mon[91136]: pgmap v7296: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:11.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:10 smithi078 ceph-mon[99959]: pgmap v7296: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:11.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:11.156Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:11.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:11.156Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:11.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:11.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:13.231 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:12 smithi146 ceph-mon[103278]: pgmap v7297: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:12 smithi078 ceph-mon[91136]: pgmap v7297: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:12 smithi078 ceph-mon[99959]: pgmap v7297: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:14 smithi146 ceph-mon[103278]: pgmap v7298: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:14 smithi078 ceph-mon[91136]: pgmap v7298: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:14 smithi078 ceph-mon[99959]: pgmap v7298: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:16 smithi146 ceph-mon[103278]: pgmap v7299: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:16 smithi078 ceph-mon[99959]: pgmap v7299: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:16 smithi078 ceph-mon[91136]: pgmap v7299: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:18 smithi146 ceph-mon[103278]: pgmap v7300: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:18 smithi078 ceph-mon[91136]: pgmap v7300: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:18 smithi078 ceph-mon[99959]: pgmap v7300: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:20 smithi146 ceph-mon[103278]: pgmap v7301: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[91136]: pgmap v7301: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:23:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:20 smithi078 ceph-mon[99959]: pgmap v7301: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:21.508 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:21.157Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:21.508 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:21.157Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:21.508 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:21.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:22 smithi146 ceph-mon[103278]: pgmap v7302: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:22 smithi078 ceph-mon[91136]: pgmap v7302: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:22 smithi078 ceph-mon[99959]: pgmap v7302: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:24 smithi146 ceph-mon[103278]: pgmap v7303: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[99959]: pgmap v7303: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[91136]: pgmap v7303: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:27.128 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:26 smithi078 ceph-mon[99959]: pgmap v7304: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:26 smithi078 ceph-mon[91136]: pgmap v7304: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:26 smithi146 ceph-mon[103278]: pgmap v7304: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:28.956 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:23:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:28 smithi146 ceph-mon[103278]: pgmap v7305: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:28 smithi078 ceph-mon[91136]: pgmap v7305: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:28 smithi078 ceph-mon[99959]: pgmap v7305: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 4h - - 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 9s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 9s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 9s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 9s ago 4h 424M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:23:29.481 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 5s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3h) 9s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 9s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 9s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 9s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 9s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 9s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 5s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 5s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 5s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 5s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 9s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:23:29.482 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 9s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:23:29.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:23:29.789 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:23:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:23:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:23:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:23:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:23:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:23:29.817 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:23:30.057 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:23:30.057 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:23:30.058 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:23:30.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:29 smithi078 ceph-mon[99959]: from='client.80522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:30.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2904176306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:30.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:29 smithi078 ceph-mon[91136]: from='client.80522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:30.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2904176306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:29 smithi146 ceph-mon[103278]: from='client.80522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2904176306' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:23:30.368 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:23:30.368 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:23:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:23:30.368 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:23:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:23:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:23:30] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-21T00:23:31.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:30 smithi146 ceph-mon[103278]: from='client.80528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:31.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:30 smithi146 ceph-mon[103278]: from='client.60615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:31.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:30 smithi146 ceph-mon[103278]: pgmap v7306: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:31.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2022839527' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:31.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[91136]: from='client.80528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:31.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[91136]: from='client.60615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:31.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[91136]: pgmap v7306: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:31.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2022839527' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:31.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[99959]: from='client.80528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:31.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[99959]: from='client.60615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:31.159 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[99959]: pgmap v7306: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:31.159 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2022839527' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:23:31.431 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:31.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:31.431 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:31.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:31.431 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:31.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:32.117 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:31 smithi146 ceph-mon[103278]: from='client.80546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:31 smithi078 ceph-mon[91136]: from='client.80546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:31 smithi078 ceph-mon[99959]: from='client.80546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:23:33.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:33.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:32 smithi146 ceph-mon[103278]: pgmap v7307: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:32 smithi078 ceph-mon[91136]: pgmap v7307: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:32 smithi078 ceph-mon[99959]: pgmap v7307: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:34 smithi146 ceph-mon[103278]: pgmap v7308: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:34 smithi078 ceph-mon[91136]: pgmap v7308: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:34 smithi078 ceph-mon[99959]: pgmap v7308: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:36 smithi146 ceph-mon[103278]: pgmap v7309: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:36 smithi078 ceph-mon[91136]: pgmap v7309: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:36 smithi078 ceph-mon[99959]: pgmap v7309: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:38 smithi146 ceph-mon[103278]: pgmap v7310: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:38 smithi078 ceph-mon[91136]: pgmap v7310: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:38 smithi078 ceph-mon[99959]: pgmap v7310: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:23:40] "GET /metrics HTTP/1.1" 200 38990 "" "Prometheus/2.43.0" 2024-09-21T00:23:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[99959]: pgmap v7311: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[91136]: pgmap v7311: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:40 smithi146 ceph-mon[103278]: pgmap v7311: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:41.528 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:41.158Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:41.528 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:41.158Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:41.528 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:41.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:42 smithi146 ceph-mon[103278]: pgmap v7312: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:42 smithi078 ceph-mon[91136]: pgmap v7312: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:42 smithi078 ceph-mon[99959]: pgmap v7312: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:44.250 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:44 smithi146 ceph-mon[103278]: pgmap v7313: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[99959]: pgmap v7313: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[91136]: pgmap v7313: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:23:47.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:46 smithi078 ceph-mon[99959]: pgmap v7314: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:47.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:46 smithi078 ceph-mon[91136]: pgmap v7314: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:46 smithi146 ceph-mon[103278]: pgmap v7314: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:49.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:48 smithi078 ceph-mon[99959]: pgmap v7315: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:49.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:48 smithi078 ceph-mon[91136]: pgmap v7315: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:48 smithi146 ceph-mon[103278]: pgmap v7315: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:50.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:23:50] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:23:51.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:50 smithi078 ceph-mon[91136]: pgmap v7316: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:51.160 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:50 smithi078 ceph-mon[99959]: pgmap v7316: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:50 smithi146 ceph-mon[103278]: pgmap v7316: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:51.159Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:51.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:51.159Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:23:51.537 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:23:51.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:23:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:52 smithi146 ceph-mon[103278]: pgmap v7317: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:52 smithi078 ceph-mon[91136]: pgmap v7317: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:23:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:52 smithi078 ceph-mon[99959]: pgmap v7317: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:54 smithi146 ceph-mon[103278]: pgmap v7318: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:54 smithi078 ceph-mon[91136]: pgmap v7318: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:54 smithi078 ceph-mon[99959]: pgmap v7318: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:23:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:56 smithi146 ceph-mon[103278]: pgmap v7319: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:56 smithi078 ceph-mon[99959]: pgmap v7319: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:56 smithi078 ceph-mon[91136]: pgmap v7319: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:58 smithi146 ceph-mon[103278]: pgmap v7320: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:23:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:58 smithi078 ceph-mon[99959]: pgmap v7320: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:23:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:23:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:58 smithi078 ceph-mon[91136]: pgmap v7320: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:23:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:23:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:00.665 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:24:00.988 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:24:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 4h - - 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 4h - - 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 1s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 16s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 1s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:24:01.192 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 16s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 1s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 1s ago 4h 424M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (3h) 16s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 1s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 1s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 16s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 1s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 1s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 1s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 1s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 16s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 16s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 16s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:24:01.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 16s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:24:01.194 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 16s ago 4h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:24:01.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:24:01.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:24:01.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:24:01.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:24:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:00 smithi146 ceph-mon[103278]: pgmap v7321: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[99959]: pgmap v7321: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[91136]: pgmap v7321: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:01.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 9 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:01.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:01.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:24:01.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:24:01.503 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:24:01.772 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:24:01.773 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:24:01.773 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:24:01.773 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:24:01.773 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:24:01.773 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:24:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-mon[99959]: from='client.60639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-mon[99959]: from='client.60645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1260687654' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-mon[91136]: from='client.60639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-mon[91136]: from='client.60645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1260687654' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:02.086 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:24:02.086 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:02.086 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:02.086 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:02.086 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:01 smithi146 ceph-mon[103278]: from='client.60639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:01 smithi146 ceph-mon[103278]: from='client.60645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1260687654' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:02 smithi146 ceph-mon[103278]: from='client.60651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:02 smithi146 ceph-mon[103278]: from='client.60663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:02 smithi146 ceph-mon[103278]: pgmap v7322: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/718390537' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[91136]: from='client.60651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[91136]: from='client.60663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[91136]: pgmap v7322: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/718390537' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[99959]: from='client.60651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[99959]: from='client.60663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[99959]: pgmap v7322: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/718390537' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:05 smithi146 ceph-mon[103278]: pgmap v7323: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[91136]: pgmap v7323: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[99959]: pgmap v7323: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:07 smithi146 ceph-mon[103278]: pgmap v7324: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:07 smithi078 ceph-mon[91136]: pgmap v7324: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:07 smithi078 ceph-mon[99959]: pgmap v7324: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:09 smithi146 ceph-mon[103278]: pgmap v7325: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:24:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:09 smithi078 ceph-mon[91136]: pgmap v7325: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:24:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:09 smithi078 ceph-mon[99959]: pgmap v7325: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:24:11.014 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:24:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:24:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:11 smithi146 ceph-mon[103278]: pgmap v7326: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:11.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:11 smithi078 ceph-mon[91136]: pgmap v7326: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:11 smithi078 ceph-mon[99959]: pgmap v7326: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:11.160Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:11.160Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:11.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:13 smithi078 ceph-mon[91136]: pgmap v7327: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:13 smithi078 ceph-mon[99959]: pgmap v7327: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:13.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:13.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:13 smithi146 ceph-mon[103278]: pgmap v7327: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:14.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:15 smithi146 ceph-mon[103278]: pgmap v7328: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:15 smithi078 ceph-mon[91136]: pgmap v7328: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:15 smithi078 ceph-mon[99959]: pgmap v7328: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:16 smithi146 ceph-mon[103278]: pgmap v7329: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:16 smithi078 ceph-mon[91136]: pgmap v7329: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:16 smithi078 ceph-mon[99959]: pgmap v7329: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:18 smithi146 ceph-mon[103278]: pgmap v7330: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:18 smithi078 ceph-mon[91136]: pgmap v7330: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:18 smithi078 ceph-mon[99959]: pgmap v7330: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:20.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:24:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:24:21.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[91136]: pgmap v7331: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:21.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[99959]: pgmap v7331: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:21.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:20 smithi146 ceph-mon[103278]: pgmap v7331: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:21.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:21.161Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:21.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:22 smithi146 ceph-mon[103278]: pgmap v7332: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:23.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:22 smithi078 ceph-mon[91136]: pgmap v7332: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:22 smithi078 ceph-mon[99959]: pgmap v7332: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:24 smithi146 ceph-mon[103278]: pgmap v7333: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[91136]: pgmap v7333: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[99959]: pgmap v7333: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 00:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T00:24:25.850016968Z level=info msg="Completed cleanup jobs" duration=14.104317ms 2024-09-21T00:24:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:26 smithi146 ceph-mon[103278]: pgmap v7334: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:26 smithi078 ceph-mon[91136]: pgmap v7334: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:26 smithi078 ceph-mon[99959]: pgmap v7334: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:29.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:28 smithi078 ceph-mon[91136]: pgmap v7335: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:29.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:29.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:28 smithi078 ceph-mon[99959]: pgmap v7335: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:29.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:28 smithi146 ceph-mon[103278]: pgmap v7335: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:30.157 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:30 smithi078 ceph-mon[91136]: pgmap v7336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:30.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:30 smithi078 ceph-mon[99959]: pgmap v7336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:30 smithi146 ceph-mon[103278]: pgmap v7336: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:24:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:24:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:31.161Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:31.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:31.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:32.438 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:24:32.965 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 4h - - 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 4h - - 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 12s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 8s ago 4h 88.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 12s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 8s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 12s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 12s ago 4h 425M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 8s ago 4h 277M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 12s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 12s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 8s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:24:32.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 12s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 12s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 12s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 12s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 8s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 8s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 8s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 8s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 8s ago 4h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 12s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 12s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:24:32.967 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:24:33.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:33.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:32 smithi146 ceph-mon[103278]: pgmap v7337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:24:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:24:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:24:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:24:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:24:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:24:33.277 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:24:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:33.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:32 smithi078 ceph-mon[99959]: pgmap v7337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:33.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:33.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:32 smithi078 ceph-mon[91136]: pgmap v7337: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:24:33.541 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:24:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:24:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:24:33.542 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:24:33.542 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:24:33.847 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:24:33.848 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:33.848 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:24:33.848 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:33.848 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:24:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:33 smithi146 ceph-mon[103278]: from='client.80594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:33 smithi146 ceph-mon[103278]: from='client.60681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:33 smithi146 ceph-mon[103278]: from='client.80606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/434329189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1542243220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[91136]: from='client.80594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[91136]: from='client.60681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[91136]: from='client.80606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/434329189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1542243220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[99959]: from='client.80594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[99959]: from='client.60681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[99959]: from='client.80606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/434329189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:24:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1542243220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:24:35.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:34 smithi146 ceph-mon[103278]: from='client.60699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:35.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:34 smithi146 ceph-mon[103278]: pgmap v7338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:34 smithi078 ceph-mon[91136]: from='client.60699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:34 smithi078 ceph-mon[91136]: pgmap v7338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:34 smithi078 ceph-mon[99959]: from='client.60699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:24:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:34 smithi078 ceph-mon[99959]: pgmap v7338: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:36 smithi146 ceph-mon[103278]: pgmap v7339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:36 smithi078 ceph-mon[91136]: pgmap v7339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:36 smithi078 ceph-mon[99959]: pgmap v7339: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:38 smithi146 ceph-mon[103278]: pgmap v7340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:38 smithi078 ceph-mon[91136]: pgmap v7340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:38 smithi078 ceph-mon[99959]: pgmap v7340: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:40.880 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:24:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[99959]: pgmap v7341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[91136]: pgmap v7341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:40 smithi146 ceph-mon[103278]: pgmap v7341: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:41.162Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:41.162Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:41.163Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:42 smithi146 ceph-mon[103278]: pgmap v7342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:42 smithi078 ceph-mon[99959]: pgmap v7342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:42 smithi078 ceph-mon[91136]: pgmap v7342: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:44 smithi146 ceph-mon[103278]: pgmap v7343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:44 smithi078 ceph-mon[99959]: pgmap v7343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:44 smithi078 ceph-mon[91136]: pgmap v7343: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:24:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:46 smithi146 ceph-mon[103278]: pgmap v7344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:46 smithi078 ceph-mon[99959]: pgmap v7344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:46 smithi078 ceph-mon[91136]: pgmap v7344: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:24:49.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:48 smithi078 ceph-mon[99959]: pgmap v7345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:24:49.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:48 smithi078 ceph-mon[91136]: pgmap v7345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:24:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:48 smithi146 ceph-mon[103278]: pgmap v7345: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:24:50.934 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:24:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:24:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:50 smithi146 ceph-mon[103278]: pgmap v7346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:51.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:50 smithi078 ceph-mon[91136]: pgmap v7346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:50 smithi078 ceph-mon[99959]: pgmap v7346: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:51.163Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:51.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:24:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:24:51.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:24:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:52 smithi146 ceph-mon[103278]: pgmap v7347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:52 smithi078 ceph-mon[99959]: pgmap v7347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:24:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:52 smithi078 ceph-mon[91136]: pgmap v7347: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:54 smithi146 ceph-mon[103278]: pgmap v7348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:54 smithi078 ceph-mon[91136]: pgmap v7348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:54 smithi078 ceph-mon[99959]: pgmap v7348: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:24:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:56 smithi146 ceph-mon[103278]: pgmap v7349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:56 smithi078 ceph-mon[91136]: pgmap v7349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:56 smithi078 ceph-mon[99959]: pgmap v7349: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:58 smithi146 ceph-mon[103278]: pgmap v7350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:58 smithi078 ceph-mon[91136]: pgmap v7350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:24:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:58 smithi078 ceph-mon[99959]: pgmap v7350: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:24:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:25:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:25:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:00 smithi146 ceph-mon[103278]: pgmap v7351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[91136]: pgmap v7351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[99959]: pgmap v7351: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:01.164Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:01.164Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:01.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:02 smithi146 ceph-mon[103278]: pgmap v7352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:03.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:02 smithi078 ceph-mon[91136]: pgmap v7352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:02 smithi078 ceph-mon[99959]: pgmap v7352: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:04.147 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:25:04.673 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 4h - - 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 4h - - 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 4s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 88.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 4s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 4s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 4s ago 4h 426M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 4s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 4s ago 4h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (3h) 4s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:25:04.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 4s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 4s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 4s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 19s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 19s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 19s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 19s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 19s ago 4h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:25:04.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:25:04.983 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:25:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:25:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:25:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:25:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:25:04.984 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:25:05.258 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:25:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:25:05.259 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:25:05.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:25:05.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:04 smithi078 ceph-mon[99959]: pgmap v7353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:05.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:04 smithi078 ceph-mon[91136]: pgmap v7353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:04 smithi146 ceph-mon[103278]: pgmap v7353: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:05.569 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:25:05.569 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:25:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:25:05.569 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:25:05.569 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='client.80630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='client.60717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='client.60723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2656585482' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2826713509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='client.80630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='client.60717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='client.60723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2656585482' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2826713509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='client.80630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='client.60717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='client.60723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2656585482' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2826713509' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:06 smithi146 ceph-mon[103278]: from='client.80654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:06 smithi146 ceph-mon[103278]: pgmap v7354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:06 smithi078 ceph-mon[91136]: from='client.80654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:06 smithi078 ceph-mon[91136]: pgmap v7354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:06 smithi078 ceph-mon[99959]: from='client.80654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:06 smithi078 ceph-mon[99959]: pgmap v7354: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:09.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:08 smithi078 ceph-mon[91136]: pgmap v7355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:09.250 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:08 smithi078 ceph-mon[99959]: pgmap v7355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:08 smithi146 ceph-mon[103278]: pgmap v7355: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:10.295 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:10 smithi078 ceph-mon[99959]: pgmap v7356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:10.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:10 smithi078 ceph-mon[91136]: pgmap v7356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:10 smithi146 ceph-mon[103278]: pgmap v7356: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:25:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:25:11.394 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:11.164Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:11.394 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:11.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:11.394 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:11.165Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:12 smithi078 ceph-mon[91136]: pgmap v7357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:12 smithi078 ceph-mon[99959]: pgmap v7357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:13.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:13.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:12 smithi146 ceph-mon[103278]: pgmap v7357: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:14.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:14 smithi146 ceph-mon[103278]: pgmap v7358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:14 smithi078 ceph-mon[99959]: pgmap v7358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:14 smithi078 ceph-mon[91136]: pgmap v7358: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:17.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:16 smithi146 ceph-mon[103278]: pgmap v7359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:16 smithi078 ceph-mon[91136]: pgmap v7359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:16 smithi078 ceph-mon[99959]: pgmap v7359: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:18 smithi146 ceph-mon[103278]: pgmap v7360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:18 smithi078 ceph-mon[91136]: pgmap v7360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:18 smithi078 ceph-mon[99959]: pgmap v7360: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:20.893 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:25:20] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:25:21.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[91136]: pgmap v7361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[99959]: pgmap v7361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:20 smithi146 ceph-mon[103278]: pgmap v7361: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:21.166Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:21.166Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:21.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:22 smithi146 ceph-mon[103278]: pgmap v7362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:22 smithi078 ceph-mon[91136]: pgmap v7362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:22 smithi078 ceph-mon[99959]: pgmap v7362: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:25.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:24 smithi146 ceph-mon[103278]: pgmap v7363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:24 smithi078 ceph-mon[91136]: pgmap v7363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:24 smithi078 ceph-mon[99959]: pgmap v7363: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:26 smithi146 ceph-mon[103278]: pgmap v7364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[91136]: pgmap v7364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:26 smithi078 ceph-mon[99959]: pgmap v7364: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:28 smithi146 ceph-mon[103278]: pgmap v7365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:28 smithi078 ceph-mon[91136]: pgmap v7365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:28 smithi078 ceph-mon[99959]: pgmap v7365: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:30.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:25:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:25:31.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:30 smithi078 ceph-mon[91136]: pgmap v7366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:31.167 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:30 smithi078 ceph-mon[99959]: pgmap v7366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:30 smithi146 ceph-mon[103278]: pgmap v7366: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:31.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:31.166Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:31.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:31.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:31.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:31.167Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:32 smithi146 ceph-mon[103278]: pgmap v7367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:32 smithi078 ceph-mon[91136]: pgmap v7367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:32 smithi078 ceph-mon[99959]: pgmap v7367: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:34 smithi146 ceph-mon[103278]: pgmap v7368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:34 smithi078 ceph-mon[91136]: pgmap v7368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:34 smithi078 ceph-mon[99959]: pgmap v7368: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:35.864 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 4h - - 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 4h - - 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 15s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 11s ago 4h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 15s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 11s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 15s ago 4h 579M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 15s ago 4h 428M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:25:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 11s ago 4h 276M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 15s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 15s ago 4h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 11s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 15s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 15s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 15s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 15s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 11s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 11s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 11s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 11s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 11s ago 4h 75.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 15s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:25:36.388 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 15s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:25:36.389 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:25:36.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:25:36.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:25:36.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:25:36.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:25:36.696 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:25:36.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:36 smithi146 ceph-mon[103278]: pgmap v7369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:36.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:36 smithi146 ceph-mon[103278]: from='client.80666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:36.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1280126835' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:36.963 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:25:36.964 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:25:36.965 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:25:36.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:36 smithi078 ceph-mon[99959]: pgmap v7369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:36.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:36 smithi078 ceph-mon[99959]: from='client.80666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:36.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1280126835' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:36.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:36 smithi078 ceph-mon[91136]: pgmap v7369: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:36.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:36 smithi078 ceph-mon[91136]: from='client.80666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:36.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1280126835' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:25:37.269 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:25:37.270 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:25:37.270 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:25:37.270 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:25:37.270 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:25:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:37 smithi146 ceph-mon[103278]: from='client.80672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:37 smithi146 ceph-mon[103278]: from='client.80678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:37 smithi146 ceph-mon[103278]: from='client.80690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3449613137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[91136]: from='client.80672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[91136]: from='client.80678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[91136]: from='client.80690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3449613137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[99959]: from='client.80672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[99959]: from='client.80678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[99959]: from='client.80690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:25:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3449613137' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:25:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:38 smithi146 ceph-mon[103278]: pgmap v7370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:38 smithi078 ceph-mon[91136]: pgmap v7370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:38 smithi078 ceph-mon[99959]: pgmap v7370: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:40.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:25:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:25:41.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[91136]: pgmap v7371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:41.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[99959]: pgmap v7371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:41.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:40 smithi146 ceph-mon[103278]: pgmap v7371: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:41.167Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:41.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:41.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:42 smithi146 ceph-mon[103278]: pgmap v7372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:42 smithi078 ceph-mon[91136]: pgmap v7372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:42 smithi078 ceph-mon[99959]: pgmap v7372: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:44.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:44 smithi146 ceph-mon[103278]: pgmap v7373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:44 smithi078 ceph-mon[91136]: pgmap v7373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:44 smithi078 ceph-mon[99959]: pgmap v7373: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:46 smithi146 ceph-mon[103278]: pgmap v7374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[91136]: pgmap v7374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:25:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:46 smithi078 ceph-mon[99959]: pgmap v7374: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:48 smithi146 ceph-mon[103278]: pgmap v7375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:48 smithi078 ceph-mon[91136]: pgmap v7375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:48 smithi078 ceph-mon[99959]: pgmap v7375: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:50.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:25:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:25:51.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:50 smithi078 ceph-mon[99959]: pgmap v7376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:51.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:50 smithi078 ceph-mon[91136]: pgmap v7376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:50 smithi146 ceph-mon[103278]: pgmap v7376: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:51.168Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:51.168Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:25:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:25:51.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:25:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:52 smithi078 ceph-mon[91136]: pgmap v7377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:53.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:53.260 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:52 smithi078 ceph-mon[99959]: pgmap v7377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:25:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:52 smithi146 ceph-mon[103278]: pgmap v7377: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:55.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:54 smithi146 ceph-mon[103278]: pgmap v7378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:54 smithi078 ceph-mon[99959]: pgmap v7378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:54 smithi078 ceph-mon[91136]: pgmap v7378: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:25:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:56 smithi146 ceph-mon[103278]: pgmap v7379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:56 smithi078 ceph-mon[91136]: pgmap v7379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:56 smithi078 ceph-mon[99959]: pgmap v7379: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:58 smithi146 ceph-mon[103278]: pgmap v7380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:58 smithi078 ceph-mon[91136]: pgmap v7380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:25:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:58 smithi078 ceph-mon[99959]: pgmap v7380: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:25:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:00.915 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:26:00] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:26:01.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[99959]: pgmap v7381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:01.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[91136]: pgmap v7381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:01.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:00 smithi146 ceph-mon[103278]: pgmap v7381: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:01.169Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:01.169Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:01.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:02 smithi078 ceph-mon[91136]: pgmap v7382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:02 smithi078 ceph-mon[99959]: pgmap v7382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:02 smithi146 ceph-mon[103278]: pgmap v7382: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:04 smithi146 ceph-mon[103278]: pgmap v7383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:04 smithi078 ceph-mon[91136]: pgmap v7383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:04 smithi078 ceph-mon[99959]: pgmap v7383: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:06 smithi146 ceph-mon[103278]: pgmap v7384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[91136]: pgmap v7384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:06 smithi078 ceph-mon[99959]: pgmap v7384: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:07.564 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:26:08.091 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:26:08.091 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 4h - - 2024-09-21T00:26:08.091 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 4h - - 2024-09-21T00:26:08.091 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 7s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:26:08.091 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 2s ago 4h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:26:08.091 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 7s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 2s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 7s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 7s ago 4h 428M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 2s ago 4h 278M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 7s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 7s ago 4h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 2s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 7s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (3h) 7s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 7s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 7s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 2s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 2s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:26:08.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 2s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:26:08.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 2s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:26:08.093 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 2s ago 4h 75.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:26:08.093 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:26:08.093 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:26:08.093 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:26:08.093 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:26:08.402 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:26:08.402 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:26:08.402 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:26:08.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:08.402 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:26:08.402 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:26:08.403 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:26:08.672 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:26:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:26:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:26:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:26:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:26:08.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:26:08.984 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:26:08.984 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:08.985 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:08.985 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:08.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[99959]: from='client.80702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:08.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[99959]: from='client.60789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:08.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[99959]: pgmap v7385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:08.985 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3020462321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:08.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[91136]: from='client.80702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:08.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[91136]: from='client.60789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:08.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[91136]: pgmap v7385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:08.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3020462321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:09.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:08 smithi146 ceph-mon[103278]: from='client.80702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:09.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:08 smithi146 ceph-mon[103278]: from='client.60789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:09.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:08 smithi146 ceph-mon[103278]: pgmap v7385: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:09.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3020462321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:09 smithi146 ceph-mon[103278]: from='client.80714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:09 smithi146 ceph-mon[103278]: from='client.80726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3175819555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:09 smithi078 ceph-mon[99959]: from='client.80714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:09 smithi078 ceph-mon[99959]: from='client.80726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3175819555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:09 smithi078 ceph-mon[91136]: from='client.80714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:09 smithi078 ceph-mon[91136]: from='client.80726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3175819555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:10.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:26:10] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:26:11.170 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:10 smithi078 ceph-mon[99959]: pgmap v7386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:11.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:10 smithi078 ceph-mon[91136]: pgmap v7386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:10 smithi146 ceph-mon[103278]: pgmap v7386: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:11.515 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:11.170Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:11.515 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:11.170Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:11.516 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:11.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:12 smithi146 ceph-mon[103278]: pgmap v7387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:12 smithi078 ceph-mon[91136]: pgmap v7387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:12 smithi078 ceph-mon[99959]: pgmap v7387: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:14.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:14 smithi146 ceph-mon[103278]: pgmap v7388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:14 smithi078 ceph-mon[91136]: pgmap v7388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:14 smithi078 ceph-mon[99959]: pgmap v7388: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:16 smithi146 ceph-mon[103278]: pgmap v7389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:16 smithi078 ceph-mon[91136]: pgmap v7389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:16 smithi078 ceph-mon[99959]: pgmap v7389: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:18 smithi146 ceph-mon[103278]: pgmap v7390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:18 smithi078 ceph-mon[91136]: pgmap v7390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:18 smithi078 ceph-mon[99959]: pgmap v7390: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:20.922 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:26:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:26:21.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:20 smithi078 ceph-mon[91136]: pgmap v7391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:21.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:20 smithi078 ceph-mon[99959]: pgmap v7391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:21.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:21.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:21.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:21.171Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:21.173 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:21.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:20 smithi146 ceph-mon[103278]: pgmap v7391: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:22 smithi146 ceph-mon[103278]: pgmap v7392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:23.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:22 smithi078 ceph-mon[99959]: pgmap v7392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:22 smithi078 ceph-mon[91136]: pgmap v7392: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:24 smithi146 ceph-mon[103278]: pgmap v7393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:24 smithi078 ceph-mon[91136]: pgmap v7393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:24 smithi078 ceph-mon[99959]: pgmap v7393: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:26 smithi146 ceph-mon[103278]: pgmap v7394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[99959]: pgmap v7394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:26 smithi078 ceph-mon[91136]: pgmap v7394: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:28 smithi146 ceph-mon[103278]: pgmap v7395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:28 smithi078 ceph-mon[99959]: pgmap v7395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:28 smithi078 ceph-mon[91136]: pgmap v7395: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:30.907 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:26:30] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:26:31.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:30 smithi078 ceph-mon[99959]: pgmap v7396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:31.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:30 smithi078 ceph-mon[91136]: pgmap v7396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:30 smithi146 ceph-mon[103278]: pgmap v7396: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:31.171Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:31.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:31.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:33.171 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:33.171 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:32 smithi078 ceph-mon[99959]: pgmap v7397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:33.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:33.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:32 smithi078 ceph-mon[91136]: pgmap v7397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:32 smithi146 ceph-mon[103278]: pgmap v7397: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:34 smithi146 ceph-mon[103278]: pgmap v7398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:34 smithi078 ceph-mon[91136]: pgmap v7398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:34 smithi078 ceph-mon[99959]: pgmap v7398: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:37.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:36 smithi146 ceph-mon[103278]: pgmap v7399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:36 smithi078 ceph-mon[91136]: pgmap v7399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:36 smithi078 ceph-mon[99959]: pgmap v7399: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:39.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:38 smithi146 ceph-mon[103278]: pgmap v7400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:39.281 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:38 smithi078 ceph-mon[99959]: pgmap v7400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:39.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:38 smithi078 ceph-mon[91136]: pgmap v7400: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:39.298 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:26:39.807 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:26:39.807 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 4h - - 2024-09-21T00:26:39.807 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 4h - - 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 18s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 14s ago 4h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 18s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 14s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 18s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 18s ago 4h 428M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 14s ago 4h 279M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 18s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 18s ago 4h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:26:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 14s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 18s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 18s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (3h) 18s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 18s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 14s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 14s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 14s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 14s ago 4h 75.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 18s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:26:39.809 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:26:39.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 18s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:26:39.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:26:40.117 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:26:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:26:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:26:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:26:40.119 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:26:40.384 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:26:40.385 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:26:40.693 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:26:40.693 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:40.693 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:26:40.693 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:40.693 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:26:40.694 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:26:40] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[99959]: from='client.60819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[99959]: from='client.60825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[99959]: from='client.60831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[99959]: pgmap v7401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3349287259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3239034507' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[91136]: from='client.60819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[91136]: from='client.60825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[91136]: from='client.60831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[91136]: pgmap v7401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3349287259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3239034507' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:40 smithi146 ceph-mon[103278]: from='client.60819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:40 smithi146 ceph-mon[103278]: from='client.60825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:40 smithi146 ceph-mon[103278]: from='client.60831 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:40 smithi146 ceph-mon[103278]: pgmap v7401: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3349287259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:26:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3239034507' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:26:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:41.172Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:41.172Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:41.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[91136]: from='client.60843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[99959]: from='client.60843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:41 smithi146 ceph-mon[103278]: from='client.60843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:26:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:42 smithi146 ceph-mon[103278]: pgmap v7402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:42 smithi078 ceph-mon[91136]: pgmap v7402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:42 smithi078 ceph-mon[99959]: pgmap v7402: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:44 smithi146 ceph-mon[103278]: pgmap v7403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:44 smithi078 ceph-mon[91136]: pgmap v7403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:44 smithi078 ceph-mon[99959]: pgmap v7403: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:46 smithi146 ceph-mon[103278]: pgmap v7404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[91136]: pgmap v7404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:26:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:46 smithi078 ceph-mon[99959]: pgmap v7404: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:48 smithi146 ceph-mon[103278]: pgmap v7405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:48 smithi078 ceph-mon[99959]: pgmap v7405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:48 smithi078 ceph-mon[91136]: pgmap v7405: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:50.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:26:50] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:26:51.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:50 smithi078 ceph-mon[91136]: pgmap v7406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:51.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:50 smithi078 ceph-mon[99959]: pgmap v7406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:50 smithi146 ceph-mon[103278]: pgmap v7406: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:51.172Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:51.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:26:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:26:51.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:26:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:52 smithi146 ceph-mon[103278]: pgmap v7407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:53.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:52 smithi078 ceph-mon[91136]: pgmap v7407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:26:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:52 smithi078 ceph-mon[99959]: pgmap v7407: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:55.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:54 smithi078 ceph-mon[91136]: pgmap v7408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:55.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:54 smithi078 ceph-mon[99959]: pgmap v7408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:54 smithi146 ceph-mon[103278]: pgmap v7408: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:26:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:56 smithi146 ceph-mon[103278]: pgmap v7409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:56 smithi078 ceph-mon[91136]: pgmap v7409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:56 smithi078 ceph-mon[99959]: pgmap v7409: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:58 smithi146 ceph-mon[103278]: pgmap v7410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:58 smithi078 ceph-mon[99959]: pgmap v7410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:26:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:58 smithi078 ceph-mon[91136]: pgmap v7410: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:26:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:00.929 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:27:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:27:01.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:00 smithi078 ceph-mon[91136]: pgmap v7411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:01.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:00 smithi078 ceph-mon[99959]: pgmap v7411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:01.190 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:01.173Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:01.190 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:01.173Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:01.190 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:01.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:00 smithi146 ceph-mon[103278]: pgmap v7411: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:02 smithi146 ceph-mon[103278]: pgmap v7412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[99959]: pgmap v7412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:02 smithi078 ceph-mon[91136]: pgmap v7412: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:04 smithi146 ceph-mon[103278]: pgmap v7413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:04 smithi078 ceph-mon[91136]: pgmap v7413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:04 smithi078 ceph-mon[99959]: pgmap v7413: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:06 smithi146 ceph-mon[103278]: pgmap v7414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[91136]: pgmap v7414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:06 smithi078 ceph-mon[99959]: pgmap v7414: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:08 smithi146 ceph-mon[103278]: pgmap v7415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:08 smithi078 ceph-mon[91136]: pgmap v7415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:08 smithi078 ceph-mon[99959]: pgmap v7415: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:10.921 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:27:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:27:10.989 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:27:11.174 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:10 smithi078 ceph-mon[99959]: pgmap v7416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:11.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:10 smithi078 ceph-mon[91136]: pgmap v7416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:10 smithi146 ceph-mon[103278]: pgmap v7416: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:11.514 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:27:11.514 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 4h - - 2024-09-21T00:27:11.514 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 10s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 10s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 10s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 10s ago 4h 428M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 5s ago 4h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 10s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 10s ago 4h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 10s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 10s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 10s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (3h) 10s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:27:11.515 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 5s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 5s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 5s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 5s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 75.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 10s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 10s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:27:11.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:27:11.516 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:11.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:11.516 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:11.174Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:11.516 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:11.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:11.822 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:27:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:27:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:27:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:27:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:27:11.824 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:27:12.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:27:12.093 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:27:12.093 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:27:12.093 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:27:12.093 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:27:12.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:27:12.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-mon[91136]: from='client.80774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:12.094 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3992270392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:12.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-mon[99959]: from='client.80774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:12.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3992270392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:11 smithi146 ceph-mon[103278]: from='client.80774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3992270392' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:12.404 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:27:12.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:12.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:12.405 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:13.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:12 smithi146 ceph-mon[103278]: from='client.60861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:13.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:12 smithi146 ceph-mon[103278]: from='client.80786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:13.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:13.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:12 smithi146 ceph-mon[103278]: pgmap v7417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:13.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1465765310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[99959]: from='client.60861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[99959]: from='client.80786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[99959]: pgmap v7417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1465765310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[91136]: from='client.60861 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[91136]: from='client.80786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[91136]: pgmap v7417: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1465765310' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:13 smithi146 ceph-mon[103278]: from='client.60879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:13 smithi078 ceph-mon[91136]: from='client.60879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:13 smithi078 ceph-mon[99959]: from='client.60879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:14 smithi146 ceph-mon[103278]: pgmap v7418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:14 smithi078 ceph-mon[91136]: pgmap v7418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:14 smithi078 ceph-mon[99959]: pgmap v7418: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:17.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:16 smithi146 ceph-mon[103278]: pgmap v7419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:16 smithi078 ceph-mon[91136]: pgmap v7419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:16 smithi078 ceph-mon[99959]: pgmap v7419: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:18 smithi146 ceph-mon[103278]: pgmap v7420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:18 smithi078 ceph-mon[91136]: pgmap v7420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:18 smithi078 ceph-mon[99959]: pgmap v7420: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:20.945 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:27:20] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:27:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:20 smithi146 ceph-mon[103278]: pgmap v7421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:21.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:20 smithi078 ceph-mon[99959]: pgmap v7421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:20 smithi078 ceph-mon[91136]: pgmap v7421: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:21.174Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:21.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:21.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:22.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[99959]: pgmap v7422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:22.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:22 smithi078 ceph-mon[91136]: pgmap v7422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:22 smithi146 ceph-mon[103278]: pgmap v7422: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:24 smithi146 ceph-mon[103278]: pgmap v7423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:24 smithi078 ceph-mon[91136]: pgmap v7423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:24 smithi078 ceph-mon[99959]: pgmap v7423: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:26 smithi146 ceph-mon[103278]: pgmap v7424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[91136]: pgmap v7424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[99959]: pgmap v7424: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:28 smithi146 ceph-mon[103278]: pgmap v7425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:28 smithi078 ceph-mon[91136]: pgmap v7425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:28 smithi078 ceph-mon[99959]: pgmap v7425: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:30.972 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:27:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:27:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:30 smithi146 ceph-mon[103278]: pgmap v7426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:30 smithi078 ceph-mon[91136]: pgmap v7426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:30 smithi078 ceph-mon[99959]: pgmap v7426: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:31.175Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:31.175Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:31.176Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:32 smithi146 ceph-mon[103278]: pgmap v7427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:32 smithi078 ceph-mon[99959]: pgmap v7427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:32 smithi078 ceph-mon[91136]: pgmap v7427: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:35.165 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:34 smithi078 ceph-mon[99959]: pgmap v7428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:35.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:34 smithi078 ceph-mon[91136]: pgmap v7428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:34 smithi146 ceph-mon[103278]: pgmap v7428: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:36 smithi078 ceph-mon[91136]: pgmap v7429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:36 smithi078 ceph-mon[99959]: pgmap v7429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:36 smithi146 ceph-mon[103278]: pgmap v7429: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:39.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:38 smithi146 ceph-mon[103278]: pgmap v7430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:38 smithi078 ceph-mon[91136]: pgmap v7430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:38 smithi078 ceph-mon[99959]: pgmap v7430: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:40.987 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:27:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:27:41.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:40 smithi146 ceph-mon[103278]: pgmap v7431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:41.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:40 smithi078 ceph-mon[91136]: pgmap v7431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:41.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:40 smithi078 ceph-mon[99959]: pgmap v7431: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:41.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:41.176Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:41.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:41.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:41.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:41.177Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:42.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[99959]: pgmap v7432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:42.703 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:42 smithi078 ceph-mon[91136]: pgmap v7432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:42.703 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:27:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:42 smithi146 ceph-mon[103278]: pgmap v7432: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:43.228 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:27:43.228 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 4h - - 2024-09-21T00:27:43.228 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 4h - - 2024-09-21T00:27:43.228 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 1s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:27:43.228 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 17s ago 4h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:27:43.228 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 1s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 17s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 1s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 1s ago 4h 430M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 17s ago 4h 281M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 1s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 1s ago 4h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:27:43.229 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 17s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:27:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 1s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:27:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 1s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:27:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 1s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:27:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 1s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:27:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (3h) 17s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:27:43.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 17s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:27:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 17s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:27:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 17s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:27:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 17s ago 4h 75.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:27:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 1s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:27:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:27:43.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 1s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:27:43.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:27:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:43 smithi078 ceph-mon[99959]: from='client.60891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:43 smithi078 ceph-mon[99959]: from='client.60897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:43 smithi078 ceph-mon[91136]: from='client.60891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:43 smithi078 ceph-mon[91136]: from='client.60897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:43.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:27:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:27:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:27:43.539 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:27:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:43 smithi146 ceph-mon[103278]: from='client.60891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:43 smithi146 ceph-mon[103278]: from='client.60897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:43.805 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:27:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:27:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:27:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:27:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:27:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:27:43.807 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:27:44.116 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:27:44.116 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:27:44.116 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:27:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:44 smithi146 ceph-mon[103278]: from='client.60903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4080390492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:44 smithi146 ceph-mon[103278]: from='client.60915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:44 smithi146 ceph-mon[103278]: pgmap v7433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1167383980' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[91136]: from='client.60903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4080390492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[91136]: from='client.60915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[91136]: pgmap v7433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1167383980' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[99959]: from='client.60903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4080390492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:27:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[99959]: from='client.60915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:27:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[99959]: pgmap v7433: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1167383980' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[91136]: pgmap v7434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[99959]: pgmap v7434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:46 smithi146 ceph-mon[103278]: pgmap v7434: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:27:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:48 smithi146 ceph-mon[103278]: pgmap v7435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:48 smithi078 ceph-mon[91136]: pgmap v7435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:48 smithi078 ceph-mon[99959]: pgmap v7435: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:50.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:27:50] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-21T00:27:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:50 smithi146 ceph-mon[103278]: pgmap v7436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:50 smithi078 ceph-mon[91136]: pgmap v7436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:50 smithi078 ceph-mon[99959]: pgmap v7436: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:51.177Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:51.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:27:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:27:51.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:27:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:52 smithi146 ceph-mon[103278]: pgmap v7437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:52 smithi078 ceph-mon[91136]: pgmap v7437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:27:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:52 smithi078 ceph-mon[99959]: pgmap v7437: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:55.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:54 smithi078 ceph-mon[99959]: pgmap v7438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:55.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:54 smithi078 ceph-mon[91136]: pgmap v7438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:54 smithi146 ceph-mon[103278]: pgmap v7438: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:56 smithi078 ceph-mon[91136]: pgmap v7439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:56 smithi078 ceph-mon[99959]: pgmap v7439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:56 smithi146 ceph-mon[103278]: pgmap v7439: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:27:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:58 smithi146 ceph-mon[103278]: pgmap v7440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:58 smithi078 ceph-mon[91136]: pgmap v7440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:27:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:58 smithi078 ceph-mon[99959]: pgmap v7440: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:27:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:00.952 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:28:00] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:28:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:00 smithi146 ceph-mon[103278]: pgmap v7441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:00 smithi078 ceph-mon[91136]: pgmap v7441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:00 smithi078 ceph-mon[99959]: pgmap v7441: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:01.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:01.178Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:01.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[91136]: pgmap v7442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:02.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:02.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:02 smithi078 ceph-mon[99959]: pgmap v7442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:02 smithi146 ceph-mon[103278]: pgmap v7442: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:04 smithi146 ceph-mon[103278]: pgmap v7443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:04 smithi078 ceph-mon[99959]: pgmap v7443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:04 smithi078 ceph-mon[91136]: pgmap v7443: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:06 smithi146 ceph-mon[103278]: pgmap v7444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[91136]: pgmap v7444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[99959]: pgmap v7444: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:08 smithi146 ceph-mon[103278]: pgmap v7445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:08 smithi078 ceph-mon[91136]: pgmap v7445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:08 smithi078 ceph-mon[99959]: pgmap v7445: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:10.947 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:28:10] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:28:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:10 smithi146 ceph-mon[103278]: pgmap v7446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:10 smithi078 ceph-mon[91136]: pgmap v7446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:10 smithi078 ceph-mon[99959]: pgmap v7446: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:11.178Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:11.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:11.179Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:12 smithi146 ceph-mon[103278]: pgmap v7447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:12 smithi078 ceph-mon[91136]: pgmap v7447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:12 smithi078 ceph-mon[99959]: pgmap v7447: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:14.412 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:28:14.938 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:28:14.938 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 4h - - 2024-09-21T00:28:14.938 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 4h - - 2024-09-21T00:28:14.938 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 13s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:28:14.938 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 8s ago 4h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 13s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 8s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 13s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 13s ago 4h 430M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 8s ago 4h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 13s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 13s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 8s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 13s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 13s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 13s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 13s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 8s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:28:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 8s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 8s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 8s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 8s ago 4h 75.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 13s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 13s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:28:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:28:15.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:28:15.250 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:28:15.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:14 smithi078 ceph-mon[99959]: pgmap v7448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:15.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:14 smithi078 ceph-mon[91136]: pgmap v7448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:14 smithi146 ceph-mon[103278]: pgmap v7448: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:15.527 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:28:15.527 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:28:15.527 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:28:15.527 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:28:15.528 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:28:15.845 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:28:15.845 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:15.845 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:16.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[99959]: from='client.60927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[99959]: from='client.60933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[99959]: from='client.80858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3206039737' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:16.168 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1099075871' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:16.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[91136]: from='client.60927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[91136]: from='client.60933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[91136]: from='client.80858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3206039737' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:16.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1099075871' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:15 smithi146 ceph-mon[103278]: from='client.60927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:15 smithi146 ceph-mon[103278]: from='client.60933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:15 smithi146 ceph-mon[103278]: from='client.80858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3206039737' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1099075871' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:16 smithi146 ceph-mon[103278]: from='client.80870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:16 smithi146 ceph-mon[103278]: pgmap v7449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:16 smithi078 ceph-mon[91136]: from='client.80870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:16 smithi078 ceph-mon[91136]: pgmap v7449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:16 smithi078 ceph-mon[99959]: from='client.80870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:16 smithi078 ceph-mon[99959]: pgmap v7449: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:19.089 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:18 smithi146 ceph-mon[103278]: pgmap v7450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:18 smithi078 ceph-mon[91136]: pgmap v7450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:18 smithi078 ceph-mon[99959]: pgmap v7450: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:20.978 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:28:20] "GET /metrics HTTP/1.1" 200 38989 "" "Prometheus/2.43.0" 2024-09-21T00:28:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:20 smithi146 ceph-mon[103278]: pgmap v7451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:20 smithi078 ceph-mon[91136]: pgmap v7451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:20 smithi078 ceph-mon[99959]: pgmap v7451: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:21.179Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:21.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:21.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:22 smithi146 ceph-mon[103278]: pgmap v7452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[91136]: pgmap v7452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:22 smithi078 ceph-mon[99959]: pgmap v7452: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:24 smithi146 ceph-mon[103278]: pgmap v7453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:24 smithi078 ceph-mon[91136]: pgmap v7453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:24 smithi078 ceph-mon[99959]: pgmap v7453: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:26 smithi146 ceph-mon[103278]: pgmap v7454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[91136]: pgmap v7454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[99959]: pgmap v7454: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:28 smithi146 ceph-mon[103278]: pgmap v7455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:28 smithi078 ceph-mon[91136]: pgmap v7455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:28 smithi078 ceph-mon[99959]: pgmap v7455: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:30 smithi146 ceph-mon[103278]: pgmap v7456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:30 smithi078 ceph-mon[91136]: pgmap v7456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:30 smithi078 ceph-mon[99959]: pgmap v7456: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:28:30] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:28:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:31.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:31.180Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:31.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:32 smithi146 ceph-mon[103278]: pgmap v7457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:32 smithi078 ceph-mon[99959]: pgmap v7457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:32 smithi078 ceph-mon[91136]: pgmap v7457: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:34 smithi146 ceph-mon[103278]: pgmap v7458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:34 smithi078 ceph-mon[99959]: pgmap v7458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:34 smithi078 ceph-mon[91136]: pgmap v7458: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:37.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:36 smithi078 ceph-mon[99959]: pgmap v7459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:37.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:36 smithi078 ceph-mon[91136]: pgmap v7459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:36 smithi146 ceph-mon[103278]: pgmap v7459: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:38 smithi078 ceph-mon[91136]: pgmap v7460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:38 smithi078 ceph-mon[99959]: pgmap v7460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:39.242 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:38 smithi146 ceph-mon[103278]: pgmap v7460: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:40.958 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:28:40] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:28:41.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:40 smithi146 ceph-mon[103278]: pgmap v7461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:40 smithi078 ceph-mon[91136]: pgmap v7461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:40 smithi078 ceph-mon[99959]: pgmap v7461: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:41.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:41.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:41.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:42 smithi146 ceph-mon[103278]: pgmap v7462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[99959]: pgmap v7462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:43.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:42 smithi078 ceph-mon[91136]: pgmap v7462: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:44 smithi146 ceph-mon[103278]: pgmap v7463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:45.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:44 smithi078 ceph-mon[91136]: pgmap v7463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:44 smithi078 ceph-mon[99959]: pgmap v7463: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:28:46.670 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 4h - - 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 4h - - 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 4s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 0s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 4s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 0s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 4s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 4s ago 4h 429M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 0s ago 4h 280M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 4s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:28:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 4s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 0s ago 4h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 4s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 4s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 4s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 4s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 0s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (3h) 0s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 0s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:28:46.672 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 0s ago 4h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:28:46.673 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 0s ago 4h 71.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:28:46.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 4s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:28:46.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:28:46.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 4s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:28:46.673 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:28:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:28:46.979 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:28:47.246 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:28:47.246 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:28:47.246 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:28:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:28:47.248 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[99959]: pgmap v7464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[91136]: pgmap v7464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:46 smithi146 ceph-mon[103278]: pgmap v7464: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:28:47.556 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:28:47.556 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:47.556 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:28:47.556 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:47.556 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:28:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:47 smithi146 ceph-mon[103278]: from='client.80882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:47 smithi146 ceph-mon[103278]: from='client.80888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:47 smithi146 ceph-mon[103278]: from='client.80894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1828161365' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2596180719' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[91136]: from='client.80882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[91136]: from='client.80888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[91136]: from='client.80894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1828161365' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2596180719' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[99959]: from='client.80882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[99959]: from='client.80888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[99959]: from='client.80894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1828161365' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:28:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2596180719' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:28:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:48 smithi146 ceph-mon[103278]: from='client.80906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:48 smithi146 ceph-mon[103278]: pgmap v7465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:48 smithi078 ceph-mon[91136]: from='client.80906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:48 smithi078 ceph-mon[91136]: pgmap v7465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:48 smithi078 ceph-mon[99959]: from='client.80906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:28:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:48 smithi078 ceph-mon[99959]: pgmap v7465: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:50.960 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:28:50] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:28:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:50 smithi146 ceph-mon[103278]: pgmap v7466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:51.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:50 smithi078 ceph-mon[91136]: pgmap v7466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:50 smithi078 ceph-mon[99959]: pgmap v7466: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:51.180Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:51.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:28:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:28:51.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:28:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:52 smithi146 ceph-mon[103278]: pgmap v7467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:52 smithi078 ceph-mon[91136]: pgmap v7467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:28:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:52 smithi078 ceph-mon[99959]: pgmap v7467: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:54 smithi146 ceph-mon[103278]: pgmap v7468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:54 smithi078 ceph-mon[91136]: pgmap v7468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:54 smithi078 ceph-mon[99959]: pgmap v7468: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:57.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:56 smithi078 ceph-mon[99959]: pgmap v7469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:57.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:56 smithi078 ceph-mon[91136]: pgmap v7469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:56 smithi146 ceph-mon[103278]: pgmap v7469: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:28:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:58 smithi146 ceph-mon[103278]: pgmap v7470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:58 smithi078 ceph-mon[91136]: pgmap v7470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:58 smithi078 ceph-mon[99959]: pgmap v7470: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:28:59.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:00.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:29:00] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:29:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:00 smithi146 ceph-mon[103278]: pgmap v7471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:00 smithi078 ceph-mon[91136]: pgmap v7471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:00 smithi078 ceph-mon[99959]: pgmap v7471: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:01.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:01.181Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:01.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:01.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:02 smithi146 ceph-mon[103278]: pgmap v7472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[91136]: pgmap v7472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[99959]: pgmap v7472: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:04 smithi146 ceph-mon[103278]: pgmap v7473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:04 smithi078 ceph-mon[91136]: pgmap v7473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:04 smithi078 ceph-mon[99959]: pgmap v7473: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:06 smithi146 ceph-mon[103278]: pgmap v7474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[91136]: pgmap v7474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[99959]: pgmap v7474: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:08 smithi146 ceph-mon[103278]: pgmap v7475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:08 smithi078 ceph-mon[91136]: pgmap v7475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:08 smithi078 ceph-mon[99959]: pgmap v7475: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:10.979 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:29:10] "GET /metrics HTTP/1.1" 200 38991 "" "Prometheus/2.43.0" 2024-09-21T00:29:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:10 smithi146 ceph-mon[103278]: pgmap v7476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:10 smithi078 ceph-mon[91136]: pgmap v7476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:10 smithi078 ceph-mon[99959]: pgmap v7476: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:11.181Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:11.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:11.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:12 smithi146 ceph-mon[103278]: pgmap v7477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:12 smithi078 ceph-mon[91136]: pgmap v7477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:12 smithi078 ceph-mon[99959]: pgmap v7477: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:14.253 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:14 smithi146 ceph-mon[103278]: pgmap v7478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:14 smithi078 ceph-mon[91136]: pgmap v7478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:14 smithi078 ceph-mon[99959]: pgmap v7478: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:16 smithi146 ceph-mon[103278]: pgmap v7479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:16 smithi078 ceph-mon[91136]: pgmap v7479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:16 smithi078 ceph-mon[99959]: pgmap v7479: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:17.858 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 4h - - 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 4h - - 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 16s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 11s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 16s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 11s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 16s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 16s ago 4h 431M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 11s ago 4h 282M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:29:18.391 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 16s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 16s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 11s ago 4h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 16s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 16s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 16s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 16s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 11s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 11s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (3h) 11s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 11s ago 4h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 11s ago 4h 71.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 16s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:29:18.392 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:29:18.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 16s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:29:18.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:29:18.736 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:29:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:29:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:29:18.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:29:18.737 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:29:19.016 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:29:19.016 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:29:19.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:29:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:18 smithi146 ceph-mon[103278]: from='client.80918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:18 smithi146 ceph-mon[103278]: pgmap v7480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3827205051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:18 smithi078 ceph-mon[99959]: from='client.80918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:18 smithi078 ceph-mon[99959]: pgmap v7480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3827205051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:18 smithi078 ceph-mon[91136]: from='client.80918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:18 smithi078 ceph-mon[91136]: pgmap v7480: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3827205051' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:19.341 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:29:19.342 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:19.342 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:19.342 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:19.342 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:20 smithi146 ceph-mon[103278]: from='client.61005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:20 smithi146 ceph-mon[103278]: from='client.61011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:20 smithi146 ceph-mon[103278]: from='client.80942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3823155218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[91136]: from='client.61005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[91136]: from='client.61011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[91136]: from='client.80942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3823155218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[99959]: from='client.61005 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[99959]: from='client.61011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[99959]: from='client.80942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3823155218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:21.004 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:29:20] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:29:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:21 smithi146 ceph-mon[103278]: pgmap v7481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:21.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:21 smithi078 ceph-mon[91136]: pgmap v7481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:21 smithi078 ceph-mon[99959]: pgmap v7481: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:21.182Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:21.182Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:21.183Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:23 smithi146 ceph-mon[103278]: pgmap v7482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[91136]: pgmap v7482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[99959]: pgmap v7482: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:25 smithi146 ceph-mon[103278]: pgmap v7483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:25 smithi078 ceph-mon[91136]: pgmap v7483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:25 smithi078 ceph-mon[99959]: pgmap v7483: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[91136]: pgmap v7484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[99959]: pgmap v7484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:27 smithi146 ceph-mon[103278]: pgmap v7484: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:29 smithi078 ceph-mon[99959]: pgmap v7485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:29 smithi078 ceph-mon[91136]: pgmap v7485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:29 smithi146 ceph-mon[103278]: pgmap v7485: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:31.022 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:29:30] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:29:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:31 smithi078 ceph-mon[99959]: pgmap v7486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:31 smithi078 ceph-mon[91136]: pgmap v7486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:31.183Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:31.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:31.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:31 smithi146 ceph-mon[103278]: pgmap v7486: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:33 smithi078 ceph-mon[99959]: pgmap v7487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:33 smithi078 ceph-mon[91136]: pgmap v7487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:33 smithi146 ceph-mon[103278]: pgmap v7487: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:35 smithi078 ceph-mon[91136]: pgmap v7488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:35 smithi078 ceph-mon[99959]: pgmap v7488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:35 smithi146 ceph-mon[103278]: pgmap v7488: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:37.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:37 smithi078 ceph-mon[99959]: pgmap v7489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:37 smithi078 ceph-mon[91136]: pgmap v7489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:37 smithi146 ceph-mon[103278]: pgmap v7489: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:39 smithi078 ceph-mon[91136]: pgmap v7490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:39 smithi078 ceph-mon[99959]: pgmap v7490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:39 smithi146 ceph-mon[103278]: pgmap v7490: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:29:40] "GET /metrics HTTP/1.1" 200 38994 "" "Prometheus/2.43.0" 2024-09-21T00:29:41.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:41 smithi146 ceph-mon[103278]: pgmap v7491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:41 smithi078 ceph-mon[91136]: pgmap v7491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:41 smithi078 ceph-mon[99959]: pgmap v7491: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:41.184Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:41.184Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:41.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:43 smithi146 ceph-mon[103278]: pgmap v7492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[91136]: pgmap v7492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[99959]: pgmap v7492: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:44.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:45.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:45 smithi146 ceph-mon[103278]: pgmap v7493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:45 smithi078 ceph-mon[91136]: pgmap v7493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:45 smithi078 ceph-mon[99959]: pgmap v7493: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:47 smithi146 ceph-mon[103278]: pgmap v7494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[91136]: pgmap v7494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[99959]: pgmap v7494: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:29:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:49 smithi146 ceph-mon[103278]: pgmap v7495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:49 smithi078 ceph-mon[91136]: pgmap v7495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:49 smithi078 ceph-mon[99959]: pgmap v7495: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:49.638 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 4h - - 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 4h - - 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 7s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 3s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 7s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 3s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 7s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:29:50.166 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 7s ago 4h 432M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 3s ago 4h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 7s ago 4h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 7s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 3s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 7s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 7s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 7s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 7s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 3s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 3s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 3s ago 4h 85.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 3s ago 4h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 3s ago 4h 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 7s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:29:50.167 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:29:50.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 7s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:29:50.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:29:50.474 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:29:50.474 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:29:50.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:29:50.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:50.475 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:29:50.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:29:50.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:50.475 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:29:50.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:29:50.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:29:50.477 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:29:50.745 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:29:50.746 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:29:50.746 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:29:50] "GET /metrics HTTP/1.1" 200 38992 "" "Prometheus/2.43.0" 2024-09-21T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:51.060 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:29:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:51 smithi146 ceph-mon[103278]: from='client.80954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:51 smithi146 ceph-mon[103278]: from='client.61041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:51 smithi146 ceph-mon[103278]: pgmap v7496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3179458670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[91136]: from='client.80954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[91136]: from='client.61041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[91136]: pgmap v7496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3179458670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[99959]: from='client.80954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[99959]: from='client.61041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[99959]: pgmap v7496: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3179458670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:29:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:51.185Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:51.185Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:29:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:29:51.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:29:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:52 smithi078 ceph-mon[91136]: from='client.80966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:52 smithi078 ceph-mon[91136]: from='client.61059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1024557307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:52 smithi078 ceph-mon[99959]: from='client.80966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:52 smithi078 ceph-mon[99959]: from='client.61059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1024557307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:52 smithi146 ceph-mon[103278]: from='client.80966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:52 smithi146 ceph-mon[103278]: from='client.61059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:29:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1024557307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:29:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:53 smithi146 ceph-mon[103278]: pgmap v7497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:53 smithi078 ceph-mon[91136]: pgmap v7497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:29:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:53 smithi078 ceph-mon[99959]: pgmap v7497: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:54 smithi146 ceph-mon[103278]: pgmap v7498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:54 smithi078 ceph-mon[91136]: pgmap v7498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:54 smithi078 ceph-mon[99959]: pgmap v7498: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:56 smithi146 ceph-mon[103278]: pgmap v7499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:56 smithi078 ceph-mon[91136]: pgmap v7499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:56 smithi078 ceph-mon[99959]: pgmap v7499: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:29:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:58 smithi146 ceph-mon[103278]: pgmap v7500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:58 smithi078 ceph-mon[91136]: pgmap v7500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:29:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:58 smithi078 ceph-mon[99959]: pgmap v7500: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:29:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T00:29:59.998+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:30:00] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:30:01.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:00 smithi078 ceph-mon[91136]: pgmap v7501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:01.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.089 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:00 smithi078 ceph-mon[99959]: pgmap v7501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:01.089 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:00 smithi146 ceph-mon[103278]: pgmap v7501: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:01.354 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:01.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:01.354 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:01.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:01.354 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:01.186Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:03.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:03.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:02 smithi146 ceph-mon[103278]: pgmap v7502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:03.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[91136]: pgmap v7502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[99959]: pgmap v7502: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:04 smithi146 ceph-mon[103278]: pgmap v7503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:04 smithi078 ceph-mon[99959]: pgmap v7503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:04 smithi078 ceph-mon[91136]: pgmap v7503: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:06 smithi146 ceph-mon[103278]: pgmap v7504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:06 smithi078 ceph-mon[91136]: pgmap v7504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:06 smithi078 ceph-mon[99959]: pgmap v7504: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:09 smithi078 ceph-mon[91136]: pgmap v7505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:09 smithi078 ceph-mon[99959]: pgmap v7505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:09 smithi146 ceph-mon[103278]: pgmap v7505: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:11.030 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:30:10] "GET /metrics HTTP/1.1" 200 38993 "" "Prometheus/2.43.0" 2024-09-21T00:30:11.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:11 smithi078 ceph-mon[91136]: pgmap v7506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:11 smithi078 ceph-mon[99959]: pgmap v7506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:11.186Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:11.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:11.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:11 smithi146 ceph-mon[103278]: pgmap v7506: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:13 smithi078 ceph-mon[91136]: pgmap v7507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:13 smithi078 ceph-mon[99959]: pgmap v7507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:13.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:13.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:13 smithi146 ceph-mon[103278]: pgmap v7507: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:15 smithi146 ceph-mon[103278]: pgmap v7508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:15 smithi078 ceph-mon[91136]: pgmap v7508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:15 smithi078 ceph-mon[99959]: pgmap v7508: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:17 smithi146 ceph-mon[103278]: pgmap v7509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:17 smithi078 ceph-mon[91136]: pgmap v7509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:17 smithi078 ceph-mon[99959]: pgmap v7509: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:19.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:19 smithi078 ceph-mon[91136]: pgmap v7510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:19.395 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:19 smithi078 ceph-mon[99959]: pgmap v7510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:19 smithi146 ceph-mon[103278]: pgmap v7510: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:20.705 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:30:20] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:30:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:21 smithi078 ceph-mon[99959]: pgmap v7511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:21 smithi078 ceph-mon[91136]: pgmap v7511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:21.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:21.187Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:21.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:21.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:21.359 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:30:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:21 smithi146 ceph-mon[103278]: pgmap v7511: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:21.935 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:30:21.935 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 4h - - 2024-09-21T00:30:21.935 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 4h - - 2024-09-21T00:30:21.935 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 19s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 14s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 19s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 14s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 19s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 19s ago 4h 433M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 14s ago 4h 283M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 19s ago 4h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 19s ago 4h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 14s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 19s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 19s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 19s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 19s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 14s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:30:21.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 14s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 14s ago 4h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (3h) 14s ago 4h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 14s ago 4h 74.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (3h) 19s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 19s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:30:21.937 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:30:22.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:30:22.286 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:30:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:30:22.571 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:30:22.880 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:22.880 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:22.880 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:22.880 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:22.880 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:23.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='client.61071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='client.80996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:23.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='client.61083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: pgmap v7512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:23.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1529659047' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:23.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/448304419' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='client.61071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='client.80996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='client.61083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: pgmap v7512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1529659047' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/448304419' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='client.61071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='client.80996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='client.61083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: pgmap v7512: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1529659047' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/448304419' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:24.353 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:24 smithi146 ceph-mon[103278]: from='client.61095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:24 smithi078 ceph-mon[91136]: from='client.61095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:24 smithi078 ceph-mon[99959]: from='client.61095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:25 smithi146 ceph-mon[103278]: pgmap v7513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:25 smithi078 ceph-mon[91136]: pgmap v7513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:25 smithi078 ceph-mon[99959]: pgmap v7513: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:27 smithi146 ceph-mon[103278]: pgmap v7514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:27 smithi078 ceph-mon[91136]: pgmap v7514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:27 smithi078 ceph-mon[99959]: pgmap v7514: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:28 smithi146 ceph-mon[103278]: pgmap v7515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[91136]: pgmap v7515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:28 smithi078 ceph-mon[99959]: pgmap v7515: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:30 smithi146 ceph-mon[103278]: pgmap v7516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:30 smithi078 ceph-mon[91136]: pgmap v7516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:30 smithi078 ceph-mon[99959]: pgmap v7516: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:30:30] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:30:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:31.187Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:31.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:31.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:32 smithi146 ceph-mon[103278]: pgmap v7517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:32 smithi078 ceph-mon[99959]: pgmap v7517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:32 smithi078 ceph-mon[91136]: pgmap v7517: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:34 smithi146 ceph-mon[103278]: pgmap v7518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:34 smithi078 ceph-mon[91136]: pgmap v7518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:34 smithi078 ceph-mon[99959]: pgmap v7518: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:36 smithi146 ceph-mon[103278]: pgmap v7519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:36 smithi078 ceph-mon[91136]: pgmap v7519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:36 smithi078 ceph-mon[99959]: pgmap v7519: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:38 smithi146 ceph-mon[103278]: pgmap v7520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:38 smithi078 ceph-mon[99959]: pgmap v7520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:38 smithi078 ceph-mon[91136]: pgmap v7520: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:41.001 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:30:40] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:30:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:41 smithi146 ceph-mon[103278]: pgmap v7521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:41 smithi078 ceph-mon[91136]: pgmap v7521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:41 smithi078 ceph-mon[99959]: pgmap v7521: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:41.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:41.188Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:41.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:43.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:43.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:43 smithi146 ceph-mon[103278]: pgmap v7522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:43.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[91136]: pgmap v7522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[99959]: pgmap v7522: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:45 smithi146 ceph-mon[103278]: pgmap v7523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:45 smithi078 ceph-mon[99959]: pgmap v7523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:45 smithi078 ceph-mon[91136]: pgmap v7523: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:47 smithi078 ceph-mon[99959]: pgmap v7524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:47 smithi078 ceph-mon[91136]: pgmap v7524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:47.292 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:47 smithi146 ceph-mon[103278]: pgmap v7524: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[91136]: pgmap v7525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:48 smithi078 ceph-mon[99959]: pgmap v7525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:30:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:48 smithi146 ceph-mon[103278]: pgmap v7525: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:30:50.993 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:30:50] "GET /metrics HTTP/1.1" 200 38995 "" "Prometheus/2.43.0" 2024-09-21T00:30:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:50 smithi146 ceph-mon[103278]: pgmap v7526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:51.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:50 smithi078 ceph-mon[91136]: pgmap v7526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:50 smithi078 ceph-mon[99959]: pgmap v7526: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:51.188Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:51.189Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:30:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:30:51.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:30:53.167 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:30:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:52 smithi146 ceph-mon[103278]: pgmap v7527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:53.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:52 smithi078 ceph-mon[91136]: pgmap v7527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:30:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:52 smithi078 ceph-mon[99959]: pgmap v7527: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 4h - - 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 4h - - 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 11s ago 4h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 6s ago 4h 87.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 11s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 6s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:30:53.691 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 11s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 11s ago 4h 433M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 6s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 11s ago 4h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 11s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 6s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 11s ago 4h 84.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 11s ago 4h 90.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 11s ago 4h 75.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 11s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 6s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 6s ago 4h 79.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 6s ago 4h 85.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 6s ago 4h 81.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 6s ago 4h 75.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:30:53.692 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 11s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:30:53.693 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (3h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:30:53.693 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (3h) 11s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:30:53.693 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (3h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:30:54.005 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:30:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:30:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:30:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:30:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:30:54.006 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:30:54.273 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:30:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:30:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:30:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:30:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:30:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:30:54.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:30:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:30:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:30:54.581 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:30:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:54 smithi146 ceph-mon[103278]: from='client.81026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:54 smithi146 ceph-mon[103278]: from='client.61113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:54 smithi146 ceph-mon[103278]: from='client.61119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:54 smithi146 ceph-mon[103278]: pgmap v7528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:30:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3704705029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2950725304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:55 smithi078 ceph-mon[99959]: from='client.81026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:55 smithi078 ceph-mon[99959]: from='client.61113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:55 smithi078 ceph-mon[99959]: from='client.61119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:55 smithi078 ceph-mon[99959]: pgmap v7528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3704705029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2950725304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:54 smithi078 ceph-mon[91136]: from='client.81026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:54 smithi078 ceph-mon[91136]: from='client.61113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:54 smithi078 ceph-mon[91136]: from='client.61119 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:54 smithi078 ceph-mon[91136]: pgmap v7528: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:30:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3704705029' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:30:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2950725304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:30:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:56 smithi146 ceph-mon[103278]: from='client.81050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:56 smithi078 ceph-mon[99959]: from='client.81050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:56 smithi078 ceph-mon[91136]: from='client.81050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:30:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:57 smithi146 ceph-mon[103278]: pgmap v7529: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:30:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:57 smithi078 ceph-mon[91136]: pgmap v7529: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:30:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:57 smithi078 ceph-mon[99959]: pgmap v7529: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:30:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:59 smithi146 ceph-mon[103278]: pgmap v7530: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T00:30:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:30:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:59 smithi078 ceph-mon[91136]: pgmap v7530: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T00:30:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:30:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:30:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:59 smithi078 ceph-mon[99959]: pgmap v7530: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-21T00:30:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:30:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:00.950 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:31:00] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:31:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:01 smithi146 ceph-mon[103278]: pgmap v7531: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 26 op/s 2024-09-21T00:31:01.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:01 smithi078 ceph-mon[91136]: pgmap v7531: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 26 op/s 2024-09-21T00:31:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:01 smithi078 ceph-mon[99959]: pgmap v7531: 161 pgs: 161 active+clean; 585 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 26 op/s 2024-09-21T00:31:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:01.189Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:01.190Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:01.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[99959]: pgmap v7532: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[91136]: pgmap v7532: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:31:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:03.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:03 smithi146 ceph-mon[103278]: pgmap v7532: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T00:31:03.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:03.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:05 smithi078 ceph-mon[91136]: pgmap v7533: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:31:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:05 smithi078 ceph-mon[99959]: pgmap v7533: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:31:05.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:05 smithi146 ceph-mon[103278]: pgmap v7533: 161 pgs: 161 active+clean; 585 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T00:31:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:07 smithi078 ceph-mon[99959]: pgmap v7534: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T00:31:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:07 smithi078 ceph-mon[91136]: pgmap v7534: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T00:31:07.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:07 smithi146 ceph-mon[103278]: pgmap v7534: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T00:31:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:08 smithi146 ceph-mon[103278]: pgmap v7535: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[91136]: pgmap v7535: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:08 smithi078 ceph-mon[99959]: pgmap v7535: 161 pgs: 161 active+clean; 585 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:31:10.999 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:31:10] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:31:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:10 smithi146 ceph-mon[103278]: pgmap v7536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T00:31:11.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:10 smithi078 ceph-mon[91136]: pgmap v7536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T00:31:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:10 smithi078 ceph-mon[99959]: pgmap v7536: 161 pgs: 161 active+clean; 585 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T00:31:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:11.190Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:11.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:11.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:13 smithi146 ceph-mon[103278]: pgmap v7537: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 187 op/s 2024-09-21T00:31:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:13 smithi078 ceph-mon[91136]: pgmap v7537: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 187 op/s 2024-09-21T00:31:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:13 smithi078 ceph-mon[99959]: pgmap v7537: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 187 op/s 2024-09-21T00:31:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:15 smithi146 ceph-mon[103278]: pgmap v7538: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-21T00:31:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:15 smithi078 ceph-mon[91136]: pgmap v7538: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-21T00:31:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:15 smithi078 ceph-mon[99959]: pgmap v7538: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-21T00:31:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:17 smithi146 ceph-mon[103278]: pgmap v7539: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-21T00:31:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:17 smithi078 ceph-mon[99959]: pgmap v7539: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-21T00:31:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:17 smithi078 ceph-mon[91136]: pgmap v7539: 161 pgs: 161 active+clean; 585 KiB data, 297 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-21T00:31:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:19 smithi146 ceph-mon[103278]: pgmap v7540: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T00:31:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:19 smithi078 ceph-mon[91136]: pgmap v7540: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T00:31:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:19 smithi078 ceph-mon[99959]: pgmap v7540: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T00:31:21.017 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:31:20] "GET /metrics HTTP/1.1" 200 38997 "" "Prometheus/2.43.0" 2024-09-21T00:31:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:21 smithi078 ceph-mon[91136]: pgmap v7541: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:31:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:21 smithi078 ceph-mon[99959]: pgmap v7541: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:31:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:21.190Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:21.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:21.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:21.191Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:21 smithi146 ceph-mon[103278]: pgmap v7541: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:31:23.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:23.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[99959]: pgmap v7542: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T00:31:23.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:23.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[91136]: pgmap v7542: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T00:31:23.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:23 smithi146 ceph-mon[103278]: pgmap v7542: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T00:31:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:24.878 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:31:25.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:25 smithi146 ceph-mon[103278]: pgmap v7543: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T00:31:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:25 smithi078 ceph-mon[99959]: pgmap v7543: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T00:31:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:25 smithi078 ceph-mon[91136]: pgmap v7543: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T00:31:25.407 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:31:25.407 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 4h - - 2024-09-21T00:31:25.407 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 4h - - 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 2s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 17s ago 4h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 2s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 17s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 2s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 2s ago 4h 435M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 17s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 2s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 2s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 17s ago 4h 15.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 2s ago 4h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 2s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 2s ago 4h 76.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 2s ago 4h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 17s ago 4h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:31:25.408 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 17s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 17s ago 4h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 17s ago 4h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 17s ago 4h 75.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 2s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 2s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:31:25.409 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:31:25.716 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:31:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:31:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:31:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:31:25.717 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:31:25.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:31:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:31:26.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:26 smithi146 ceph-mon[103278]: from='client.81062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:26.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4189909285' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:26 smithi078 ceph-mon[91136]: from='client.81062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4189909285' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:26 smithi078 ceph-mon[99959]: from='client.81062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4189909285' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:26.298 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:31:26.298 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:26.298 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:26.298 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:26.298 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[91136]: from='client.61149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[91136]: from='client.61155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[91136]: pgmap v7544: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[91136]: from='client.61167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4176719149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[99959]: from='client.61149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[99959]: from='client.61155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[99959]: pgmap v7544: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-21T00:31:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[99959]: from='client.61167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4176719149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:27 smithi146 ceph-mon[103278]: from='client.61149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:27 smithi146 ceph-mon[103278]: from='client.61155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:27 smithi146 ceph-mon[103278]: pgmap v7544: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-09-21T00:31:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:27 smithi146 ceph-mon[103278]: from='client.61167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4176719149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:28 smithi146 ceph-mon[103278]: pgmap v7545: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:31:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[91136]: pgmap v7545: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[99959]: pgmap v7545: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:31:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:31.007 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:31:30] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:31:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:31 smithi146 ceph-mon[103278]: pgmap v7546: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:31:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:31 smithi078 ceph-mon[91136]: pgmap v7546: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:31:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:31 smithi078 ceph-mon[99959]: pgmap v7546: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T00:31:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:31.191Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:31.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:31.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:31.192Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:33 smithi146 ceph-mon[103278]: pgmap v7547: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:31:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:33 smithi078 ceph-mon[91136]: pgmap v7547: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:31:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:33 smithi078 ceph-mon[99959]: pgmap v7547: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:31:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:35 smithi146 ceph-mon[103278]: pgmap v7548: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:35 smithi078 ceph-mon[91136]: pgmap v7548: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:35 smithi078 ceph-mon[99959]: pgmap v7548: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:37 smithi078 ceph-mon[91136]: pgmap v7549: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:37 smithi078 ceph-mon[99959]: pgmap v7549: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:37.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:37 smithi146 ceph-mon[103278]: pgmap v7549: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:39 smithi078 ceph-mon[91136]: pgmap v7550: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:39 smithi078 ceph-mon[99959]: pgmap v7550: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:39 smithi146 ceph-mon[103278]: pgmap v7550: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:41.022 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:31:40] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:31:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:41 smithi078 ceph-mon[91136]: pgmap v7551: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:41 smithi078 ceph-mon[99959]: pgmap v7551: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:41.192Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:41.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:41.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:41.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:41.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:41 smithi146 ceph-mon[103278]: pgmap v7551: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[91136]: pgmap v7552: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[99959]: pgmap v7552: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:43.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:43 smithi146 ceph-mon[103278]: pgmap v7552: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:43.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:43.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:44.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:44.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:44.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:45 smithi078 ceph-mon[99959]: pgmap v7553: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:45 smithi078 ceph-mon[91136]: pgmap v7553: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:45.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:45 smithi146 ceph-mon[103278]: pgmap v7553: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:47 smithi078 ceph-mon[91136]: pgmap v7554: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:47 smithi078 ceph-mon[99959]: pgmap v7554: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:47 smithi146 ceph-mon[103278]: pgmap v7554: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:48 smithi146 ceph-mon[103278]: pgmap v7555: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[99959]: pgmap v7555: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:31:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:48 smithi078 ceph-mon[91136]: pgmap v7555: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:51.014 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:31:50] "GET /metrics HTTP/1.1" 200 38996 "" "Prometheus/2.43.0" 2024-09-21T00:31:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:51 smithi146 ceph-mon[103278]: pgmap v7556: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:51 smithi078 ceph-mon[91136]: pgmap v7556: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:51 smithi078 ceph-mon[99959]: pgmap v7556: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:51.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:51.193Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:31:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:31:51.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:31:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:53 smithi078 ceph-mon[91136]: pgmap v7557: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:53 smithi078 ceph-mon[99959]: pgmap v7557: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:31:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:53 smithi146 ceph-mon[103278]: pgmap v7557: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:55 smithi078 ceph-mon[99959]: pgmap v7558: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:55 smithi078 ceph-mon[91136]: pgmap v7558: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:55 smithi146 ceph-mon[103278]: pgmap v7558: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:56.596 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 4h - - 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 4h - - 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 14s ago 4h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 9s ago 4h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 14s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:31:57.124 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 9s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 14s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 14s ago 4h 435M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 9s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 14s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 14s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 9s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 14s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 14s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 14s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 14s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 9s ago 4h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 9s ago 4h 79.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 9s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:31:57.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 9s ago 4h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:31:57.126 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 9s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:31:57.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 14s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:31:57.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:31:57.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 14s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:31:57.126 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:31:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:57 smithi078 ceph-mon[91136]: pgmap v7559: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:57 smithi078 ceph-mon[99959]: pgmap v7559: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:31:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:31:57.438 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:31:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:57 smithi146 ceph-mon[103278]: pgmap v7559: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:31:57.707 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:31:57.707 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:31:57.707 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:31:57.708 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:31:58.018 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:31:58.018 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:58.018 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:31:58.018 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:58.018 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:31:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[91136]: from='client.81098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[91136]: from='client.81104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3573257118' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1224511136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[99959]: from='client.81098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[99959]: from='client.81104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3573257118' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1224511136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:58 smithi146 ceph-mon[103278]: from='client.81098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:58 smithi146 ceph-mon[103278]: from='client.81104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3573257118' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:31:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1224511136' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:31:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:59 smithi146 ceph-mon[103278]: from='client.81110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:59 smithi146 ceph-mon[103278]: from='client.61203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:59 smithi146 ceph-mon[103278]: pgmap v7560: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[99959]: from='client.81110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[99959]: from='client.61203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[99959]: pgmap v7560: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[91136]: from='client.81110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[91136]: from='client.61203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[91136]: pgmap v7560: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:31:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:32:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:32:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:01 smithi146 ceph-mon[103278]: pgmap v7561: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:01 smithi078 ceph-mon[91136]: pgmap v7561: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:01 smithi078 ceph-mon[99959]: pgmap v7561: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:01.193Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:01.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:01.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:01.194Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:03.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:03 smithi078 ceph-mon[91136]: pgmap v7562: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:03.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:03.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:03 smithi078 ceph-mon[99959]: pgmap v7562: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:03.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:03 smithi146 ceph-mon[103278]: pgmap v7562: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[91136]: pgmap v7563: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:04 smithi078 ceph-mon[99959]: pgmap v7563: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:04 smithi146 ceph-mon[103278]: pgmap v7563: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:07 smithi078 ceph-mon[91136]: pgmap v7564: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:07 smithi078 ceph-mon[99959]: pgmap v7564: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:07 smithi146 ceph-mon[103278]: pgmap v7564: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:08 smithi146 ceph-mon[103278]: pgmap v7565: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[91136]: pgmap v7565: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:08 smithi078 ceph-mon[99959]: pgmap v7565: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:11.021 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:32:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T00:32:11.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:11 smithi078 ceph-mon[91136]: pgmap v7566: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:11 smithi078 ceph-mon[99959]: pgmap v7566: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:11.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:11.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:11.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:11.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:11 smithi146 ceph-mon[103278]: pgmap v7566: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:13 smithi078 ceph-mon[91136]: pgmap v7567: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:13 smithi078 ceph-mon[99959]: pgmap v7567: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:13.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:13 smithi146 ceph-mon[103278]: pgmap v7567: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:13.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:15 smithi078 ceph-mon[91136]: pgmap v7568: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:15 smithi078 ceph-mon[99959]: pgmap v7568: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:15 smithi146 ceph-mon[103278]: pgmap v7568: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:17 smithi078 ceph-mon[91136]: pgmap v7569: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:17 smithi078 ceph-mon[99959]: pgmap v7569: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:17 smithi146 ceph-mon[103278]: pgmap v7569: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:19 smithi146 ceph-mon[103278]: pgmap v7570: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:19 smithi078 ceph-mon[91136]: pgmap v7570: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:19 smithi078 ceph-mon[99959]: pgmap v7570: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:32:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:32:21.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:21 smithi078 ceph-mon[91136]: pgmap v7571: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:21.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:21 smithi078 ceph-mon[99959]: pgmap v7571: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:21.488 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:21.194Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:21.488 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:21.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:21.488 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:21.195Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:21 smithi146 ceph-mon[103278]: pgmap v7571: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:23.370 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:23 smithi146 ceph-mon[103278]: pgmap v7572: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:23.370 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:23 smithi078 ceph-mon[91136]: pgmap v7572: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:23 smithi078 ceph-mon[99959]: pgmap v7572: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[91136]: pgmap v7573: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:24 smithi078 ceph-mon[99959]: pgmap v7573: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:24 smithi146 ceph-mon[103278]: pgmap v7573: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:27 smithi078 ceph-mon[91136]: pgmap v7574: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:27 smithi078 ceph-mon[99959]: pgmap v7574: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:27.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:27 smithi146 ceph-mon[103278]: pgmap v7574: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:28.312 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 4h - - 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 4h - - 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 5s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 0s ago 4h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 5s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 0s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 5s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 5s ago 4h 435M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:32:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 0s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 5s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 5s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 0s ago 4h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 5s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 5s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 5s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 5s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 0s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 0s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 0s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 0s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 0s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:32:28.840 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 5s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:32:28.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:32:28.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 5s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:32:28.841 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:32:29.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:32:29.160 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:32:29.160 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.160 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[99959]: pgmap v7575: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[91136]: pgmap v7575: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:29.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:28 smithi146 ceph-mon[103278]: pgmap v7575: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:32:29.427 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:32:29.428 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:32:29.428 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:32:29.428 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:32:29.735 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:32:29.735 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:32:29.736 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:32:29.736 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:32:29.736 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[99959]: from='client.81134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[99959]: from='client.81140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[99959]: from='client.61227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3404571362' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3957581125' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[91136]: from='client.81134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[91136]: from='client.81140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[91136]: from='client.61227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3404571362' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3957581125' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:29 smithi146 ceph-mon[103278]: from='client.81134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:29 smithi146 ceph-mon[103278]: from='client.81140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:29 smithi146 ceph-mon[103278]: from='client.61227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3404571362' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:32:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3957581125' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:32:30.938 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:32:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:32:31.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:30 smithi078 ceph-mon[99959]: from='client.81158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:31.196 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:30 smithi078 ceph-mon[99959]: pgmap v7576: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:31.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:30 smithi078 ceph-mon[91136]: from='client.81158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:31.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:30 smithi078 ceph-mon[91136]: pgmap v7576: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:30 smithi146 ceph-mon[103278]: from='client.81158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:32:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:30 smithi146 ceph-mon[103278]: pgmap v7576: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:31.196Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:31.546 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:31.196Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:31.546 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:31.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:33 smithi078 ceph-mon[99959]: pgmap v7577: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:33 smithi078 ceph-mon[91136]: pgmap v7577: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:33 smithi146 ceph-mon[103278]: pgmap v7577: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:35 smithi078 ceph-mon[99959]: pgmap v7578: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:35 smithi078 ceph-mon[91136]: pgmap v7578: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:35 smithi146 ceph-mon[103278]: pgmap v7578: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:37 smithi146 ceph-mon[103278]: pgmap v7579: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:37 smithi078 ceph-mon[91136]: pgmap v7579: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:37 smithi078 ceph-mon[99959]: pgmap v7579: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:39 smithi146 ceph-mon[103278]: pgmap v7580: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:39 smithi078 ceph-mon[91136]: pgmap v7580: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:39 smithi078 ceph-mon[99959]: pgmap v7580: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:32:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:32:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:41 smithi146 ceph-mon[103278]: pgmap v7581: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:41 smithi078 ceph-mon[91136]: pgmap v7581: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:41 smithi078 ceph-mon[99959]: pgmap v7581: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:41.196Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:41.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:41.197Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:43.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:43 smithi146 ceph-mon[103278]: pgmap v7582: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:43.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:43.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:43 smithi078 ceph-mon[99959]: pgmap v7582: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:43.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:43.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:43 smithi078 ceph-mon[91136]: pgmap v7582: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:43.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:44.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:44.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.622 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[99959]: pgmap v7583: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:44.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:44.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:44 smithi078 ceph-mon[91136]: pgmap v7583: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:44 smithi146 ceph-mon[103278]: pgmap v7583: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:47.163 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:47 smithi146 ceph-mon[103278]: pgmap v7584: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:47 smithi078 ceph-mon[91136]: pgmap v7584: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:47 smithi078 ceph-mon[99959]: pgmap v7584: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:49 smithi146 ceph-mon[103278]: pgmap v7585: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[91136]: pgmap v7585: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[99959]: pgmap v7585: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:32:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:32:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:32:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:51 smithi146 ceph-mon[103278]: pgmap v7586: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:51 smithi078 ceph-mon[99959]: pgmap v7586: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:51 smithi078 ceph-mon[91136]: pgmap v7586: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:51.197Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:51.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:32:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:32:51.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:32:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:53 smithi146 ceph-mon[103278]: pgmap v7587: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:53 smithi078 ceph-mon[91136]: pgmap v7587: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:53 smithi078 ceph-mon[99959]: pgmap v7587: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:32:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:55 smithi146 ceph-mon[103278]: pgmap v7588: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:55 smithi078 ceph-mon[99959]: pgmap v7588: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:55 smithi078 ceph-mon[91136]: pgmap v7588: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:57 smithi146 ceph-mon[103278]: pgmap v7589: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:57 smithi078 ceph-mon[99959]: pgmap v7589: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:57 smithi078 ceph-mon[91136]: pgmap v7589: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:32:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:59 smithi146 ceph-mon[103278]: pgmap v7590: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:59 smithi078 ceph-mon[99959]: pgmap v7590: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:32:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:59 smithi078 ceph-mon[91136]: pgmap v7590: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:32:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:00.026 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:33:00.563 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 4h - - 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 4h - - 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 17s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 12s ago 4h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 17s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 12s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 17s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 17s ago 4h 436M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 12s ago 4h 284M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 17s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 17s ago 4h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 12s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:33:00.564 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 17s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 17s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 17s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 17s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 12s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 12s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 12s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 12s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 12s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 17s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 17s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:33:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:33:00.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:33:00.875 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:33:00.875 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:33:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:33:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:33:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:33:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:33:01.143 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:33:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-mon[99959]: pgmap v7591: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-mon[99959]: from='client.61251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:01.143 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1187547517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:01.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-mon[91136]: pgmap v7591: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:01.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-mon[91136]: from='client.61251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:01.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1187547517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:01.451 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:33:01.451 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:01.451 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:01.451 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:01.451 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:01.198Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:01.451 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:01.198Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:01.451 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:01.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:01 smithi146 ceph-mon[103278]: pgmap v7591: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:01 smithi146 ceph-mon[103278]: from='client.61251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1187547517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:02 smithi078 ceph-mon[91136]: from='client.61257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:02 smithi078 ceph-mon[91136]: from='client.61263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/314250555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:02 smithi078 ceph-mon[99959]: from='client.61257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:02 smithi078 ceph-mon[99959]: from='client.61263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/314250555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:02 smithi146 ceph-mon[103278]: from='client.61257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:02 smithi146 ceph-mon[103278]: from='client.61263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/314250555' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:03 smithi146 ceph-mon[103278]: from='client.61275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:03 smithi146 ceph-mon[103278]: pgmap v7592: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:03 smithi078 ceph-mon[99959]: from='client.61275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:03 smithi078 ceph-mon[99959]: pgmap v7592: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:03 smithi078 ceph-mon[91136]: from='client.61275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:03 smithi078 ceph-mon[91136]: pgmap v7592: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:04.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[99959]: pgmap v7593: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:04.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:04 smithi078 ceph-mon[91136]: pgmap v7593: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:04 smithi146 ceph-mon[103278]: pgmap v7593: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:07.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:07 smithi078 ceph-mon[99959]: pgmap v7594: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:07.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:07 smithi078 ceph-mon[91136]: pgmap v7594: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:07 smithi146 ceph-mon[103278]: pgmap v7594: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:09 smithi146 ceph-mon[103278]: pgmap v7595: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[91136]: pgmap v7595: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[99959]: pgmap v7595: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:33:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:33:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:11 smithi146 ceph-mon[103278]: pgmap v7596: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:11 smithi078 ceph-mon[99959]: pgmap v7596: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:11 smithi078 ceph-mon[91136]: pgmap v7596: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:11.199Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:11.199Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:11.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:13.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:13 smithi146 ceph-mon[103278]: pgmap v7597: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:13.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:13 smithi078 ceph-mon[99959]: pgmap v7597: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:13 smithi078 ceph-mon[91136]: pgmap v7597: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:14.334 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:15.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:15 smithi146 ceph-mon[103278]: pgmap v7598: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:15 smithi078 ceph-mon[99959]: pgmap v7598: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:15 smithi078 ceph-mon[91136]: pgmap v7598: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:17.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:17 smithi146 ceph-mon[103278]: pgmap v7599: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:17 smithi078 ceph-mon[99959]: pgmap v7599: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:17 smithi078 ceph-mon[91136]: pgmap v7599: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:19 smithi146 ceph-mon[103278]: pgmap v7600: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:19 smithi078 ceph-mon[91136]: pgmap v7600: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:19 smithi078 ceph-mon[99959]: pgmap v7600: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:33:20] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T00:33:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:21 smithi146 ceph-mon[103278]: pgmap v7601: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:21 smithi078 ceph-mon[91136]: pgmap v7601: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:21 smithi078 ceph-mon[99959]: pgmap v7601: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:21.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:21.200Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:21.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:23 smithi146 ceph-mon[103278]: pgmap v7602: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:23.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:23 smithi078 ceph-mon[91136]: pgmap v7602: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:23 smithi078 ceph-mon[99959]: pgmap v7602: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:24 smithi146 ceph-mon[103278]: pgmap v7603: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[99959]: pgmap v7603: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:24 smithi078 ceph-mon[91136]: pgmap v7603: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:27.243 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:27 smithi146 ceph-mon[103278]: pgmap v7604: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:27.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:27 smithi078 ceph-mon[99959]: pgmap v7604: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:27.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:27 smithi078 ceph-mon[91136]: pgmap v7604: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:29.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:29 smithi146 ceph-mon[103278]: pgmap v7605: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:29.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[91136]: pgmap v7605: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[99959]: pgmap v7605: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:33:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:33:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:31 smithi146 ceph-mon[103278]: pgmap v7606: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:31 smithi078 ceph-mon[99959]: pgmap v7606: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:31 smithi078 ceph-mon[91136]: pgmap v7606: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:31.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:31.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:31.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:31.744 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 4h - - 2024-09-21T00:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 4h - - 2024-09-21T00:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 8s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 3s ago 4h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 8s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 3s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 8s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 8s ago 4h 436M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 3s ago 4h 285M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 8s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 8s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 3s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 8s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 8s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 8s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 8s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 3s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 3s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 3s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 3s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 3s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 8s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 8s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:33:32.581 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:33:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:33:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:33:32.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:33:32.582 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:33:32.850 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:33:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:33:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:33:32.850 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:33:32.851 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:33:33.162 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:33:33.162 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:33.162 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:33:33.162 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:33.162 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[91136]: from='client.81206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[91136]: pgmap v7607: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[91136]: from='client.81212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2860924232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[99959]: from='client.81206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[99959]: pgmap v7607: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:33.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[99959]: from='client.81212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:33.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:33.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2860924232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:33 smithi146 ceph-mon[103278]: from='client.81206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:33 smithi146 ceph-mon[103278]: pgmap v7607: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:33 smithi146 ceph-mon[103278]: from='client.81212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2860924232' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:33:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:34 smithi146 ceph-mon[103278]: from='client.81218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:34 smithi146 ceph-mon[103278]: from='client.81230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3932071897' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:34 smithi078 ceph-mon[99959]: from='client.81218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:34 smithi078 ceph-mon[99959]: from='client.81230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3932071897' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:34 smithi078 ceph-mon[91136]: from='client.81218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:34 smithi078 ceph-mon[91136]: from='client.81230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:33:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3932071897' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:33:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:35 smithi146 ceph-mon[103278]: pgmap v7608: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:35 smithi078 ceph-mon[99959]: pgmap v7608: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:35 smithi078 ceph-mon[91136]: pgmap v7608: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:37 smithi146 ceph-mon[103278]: pgmap v7609: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:37 smithi078 ceph-mon[99959]: pgmap v7609: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:37 smithi078 ceph-mon[91136]: pgmap v7609: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:39 smithi146 ceph-mon[103278]: pgmap v7610: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:39 smithi078 ceph-mon[91136]: pgmap v7610: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:39 smithi078 ceph-mon[99959]: pgmap v7610: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:33:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:33:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:41 smithi146 ceph-mon[103278]: pgmap v7611: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:41.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:41 smithi078 ceph-mon[91136]: pgmap v7611: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:41 smithi078 ceph-mon[99959]: pgmap v7611: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:41.200Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:41.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:41.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:43.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:43 smithi146 ceph-mon[103278]: pgmap v7612: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:43.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:43.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:43 smithi078 ceph-mon[91136]: pgmap v7612: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:43 smithi078 ceph-mon[99959]: pgmap v7612: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:44.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:44.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:45 smithi146 ceph-mon[103278]: pgmap v7613: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:45 smithi078 ceph-mon[91136]: pgmap v7613: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:45 smithi078 ceph-mon[99959]: pgmap v7613: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:47 smithi078 ceph-mon[91136]: pgmap v7614: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:47 smithi078 ceph-mon[99959]: pgmap v7614: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:47 smithi146 ceph-mon[103278]: pgmap v7614: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:49 smithi146 ceph-mon[103278]: pgmap v7615: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[91136]: pgmap v7615: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[99959]: pgmap v7615: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:33:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:33:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:33:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:51 smithi146 ceph-mon[103278]: pgmap v7616: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:51.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:51 smithi078 ceph-mon[91136]: pgmap v7616: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:51 smithi078 ceph-mon[99959]: pgmap v7616: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:51.201Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:51.201Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:33:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:33:51.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:33:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:53 smithi146 ceph-mon[103278]: pgmap v7617: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:53.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:53 smithi078 ceph-mon[91136]: pgmap v7617: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:53 smithi078 ceph-mon[99959]: pgmap v7617: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:33:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:55 smithi146 ceph-mon[103278]: pgmap v7618: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:55 smithi078 ceph-mon[91136]: pgmap v7618: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:55 smithi078 ceph-mon[99959]: pgmap v7618: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:57 smithi146 ceph-mon[103278]: pgmap v7619: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:57 smithi078 ceph-mon[91136]: pgmap v7619: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:57 smithi078 ceph-mon[99959]: pgmap v7619: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:33:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:59 smithi078 ceph-mon[99959]: pgmap v7620: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:59 smithi078 ceph-mon[91136]: pgmap v7620: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:33:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:59 smithi146 ceph-mon[103278]: pgmap v7620: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:33:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:00 smithi146 ceph-mon[103278]: pgmap v7621: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:00 smithi078 ceph-mon[91136]: pgmap v7621: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:00 smithi078 ceph-mon[99959]: pgmap v7621: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:34:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:34:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:01.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:01.202Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:01.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:03.461 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:03 smithi078 ceph-mon[99959]: pgmap v7622: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:03.461 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:03.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:03 smithi078 ceph-mon[91136]: pgmap v7622: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:03.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:03.462 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:34:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:03 smithi146 ceph-mon[103278]: pgmap v7622: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:03.995 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 4h - - 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 4h - - 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 0s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 15s ago 4h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 0s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 15s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 0s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 0s ago 4h 437M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 15s ago 4h 286M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 0s ago 4h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 0s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 15s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 0s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 0s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:34:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 0s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 0s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 15s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 15s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 15s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 15s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 15s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 0s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 0s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:34:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:34:04.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:34:04.309 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:34:04.577 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:34:04.578 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:34:04.886 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:34:04.886 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:34:04.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:34:04.886 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:04.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='client.61323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='client.61329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='client.81254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: pgmap v7623: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4242867100' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2652707815' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='client.61323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='client.61329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='client.81254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: pgmap v7623: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4242867100' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2652707815' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='client.61323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='client.61329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='client.81254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: pgmap v7623: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4242867100' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2652707815' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:05 smithi146 ceph-mon[103278]: from='client.81266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:05 smithi078 ceph-mon[91136]: from='client.81266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:05 smithi078 ceph-mon[99959]: from='client.81266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:07.232 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:06 smithi078 ceph-mon[99959]: pgmap v7624: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:07.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:06 smithi078 ceph-mon[91136]: pgmap v7624: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:06 smithi146 ceph-mon[103278]: pgmap v7624: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:34:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[91136]: pgmap v7625: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[99959]: pgmap v7625: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:09 smithi146 ceph-mon[103278]: pgmap v7625: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:09.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:09.369 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:34:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:34:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:11 smithi146 ceph-mon[103278]: pgmap v7626: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:11 smithi078 ceph-mon[99959]: pgmap v7626: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:11 smithi078 ceph-mon[91136]: pgmap v7626: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:11.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:11.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:11.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:13.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:13 smithi146 ceph-mon[103278]: pgmap v7627: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:13.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:13 smithi078 ceph-mon[99959]: pgmap v7627: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:13 smithi078 ceph-mon[91136]: pgmap v7627: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:15 smithi146 ceph-mon[103278]: pgmap v7628: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:15 smithi078 ceph-mon[99959]: pgmap v7628: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:15 smithi078 ceph-mon[91136]: pgmap v7628: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:17 smithi146 ceph-mon[103278]: pgmap v7629: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:17 smithi078 ceph-mon[99959]: pgmap v7629: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:17 smithi078 ceph-mon[91136]: pgmap v7629: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:19 smithi146 ceph-mon[103278]: pgmap v7630: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:19 smithi078 ceph-mon[91136]: pgmap v7630: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:19 smithi078 ceph-mon[99959]: pgmap v7630: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:34:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:34:21.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:21 smithi078 ceph-mon[99959]: pgmap v7631: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:21.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:21 smithi078 ceph-mon[91136]: pgmap v7631: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:21.202Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:21.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:21.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:21.203Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:21 smithi146 ceph-mon[103278]: pgmap v7631: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:23 smithi146 ceph-mon[103278]: pgmap v7632: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:23 smithi078 ceph-mon[91136]: pgmap v7632: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:23 smithi078 ceph-mon[99959]: pgmap v7632: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:25 smithi146 ceph-mon[103278]: pgmap v7633: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[99959]: pgmap v7633: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[91136]: pgmap v7633: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 00:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T00:34:25.84766831Z level=info msg="Completed cleanup jobs" duration=11.843733ms 2024-09-21T00:34:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:27 smithi146 ceph-mon[103278]: pgmap v7634: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:27 smithi078 ceph-mon[91136]: pgmap v7634: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:27 smithi078 ceph-mon[99959]: pgmap v7634: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:29.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[91136]: pgmap v7635: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[99959]: pgmap v7635: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:29 smithi146 ceph-mon[103278]: pgmap v7635: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:34:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:34:31.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:31 smithi146 ceph-mon[103278]: pgmap v7636: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:31 smithi078 ceph-mon[91136]: pgmap v7636: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:31 smithi078 ceph-mon[99959]: pgmap v7636: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:31.204Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:31.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:31.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:33 smithi146 ceph-mon[103278]: pgmap v7637: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:33 smithi078 ceph-mon[91136]: pgmap v7637: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:33 smithi078 ceph-mon[99959]: pgmap v7637: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:35.185 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:34:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:35 smithi146 ceph-mon[103278]: pgmap v7638: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:35 smithi078 ceph-mon[99959]: pgmap v7638: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:35 smithi078 ceph-mon[91136]: pgmap v7638: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:34:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 4h - - 2024-09-21T00:34:35.707 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 4h - - 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 11s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 7s ago 4h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 11s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 7s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 11s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 11s ago 4h 438M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 7s ago 4h 287M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 11s ago 4h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 11s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 7s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 11s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 11s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 11s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:34:35.708 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 11s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 7s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 7s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 7s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 7s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 7s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 11s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 11s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:34:35.709 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:34:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:34:36.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:34:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3960563441' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3960563441' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:34:36.290 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:34:36.291 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:34:36.291 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:34:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3960563441' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:34:36.603 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:34:36.603 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:34:36.603 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:34:36.603 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:36.603 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:34:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:37 smithi146 ceph-mon[103278]: from='client.81278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:37 smithi146 ceph-mon[103278]: from='client.81284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:37 smithi146 ceph-mon[103278]: from='client.81290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:37 smithi146 ceph-mon[103278]: pgmap v7639: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3742331291' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[91136]: from='client.81278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[91136]: from='client.81284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[91136]: from='client.81290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[91136]: pgmap v7639: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3742331291' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[99959]: from='client.81278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[99959]: from='client.81284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[99959]: from='client.81290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[99959]: pgmap v7639: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3742331291' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:34:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:38 smithi146 ceph-mon[103278]: from='client.61383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:38 smithi078 ceph-mon[99959]: from='client.61383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:38 smithi078 ceph-mon[91136]: from='client.61383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:34:39.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:39 smithi078 ceph-mon[99959]: pgmap v7640: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:39.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:39 smithi078 ceph-mon[91136]: pgmap v7640: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:39 smithi146 ceph-mon[103278]: pgmap v7640: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:34:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:34:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:41 smithi146 ceph-mon[103278]: pgmap v7641: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:41 smithi078 ceph-mon[99959]: pgmap v7641: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:41 smithi078 ceph-mon[91136]: pgmap v7641: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:41.204Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:41.204Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:41.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:43.344 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:43 smithi146 ceph-mon[103278]: pgmap v7642: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:43 smithi078 ceph-mon[99959]: pgmap v7642: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:43 smithi078 ceph-mon[91136]: pgmap v7642: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:45 smithi146 ceph-mon[103278]: pgmap v7643: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[91136]: pgmap v7643: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[99959]: pgmap v7643: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:47 smithi146 ceph-mon[103278]: pgmap v7644: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:47 smithi078 ceph-mon[91136]: pgmap v7644: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:47 smithi078 ceph-mon[99959]: pgmap v7644: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:49.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:49 smithi146 ceph-mon[103278]: pgmap v7645: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:49.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[99959]: pgmap v7645: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[91136]: pgmap v7645: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:34:50.780 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:34:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:34:51.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:51 smithi146 ceph-mon[103278]: pgmap v7646: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:51 smithi078 ceph-mon[91136]: pgmap v7646: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:51 smithi078 ceph-mon[99959]: pgmap v7646: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:51.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:51.205Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:34:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:34:51.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:34:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:52 smithi146 ceph-mon[103278]: pgmap v7647: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:52 smithi078 ceph-mon[91136]: pgmap v7647: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:52 smithi078 ceph-mon[99959]: pgmap v7647: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:34:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:54 smithi146 ceph-mon[103278]: pgmap v7648: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:54 smithi078 ceph-mon[91136]: pgmap v7648: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:54 smithi078 ceph-mon[99959]: pgmap v7648: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:57 smithi146 ceph-mon[103278]: pgmap v7649: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:57 smithi078 ceph-mon[91136]: pgmap v7649: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:57 smithi078 ceph-mon[99959]: pgmap v7649: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:34:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:59 smithi146 ceph-mon[103278]: pgmap v7650: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:59 smithi078 ceph-mon[91136]: pgmap v7650: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:34:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:59 smithi078 ceph-mon[99959]: pgmap v7650: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:34:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:35:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:35:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:01 smithi146 ceph-mon[103278]: pgmap v7651: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:01.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:01 smithi078 ceph-mon[99959]: pgmap v7651: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:01 smithi078 ceph-mon[91136]: pgmap v7651: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:01.205Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:01.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:01.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:03 smithi146 ceph-mon[103278]: pgmap v7652: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:03 smithi078 ceph-mon[91136]: pgmap v7652: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:03 smithi078 ceph-mon[99959]: pgmap v7652: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:05 smithi146 ceph-mon[103278]: pgmap v7653: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[91136]: pgmap v7653: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[99959]: pgmap v7653: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:06.900 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:35:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:07 smithi078 ceph-mon[99959]: pgmap v7654: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:07 smithi078 ceph-mon[91136]: pgmap v7654: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:07.427 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:35:07.427 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 4h - - 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 4h - - 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 3s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 18s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 3s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 18s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 3s ago 4h 439M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 18s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 3s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 18s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:35:07.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 18s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 18s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 18s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 18s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 18s ago 4h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:35:07.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:35:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:07 smithi146 ceph-mon[103278]: pgmap v7654: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:35:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:35:07.738 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:35:08.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:35:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:35:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:35:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:35:08.001 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:35:08.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:35:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:08 smithi078 ceph-mon[99959]: from='client.61395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1598830563' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:08 smithi078 ceph-mon[91136]: from='client.61395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1598830563' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:08.314 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:35:08.314 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:08.314 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:08 smithi146 ceph-mon[103278]: from='client.61395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1598830563' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='client.81320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='client.61407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='client.61419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: pgmap v7655: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2730461376' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.370 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='client.81320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='client.61407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='client.61419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: pgmap v7655: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2730461376' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='client.81320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='client.61407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='client.61419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:09.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: pgmap v7655: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:09.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2730461376' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:09.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:09.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:10.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:10 smithi146 ceph-mon[103278]: pgmap v7656: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:10.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:10 smithi078 ceph-mon[91136]: pgmap v7656: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:10.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:10 smithi078 ceph-mon[99959]: pgmap v7656: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:10.694 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:35:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:35:11.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:11.206Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:11.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:11.206Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:11.445 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:11.207Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:13.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:13 smithi146 ceph-mon[103278]: pgmap v7657: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:13 smithi078 ceph-mon[91136]: pgmap v7657: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:13 smithi078 ceph-mon[99959]: pgmap v7657: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:15 smithi146 ceph-mon[103278]: pgmap v7658: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:15 smithi078 ceph-mon[91136]: pgmap v7658: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:15 smithi078 ceph-mon[99959]: pgmap v7658: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:17 smithi146 ceph-mon[103278]: pgmap v7659: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:17 smithi078 ceph-mon[91136]: pgmap v7659: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:17 smithi078 ceph-mon[99959]: pgmap v7659: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:19 smithi146 ceph-mon[103278]: pgmap v7660: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:19 smithi078 ceph-mon[91136]: pgmap v7660: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:19 smithi078 ceph-mon[99959]: pgmap v7660: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:35:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:35:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:21 smithi146 ceph-mon[103278]: pgmap v7661: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:21 smithi078 ceph-mon[91136]: pgmap v7661: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:21 smithi078 ceph-mon[99959]: pgmap v7661: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:21.207Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:21.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:21.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:23 smithi146 ceph-mon[103278]: pgmap v7662: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:23 smithi078 ceph-mon[91136]: pgmap v7662: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:23 smithi078 ceph-mon[99959]: pgmap v7662: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:24.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:24.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:25 smithi146 ceph-mon[103278]: pgmap v7663: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[91136]: pgmap v7663: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[99959]: pgmap v7663: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:26 smithi146 ceph-mon[103278]: pgmap v7664: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:26 smithi078 ceph-mon[91136]: pgmap v7664: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:26 smithi078 ceph-mon[99959]: pgmap v7664: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:29 smithi146 ceph-mon[103278]: pgmap v7665: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[99959]: pgmap v7665: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[91136]: pgmap v7665: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:30.987 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:35:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:35:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:31 smithi146 ceph-mon[103278]: pgmap v7666: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:31 smithi078 ceph-mon[91136]: pgmap v7666: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:31 smithi078 ceph-mon[99959]: pgmap v7666: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:31.207Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:31.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:31.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:31.208Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:33 smithi146 ceph-mon[103278]: pgmap v7667: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:33 smithi078 ceph-mon[91136]: pgmap v7667: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:33 smithi078 ceph-mon[99959]: pgmap v7667: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:35 smithi146 ceph-mon[103278]: pgmap v7668: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:35 smithi078 ceph-mon[91136]: pgmap v7668: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:35 smithi078 ceph-mon[99959]: pgmap v7668: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:37 smithi146 ceph-mon[103278]: pgmap v7669: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:37 smithi078 ceph-mon[91136]: pgmap v7669: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:37 smithi078 ceph-mon[99959]: pgmap v7669: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:38.608 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 4h - - 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 4h - - 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 14s ago 4h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 10s ago 4h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 14s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 10s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:35:39.139 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 14s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 14s ago 4h 439M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 10s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 14s ago 4h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 14s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 10s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 14s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 14s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 14s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 14s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:35:39.140 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:35:39.141 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 4h 75.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:35:39.141 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 14s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:35:39.141 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:35:39.141 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 14s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:35:39.141 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:35:39.449 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:35:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:35:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:35:39.451 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:35:39.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:39 smithi078 ceph-mon[91136]: pgmap v7670: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:39.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:39 smithi078 ceph-mon[99959]: pgmap v7670: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:39 smithi146 ceph-mon[103278]: pgmap v7670: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:39.717 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:35:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:35:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:35:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:35:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:35:39.718 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:35:40.028 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:35:40.028 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:35:40.028 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[91136]: from='client.81350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[91136]: from='client.81356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/522713080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/950583951' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[99959]: from='client.81350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[99959]: from='client.81356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/522713080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/950583951' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:40 smithi146 ceph-mon[103278]: from='client.81350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:40 smithi146 ceph-mon[103278]: from='client.81356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/522713080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:35:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/950583951' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:35:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:35:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:35:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:41 smithi146 ceph-mon[103278]: from='client.81362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:41 smithi146 ceph-mon[103278]: from='client.61455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:41 smithi146 ceph-mon[103278]: pgmap v7671: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-mon[91136]: from='client.81362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-mon[91136]: from='client.61455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-mon[91136]: pgmap v7671: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-mon[99959]: from='client.81362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-mon[99959]: from='client.61455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:35:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-mon[99959]: pgmap v7671: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:41.208Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:41.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:41.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:41.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:42 smithi146 ceph-mon[103278]: pgmap v7672: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:42 smithi078 ceph-mon[91136]: pgmap v7672: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:42 smithi078 ceph-mon[99959]: pgmap v7672: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:43.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:44 smithi146 ceph-mon[103278]: pgmap v7673: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:44 smithi078 ceph-mon[99959]: pgmap v7673: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:44 smithi078 ceph-mon[91136]: pgmap v7673: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:46 smithi146 ceph-mon[103278]: pgmap v7674: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:46 smithi078 ceph-mon[91136]: pgmap v7674: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:46 smithi078 ceph-mon[99959]: pgmap v7674: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:49 smithi146 ceph-mon[103278]: pgmap v7675: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:49 smithi078 ceph-mon[91136]: pgmap v7675: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:49 smithi078 ceph-mon[99959]: pgmap v7675: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:50 smithi146 ceph-mon[103278]: pgmap v7676: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[91136]: pgmap v7676: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:35:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-mon[99959]: pgmap v7676: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:50.850 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:35:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:35:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:51.209Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:51.209Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:35:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:35:51.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:35:53.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:53 smithi078 ceph-mon[91136]: pgmap v7677: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:53.238 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:53 smithi078 ceph-mon[99959]: pgmap v7677: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:53 smithi146 ceph-mon[103278]: pgmap v7677: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:35:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:55 smithi146 ceph-mon[103278]: pgmap v7678: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:55 smithi078 ceph-mon[91136]: pgmap v7678: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:55 smithi078 ceph-mon[99959]: pgmap v7678: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:57 smithi146 ceph-mon[103278]: pgmap v7679: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:57 smithi078 ceph-mon[91136]: pgmap v7679: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:57 smithi078 ceph-mon[99959]: pgmap v7679: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:35:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:59 smithi146 ceph-mon[103278]: pgmap v7680: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:35:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:59 smithi078 ceph-mon[91136]: pgmap v7680: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:35:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:35:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:59 smithi078 ceph-mon[99959]: pgmap v7680: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:35:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:35:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:36:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:36:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:01 smithi146 ceph-mon[103278]: pgmap v7681: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:01.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:01 smithi078 ceph-mon[99959]: pgmap v7681: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:01 smithi078 ceph-mon[91136]: pgmap v7681: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:01.210Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:01.210Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:01.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:02 smithi146 ceph-mon[103278]: pgmap v7682: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:02 smithi078 ceph-mon[91136]: pgmap v7682: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:02 smithi078 ceph-mon[99959]: pgmap v7682: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:04 smithi146 ceph-mon[103278]: pgmap v7683: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:04 smithi078 ceph-mon[91136]: pgmap v7683: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:04 smithi078 ceph-mon[99959]: pgmap v7683: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:06 smithi146 ceph-mon[103278]: pgmap v7684: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:06 smithi078 ceph-mon[99959]: pgmap v7684: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:06 smithi078 ceph-mon[91136]: pgmap v7684: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:09 smithi146 ceph-mon[103278]: pgmap v7685: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:09.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:09 smithi078 ceph-mon[91136]: pgmap v7685: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:09 smithi078 ceph-mon[99959]: pgmap v7685: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:10.324 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:36:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[91136]: pgmap v7686: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-mon[99959]: pgmap v7686: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:10 smithi146 ceph-mon[103278]: pgmap v7686: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 4h - - 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 4h - - 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 1s ago 4h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 1s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 6s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:36:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 6s ago 4h 440M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 1s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 6s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 1s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 1s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 1s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 1s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 1s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 1s ago 4h 75.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:36:10.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:36:10.859 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:36:10.859 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:36:10.859 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:36:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:36:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:36:11.177 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:36:11.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:36:11.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:36:11.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:36:11.179 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:36:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:36:11.458 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:36:11.458 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:36:11.458 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:36:11.458 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:36:11.458 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:36:11.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[91136]: from='client.81386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[91136]: from='client.61473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[91136]: from='client.61479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/297777627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:11.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[99959]: from='client.81386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[99959]: from='client.61473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[99959]: from='client.61479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/297777627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:11.459 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:11.210Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:11.459 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:11.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:11.459 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:11.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:11.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:11 smithi146 ceph-mon[103278]: from='client.81386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:11 smithi146 ceph-mon[103278]: from='client.61473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:11 smithi146 ceph-mon[103278]: from='client.61479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:11.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/297777627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:11.783 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:36:11.784 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:11.784 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:11.784 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:11.784 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:12.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:12 smithi078 ceph-mon[91136]: from='client.61491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:12.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3156356695' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:12.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:12 smithi078 ceph-mon[91136]: pgmap v7687: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:12.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:12 smithi078 ceph-mon[99959]: from='client.61491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:12.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3156356695' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:12.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:12 smithi078 ceph-mon[99959]: pgmap v7687: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:12.633 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:12 smithi146 ceph-mon[103278]: from='client.61491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:12.633 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3156356695' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:12.633 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:12 smithi146 ceph-mon[103278]: pgmap v7687: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:13.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:14 smithi146 ceph-mon[103278]: pgmap v7688: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:14 smithi078 ceph-mon[91136]: pgmap v7688: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:14 smithi078 ceph-mon[99959]: pgmap v7688: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:17.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:17 smithi146 ceph-mon[103278]: pgmap v7689: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:17.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:17 smithi078 ceph-mon[91136]: pgmap v7689: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:17 smithi078 ceph-mon[99959]: pgmap v7689: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:19 smithi146 ceph-mon[103278]: pgmap v7690: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:19 smithi078 ceph-mon[91136]: pgmap v7690: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:19 smithi078 ceph-mon[99959]: pgmap v7690: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:36:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:36:21.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:21 smithi078 ceph-mon[91136]: pgmap v7691: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:21.492 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:21 smithi078 ceph-mon[99959]: pgmap v7691: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:21.492 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:21.210Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:21.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:21.211Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:21.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:21.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:21 smithi146 ceph-mon[103278]: pgmap v7691: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:23 smithi146 ceph-mon[103278]: pgmap v7692: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:23 smithi078 ceph-mon[91136]: pgmap v7692: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:23 smithi078 ceph-mon[99959]: pgmap v7692: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:24.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:24.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:24 smithi146 ceph-mon[103278]: pgmap v7693: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:24 smithi078 ceph-mon[91136]: pgmap v7693: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:24 smithi078 ceph-mon[99959]: pgmap v7693: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:26 smithi146 ceph-mon[103278]: pgmap v7694: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:26 smithi078 ceph-mon[91136]: pgmap v7694: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:26 smithi078 ceph-mon[99959]: pgmap v7694: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:29.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:29 smithi146 ceph-mon[103278]: pgmap v7695: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:29.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:29 smithi078 ceph-mon[91136]: pgmap v7695: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:29 smithi078 ceph-mon[99959]: pgmap v7695: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:30 smithi146 ceph-mon[103278]: pgmap v7696: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:36:30] "GET /metrics HTTP/1.1" 200 39013 "" "Prometheus/2.43.0" 2024-09-21T00:36:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[91136]: pgmap v7696: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:30 smithi078 ceph-mon[99959]: pgmap v7696: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:31.211Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:31.212Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:31.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:33 smithi078 ceph-mon[91136]: pgmap v7697: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:33 smithi078 ceph-mon[99959]: pgmap v7697: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:33.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:33 smithi146 ceph-mon[103278]: pgmap v7697: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:34.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:34.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:35 smithi078 ceph-mon[91136]: pgmap v7698: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:35 smithi078 ceph-mon[99959]: pgmap v7698: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:35 smithi146 ceph-mon[103278]: pgmap v7698: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:36 smithi146 ceph-mon[103278]: pgmap v7699: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:36 smithi078 ceph-mon[91136]: pgmap v7699: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:36 smithi078 ceph-mon[99959]: pgmap v7699: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:39 smithi146 ceph-mon[103278]: pgmap v7700: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:39 smithi078 ceph-mon[91136]: pgmap v7700: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:39 smithi078 ceph-mon[99959]: pgmap v7700: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:36:40] "GET /metrics HTTP/1.1" 200 39013 "" "Prometheus/2.43.0" 2024-09-21T00:36:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:41 smithi146 ceph-mon[103278]: pgmap v7701: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:41 smithi078 ceph-mon[91136]: pgmap v7701: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:41 smithi078 ceph-mon[99959]: pgmap v7701: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:41.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:41.213Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:41.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:42.077 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:36:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:42 smithi146 ceph-mon[103278]: pgmap v7702: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:42 smithi146 ceph-mon[103278]: from='client.61503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:42 smithi078 ceph-mon[91136]: pgmap v7702: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:42 smithi078 ceph-mon[91136]: from='client.61503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:42 smithi078 ceph-mon[99959]: pgmap v7702: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:42 smithi078 ceph-mon[99959]: from='client.61503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:42.605 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:36:42.605 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 4h - - 2024-09-21T00:36:42.605 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 4h - - 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 17s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 13s ago 4h 88.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 17s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 13s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 17s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 17s ago 4h 441M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 13s ago 4h 289M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 17s ago 4h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 17s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 13s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 17s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 17s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 17s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 17s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:36:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 13s ago 4h 76.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 17s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 17s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:36:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:36:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:36:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:36:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:36:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:36:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:36:42.918 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:36:43.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:36:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:36:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:36:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:36:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:36:43.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:36:43.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:43.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[99959]: from='client.61509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:43.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[99959]: from='client.61515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:43.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/289800131' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:43.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:43.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[91136]: from='client.61509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:43.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[91136]: from='client.61515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:43.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/289800131' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:43.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:43.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:43 smithi146 ceph-mon[103278]: from='client.61509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:43.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:43 smithi146 ceph-mon[103278]: from='client.61515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:43.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/289800131' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:36:43.496 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:36:43.496 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:36:43.496 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:36:44.436 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:44 smithi146 ceph-mon[103278]: from='client.61527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:44.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:44.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/285097811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:44.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:44 smithi146 ceph-mon[103278]: pgmap v7703: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[91136]: from='client.61527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/285097811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[91136]: pgmap v7703: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[99959]: from='client.61527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/285097811' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:36:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:44 smithi078 ceph-mon[99959]: pgmap v7703: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:46 smithi146 ceph-mon[103278]: pgmap v7704: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:47.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:46 smithi078 ceph-mon[91136]: pgmap v7704: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:46 smithi078 ceph-mon[99959]: pgmap v7704: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:49 smithi146 ceph-mon[103278]: pgmap v7705: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:49.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:49 smithi078 ceph-mon[91136]: pgmap v7705: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:49 smithi078 ceph-mon[99959]: pgmap v7705: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:50 smithi146 ceph-mon[103278]: pgmap v7706: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:36:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:36:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[91136]: pgmap v7706: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:36:51.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:50 smithi078 ceph-mon[99959]: pgmap v7706: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:51.486 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:51.213Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:51.487 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:51.214Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:36:51.487 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:36:51.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:36:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:53 smithi146 ceph-mon[103278]: pgmap v7707: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:53 smithi078 ceph-mon[91136]: pgmap v7707: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:53 smithi078 ceph-mon[99959]: pgmap v7707: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:54.334 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:54.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:36:55.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:55 smithi078 ceph-mon[91136]: pgmap v7708: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:55.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:55 smithi078 ceph-mon[99959]: pgmap v7708: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:55 smithi146 ceph-mon[103278]: pgmap v7708: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:56 smithi146 ceph-mon[103278]: pgmap v7709: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:56 smithi078 ceph-mon[91136]: pgmap v7709: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:56 smithi078 ceph-mon[99959]: pgmap v7709: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:36:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:58 smithi146 ceph-mon[103278]: pgmap v7710: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:58 smithi078 ceph-mon[91136]: pgmap v7710: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:58 smithi078 ceph-mon[99959]: pgmap v7710: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:36:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:36:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:00 smithi146 ceph-mon[103278]: pgmap v7711: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:00 smithi078 ceph-mon[91136]: pgmap v7711: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:00 smithi078 ceph-mon[99959]: pgmap v7711: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:37:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:37:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:01.214Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:01.215Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:01.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:02 smithi146 ceph-mon[103278]: pgmap v7712: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:02 smithi078 ceph-mon[99959]: pgmap v7712: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:02 smithi078 ceph-mon[91136]: pgmap v7712: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:04 smithi146 ceph-mon[103278]: pgmap v7713: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:04 smithi078 ceph-mon[99959]: pgmap v7713: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:04 smithi078 ceph-mon[91136]: pgmap v7713: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:07 smithi146 ceph-mon[103278]: pgmap v7714: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:07.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:07 smithi078 ceph-mon[99959]: pgmap v7714: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:07 smithi078 ceph-mon[91136]: pgmap v7714: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:09 smithi146 ceph-mon[103278]: pgmap v7715: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:09.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:09 smithi078 ceph-mon[99959]: pgmap v7715: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:09 smithi078 ceph-mon[91136]: pgmap v7715: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:10 smithi146 ceph-mon[103278]: pgmap v7716: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:37:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[91136]: pgmap v7716: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:10 smithi078 ceph-mon[99959]: pgmap v7716: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:11.215Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:11.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:11.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:12 smithi146 ceph-mon[103278]: pgmap v7717: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:12 smithi078 ceph-mon[99959]: pgmap v7717: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:12 smithi078 ceph-mon[91136]: pgmap v7717: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:13.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:13.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:13.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:13.789 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 4h - - 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 4h - - 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 9s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 4s ago 4h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 9s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 4s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 9s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 9s ago 4h 440M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 4s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:37:14.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 9s ago 4h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 9s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 4s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 9s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 9s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 9s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 9s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 4s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 4s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 4s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 4s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 4s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 9s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 9s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:37:14.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[99959]: from='client.61539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[99959]: from='client.81464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[99959]: pgmap v7718: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[91136]: from='client.61539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:14.329 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[91136]: from='client.81464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:14.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:14 smithi078 ceph-mon[91136]: pgmap v7718: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:14.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:14.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:14 smithi146 ceph-mon[103278]: from='client.61539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:14.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:14 smithi146 ceph-mon[103278]: from='client.81464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:14.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:14 smithi146 ceph-mon[103278]: pgmap v7718: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:37:14.660 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:37:14.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:14.661 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:37:14.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:37:14.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:37:14.661 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:37:14.940 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:37:14.941 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:37:14.942 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:37:14.942 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:37:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:15 smithi146 ceph-mon[103278]: from='client.61551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1125061446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:15 smithi146 ceph-mon[103278]: from='client.81482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:15.325 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:37:15.325 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:15.325 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:15.325 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:15.326 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:15.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:15 smithi078 ceph-mon[91136]: from='client.61551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:15.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1125061446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:15.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:15 smithi078 ceph-mon[91136]: from='client.81482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:15.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:15 smithi078 ceph-mon[99959]: from='client.61551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:15.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1125061446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:15.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:15 smithi078 ceph-mon[99959]: from='client.81482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:16.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1142082859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:16.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:16 smithi078 ceph-mon[99959]: pgmap v7719: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:16.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1142082859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:16.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:16 smithi078 ceph-mon[91136]: pgmap v7719: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1142082859' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:16 smithi146 ceph-mon[103278]: pgmap v7719: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:18 smithi146 ceph-mon[103278]: pgmap v7720: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:18 smithi078 ceph-mon[99959]: pgmap v7720: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:18 smithi078 ceph-mon[91136]: pgmap v7720: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:20 smithi146 ceph-mon[103278]: pgmap v7721: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:20 smithi078 ceph-mon[99959]: pgmap v7721: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:20 smithi078 ceph-mon[91136]: pgmap v7721: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:37:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:37:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:21.215Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:21.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:21.216Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:22 smithi146 ceph-mon[103278]: pgmap v7722: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:22 smithi078 ceph-mon[99959]: pgmap v7722: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:22 smithi078 ceph-mon[91136]: pgmap v7722: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:24 smithi146 ceph-mon[103278]: pgmap v7723: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:24 smithi078 ceph-mon[91136]: pgmap v7723: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:24 smithi078 ceph-mon[99959]: pgmap v7723: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[91136]: pgmap v7724: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:26 smithi078 ceph-mon[99959]: pgmap v7724: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:26 smithi146 ceph-mon[103278]: pgmap v7724: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:28 smithi146 ceph-mon[103278]: pgmap v7725: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:28 smithi078 ceph-mon[91136]: pgmap v7725: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:28 smithi078 ceph-mon[99959]: pgmap v7725: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[91136]: pgmap v7726: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:37:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:37:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:30 smithi078 ceph-mon[99959]: pgmap v7726: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:30 smithi146 ceph-mon[103278]: pgmap v7726: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:31.216Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 9 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:31.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:31.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:32 smithi146 ceph-mon[103278]: pgmap v7727: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:32 smithi078 ceph-mon[99959]: pgmap v7727: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:32 smithi078 ceph-mon[91136]: pgmap v7727: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:34.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:34 smithi078 ceph-mon[91136]: pgmap v7728: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:34.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:34 smithi078 ceph-mon[99959]: pgmap v7728: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:34 smithi146 ceph-mon[103278]: pgmap v7728: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:36 smithi146 ceph-mon[103278]: pgmap v7729: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:36 smithi078 ceph-mon[99959]: pgmap v7729: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:36 smithi078 ceph-mon[91136]: pgmap v7729: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:38 smithi146 ceph-mon[103278]: pgmap v7730: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:38 smithi078 ceph-mon[91136]: pgmap v7730: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:38 smithi078 ceph-mon[99959]: pgmap v7730: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:40 smithi146 ceph-mon[103278]: pgmap v7731: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:40 smithi078 ceph-mon[91136]: pgmap v7731: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:40 smithi078 ceph-mon[99959]: pgmap v7731: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:37:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:37:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:41.217Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:41.217Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:41.218Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:42 smithi146 ceph-mon[103278]: pgmap v7732: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:42 smithi078 ceph-mon[91136]: pgmap v7732: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:42 smithi078 ceph-mon[99959]: pgmap v7732: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:44 smithi146 ceph-mon[103278]: pgmap v7733: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:44 smithi078 ceph-mon[91136]: pgmap v7733: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:44 smithi078 ceph-mon[99959]: pgmap v7733: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:45.613 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:37:46.140 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:37:46.140 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 4h - - 2024-09-21T00:37:46.140 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 4h - - 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 0s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 16s ago 4h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 0s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 16s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 0s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 0s ago 4h 441M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 16s ago 4h 288M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 0s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 0s ago 4h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 16s ago 4h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 0s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 0s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:37:46.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 0s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 0s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 16s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 16s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 16s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 16s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 16s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 0s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 0s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:37:46.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:37:46.456 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:37:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:37:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[99959]: from='client.81494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[99959]: from='client.81500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[99959]: pgmap v7734: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[91136]: from='client.81494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:46.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[91136]: from='client.81500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:46.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:46 smithi078 ceph-mon[91136]: pgmap v7734: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:37:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:37:46.731 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:37:46.731 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:37:46.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:37:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:46 smithi146 ceph-mon[103278]: from='client.81494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:46 smithi146 ceph-mon[103278]: from='client.81500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:46 smithi146 ceph-mon[103278]: pgmap v7734: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:47.042 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:37:47.042 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:47.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:37:47.042 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:47.043 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:37:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:47 smithi146 ceph-mon[103278]: from='client.81506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/59754324' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:47 smithi146 ceph-mon[103278]: from='client.81518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1081526600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[99959]: from='client.81506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/59754324' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[99959]: from='client.81518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1081526600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[91136]: from='client.81506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/59754324' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[91136]: from='client.81518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:37:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1081526600' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:37:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:48 smithi146 ceph-mon[103278]: pgmap v7735: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:48 smithi078 ceph-mon[99959]: pgmap v7735: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:48 smithi078 ceph-mon[91136]: pgmap v7735: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:50.982 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:37:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:37:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:50 smithi146 ceph-mon[103278]: pgmap v7736: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[99959]: pgmap v7736: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:37:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:50 smithi078 ceph-mon[91136]: pgmap v7736: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:51.217Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:51.218Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:37:51.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:37:51.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:37:52.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:52 smithi078 ceph-mon[91136]: pgmap v7737: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:52.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:52 smithi078 ceph-mon[99959]: pgmap v7737: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:52 smithi146 ceph-mon[103278]: pgmap v7737: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:37:54.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:54 smithi146 ceph-mon[103278]: pgmap v7738: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:54 smithi078 ceph-mon[91136]: pgmap v7738: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:54 smithi078 ceph-mon[99959]: pgmap v7738: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:56.363 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:56 smithi078 ceph-mon[91136]: pgmap v7739: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:56.363 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:56 smithi078 ceph-mon[99959]: pgmap v7739: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:56 smithi146 ceph-mon[103278]: pgmap v7739: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:37:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:58 smithi146 ceph-mon[103278]: pgmap v7740: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:58 smithi078 ceph-mon[91136]: pgmap v7740: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:58 smithi078 ceph-mon[99959]: pgmap v7740: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:37:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:37:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:37:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:37:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:37:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:00 smithi146 ceph-mon[103278]: pgmap v7741: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:00 smithi078 ceph-mon[99959]: pgmap v7741: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:00 smithi078 ceph-mon[91136]: pgmap v7741: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:38:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:38:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:01.218Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:01.218Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:01.219Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:02 smithi146 ceph-mon[103278]: pgmap v7742: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:02 smithi078 ceph-mon[91136]: pgmap v7742: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:02 smithi078 ceph-mon[99959]: pgmap v7742: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:04 smithi146 ceph-mon[103278]: pgmap v7743: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:04 smithi078 ceph-mon[99959]: pgmap v7743: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:04 smithi078 ceph-mon[91136]: pgmap v7743: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:06 smithi146 ceph-mon[103278]: pgmap v7744: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[99959]: pgmap v7744: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:06 smithi078 ceph-mon[91136]: pgmap v7744: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:08 smithi146 ceph-mon[103278]: pgmap v7745: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:08 smithi078 ceph-mon[91136]: pgmap v7745: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:08 smithi078 ceph-mon[99959]: pgmap v7745: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:38:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:38:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:11 smithi146 ceph-mon[103278]: pgmap v7746: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[99959]: pgmap v7746: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:11.219Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:11.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:11.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[91136]: pgmap v7746: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:13 smithi146 ceph-mon[103278]: pgmap v7747: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:13 smithi078 ceph-mon[99959]: pgmap v7747: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:13 smithi078 ceph-mon[91136]: pgmap v7747: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:15.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:15 smithi078 ceph-mon[99959]: pgmap v7748: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:15.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:15 smithi078 ceph-mon[91136]: pgmap v7748: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:15.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:15 smithi146 ceph-mon[103278]: pgmap v7748: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:17.214 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:17 smithi146 ceph-mon[103278]: pgmap v7749: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:17.337 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:38:17.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:17 smithi078 ceph-mon[91136]: pgmap v7749: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:17 smithi078 ceph-mon[99959]: pgmap v7749: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 4h - - 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 4h - - 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 12s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 7s ago 4h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 12s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:38:17.871 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 7s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 12s ago 4h 580M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 12s ago 4h 443M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 7s ago 4h 290M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 12s ago 4h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 12s ago 4h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 7s ago 4h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 12s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 12s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 12s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 12s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 7s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 7s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:38:17.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 7s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:38:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 7s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:38:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 7s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:38:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 12s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:38:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:38:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 12s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:38:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:38:18.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:38:18.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:18.188 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:38:18.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:38:18.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:38:18.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:38:18.456 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:38:18.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:38:18.766 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:38:18.766 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:18.766 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:18.766 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:18.766 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:19 smithi146 ceph-mon[103278]: from='client.61611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:19 smithi146 ceph-mon[103278]: from='client.61617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:19 smithi146 ceph-mon[103278]: from='client.81542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:19 smithi146 ceph-mon[103278]: pgmap v7750: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2589569108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2827130218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[91136]: from='client.61611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[91136]: from='client.61617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[91136]: from='client.81542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[91136]: pgmap v7750: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2589569108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2827130218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[99959]: from='client.61611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[99959]: from='client.61617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[99959]: from='client.81542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[99959]: pgmap v7750: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2589569108' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2827130218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:20 smithi146 ceph-mon[103278]: from='client.81554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:20 smithi078 ceph-mon[91136]: from='client.81554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:20 smithi078 ceph-mon[99959]: from='client.81554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:38:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T00:38:21.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:21 smithi078 ceph-mon[99959]: pgmap v7751: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:21.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:21 smithi078 ceph-mon[91136]: pgmap v7751: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:21.220Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:21.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:21.490 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:21.220Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:21 smithi146 ceph-mon[103278]: pgmap v7751: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:22 smithi146 ceph-mon[103278]: pgmap v7752: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:22 smithi078 ceph-mon[99959]: pgmap v7752: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:22 smithi078 ceph-mon[91136]: pgmap v7752: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:24 smithi146 ceph-mon[103278]: pgmap v7753: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:24 smithi078 ceph-mon[99959]: pgmap v7753: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:24 smithi078 ceph-mon[91136]: pgmap v7753: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:26 smithi146 ceph-mon[103278]: pgmap v7754: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[99959]: pgmap v7754: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:26 smithi078 ceph-mon[91136]: pgmap v7754: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:28 smithi146 ceph-mon[103278]: pgmap v7755: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:28 smithi078 ceph-mon[91136]: pgmap v7755: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:28 smithi078 ceph-mon[99959]: pgmap v7755: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:30 smithi146 ceph-mon[103278]: pgmap v7756: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:30 smithi078 ceph-mon[91136]: pgmap v7756: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:30 smithi078 ceph-mon[99959]: pgmap v7756: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:38:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:38:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:31.220Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:31.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:31.221Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:32 smithi146 ceph-mon[103278]: pgmap v7757: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:32 smithi078 ceph-mon[91136]: pgmap v7757: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:32 smithi078 ceph-mon[99959]: pgmap v7757: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:34 smithi146 ceph-mon[103278]: pgmap v7758: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:34 smithi078 ceph-mon[91136]: pgmap v7758: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:34 smithi078 ceph-mon[99959]: pgmap v7758: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:36 smithi146 ceph-mon[103278]: pgmap v7759: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:36.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:36 smithi078 ceph-mon[99959]: pgmap v7759: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:36.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:36 smithi078 ceph-mon[91136]: pgmap v7759: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:38.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:38 smithi078 ceph-mon[91136]: pgmap v7760: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:38.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:38 smithi078 ceph-mon[99959]: pgmap v7760: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:38 smithi146 ceph-mon[103278]: pgmap v7760: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:40 smithi146 ceph-mon[103278]: pgmap v7761: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:40 smithi078 ceph-mon[91136]: pgmap v7761: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:40 smithi078 ceph-mon[99959]: pgmap v7761: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:38:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:38:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:41.221Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:41.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:41.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:42 smithi146 ceph-mon[103278]: pgmap v7762: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:42 smithi078 ceph-mon[91136]: pgmap v7762: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:42 smithi078 ceph-mon[99959]: pgmap v7762: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:44 smithi146 ceph-mon[103278]: pgmap v7763: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:44 smithi078 ceph-mon[99959]: pgmap v7763: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:44 smithi078 ceph-mon[91136]: pgmap v7763: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:46 smithi146 ceph-mon[103278]: pgmap v7764: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[99959]: pgmap v7764: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:46 smithi078 ceph-mon[91136]: pgmap v7764: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:48 smithi146 ceph-mon[103278]: pgmap v7765: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:48 smithi078 ceph-mon[99959]: pgmap v7765: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:48 smithi078 ceph-mon[91136]: pgmap v7765: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:49.059 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:38:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:49 smithi146 ceph-mon[103278]: from='client.81566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:49.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:49 smithi078 ceph-mon[91136]: from='client.81566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:49 smithi078 ceph-mon[99959]: from='client.81566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:49.591 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:38:49.591 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 4h - - 2024-09-21T00:38:49.591 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 4h - - 2024-09-21T00:38:49.591 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 3s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 3s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 3s ago 4h 444M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 4h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 3s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:38:49.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 19s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:38:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:38:49.899 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:38:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:38:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:38:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:38:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:38:49.901 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:38:50.172 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:38:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:38:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:38:50.173 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:38:50.486 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:38:50.486 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:50.486 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:38:50.486 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:50.486 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:38:50.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[99959]: from='client.61653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:50.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[99959]: from='client.81578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:50.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/86248590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:50.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[99959]: pgmap v7766: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:50.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[91136]: from='client.61653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:50.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[91136]: from='client.81578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:50.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/86248590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:50.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-mon[91136]: pgmap v7766: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:50 smithi146 ceph-mon[103278]: from='client.61653 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:50 smithi146 ceph-mon[103278]: from='client.81578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/86248590' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:38:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:50 smithi146 ceph-mon[103278]: pgmap v7766: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:38:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:38:51.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[99959]: from='client.81590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2403031639' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[91136]: from='client.81590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2403031639' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:51.222Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:51.222Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:38:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:38:51.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:38:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:51 smithi146 ceph-mon[103278]: from='client.81590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:38:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:38:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2403031639' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:38:52.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:52 smithi078 ceph-mon[91136]: pgmap v7767: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:52.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:52 smithi078 ceph-mon[99959]: pgmap v7767: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:52 smithi146 ceph-mon[103278]: pgmap v7767: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:38:54.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:54 smithi146 ceph-mon[103278]: pgmap v7768: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:54 smithi078 ceph-mon[91136]: pgmap v7768: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:54 smithi078 ceph-mon[99959]: pgmap v7768: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:56 smithi146 ceph-mon[103278]: pgmap v7769: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:56 smithi078 ceph-mon[99959]: pgmap v7769: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:56 smithi078 ceph-mon[91136]: pgmap v7769: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:38:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:58 smithi146 ceph-mon[103278]: pgmap v7770: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:58 smithi078 ceph-mon[91136]: pgmap v7770: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:58 smithi078 ceph-mon[99959]: pgmap v7770: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:38:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:38:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:38:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:38:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:38:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:00 smithi146 ceph-mon[103278]: pgmap v7771: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:00 smithi078 ceph-mon[91136]: pgmap v7771: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:00 smithi078 ceph-mon[99959]: pgmap v7771: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:39:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:39:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:01.223Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:01.223Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:01.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:02 smithi146 ceph-mon[103278]: pgmap v7772: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:02 smithi078 ceph-mon[99959]: pgmap v7772: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:02 smithi078 ceph-mon[91136]: pgmap v7772: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:04 smithi146 ceph-mon[103278]: pgmap v7773: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:04 smithi078 ceph-mon[99959]: pgmap v7773: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:04 smithi078 ceph-mon[91136]: pgmap v7773: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:06 smithi146 ceph-mon[103278]: pgmap v7774: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[99959]: pgmap v7774: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:06 smithi078 ceph-mon[91136]: pgmap v7774: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:08 smithi146 ceph-mon[103278]: pgmap v7775: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:08 smithi078 ceph-mon[99959]: pgmap v7775: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:08 smithi078 ceph-mon[91136]: pgmap v7775: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:10.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:10 smithi146 ceph-mon[103278]: pgmap v7776: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:10 smithi078 ceph-mon[99959]: pgmap v7776: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:10 smithi078 ceph-mon[91136]: pgmap v7776: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:39:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:39:11.497 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:11.223Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:11.497 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:11.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:11.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:11.224Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:12 smithi146 ceph-mon[103278]: pgmap v7777: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:12 smithi078 ceph-mon[91136]: pgmap v7777: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:12 smithi078 ceph-mon[99959]: pgmap v7777: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:14 smithi146 ceph-mon[103278]: pgmap v7778: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:14 smithi078 ceph-mon[91136]: pgmap v7778: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:14 smithi078 ceph-mon[99959]: pgmap v7778: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:16 smithi146 ceph-mon[103278]: pgmap v7779: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:16 smithi078 ceph-mon[91136]: pgmap v7779: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:16 smithi078 ceph-mon[99959]: pgmap v7779: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:18.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:18 smithi078 ceph-mon[91136]: pgmap v7780: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:18.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:18 smithi078 ceph-mon[99959]: pgmap v7780: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:18 smithi146 ceph-mon[103278]: pgmap v7780: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:20.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:20 smithi078 ceph-mon[91136]: pgmap v7781: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:20.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:20 smithi078 ceph-mon[99959]: pgmap v7781: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:20 smithi146 ceph-mon[103278]: pgmap v7781: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:20.771 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:39:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:39:20.772 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:39:21.299 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:39:21.299 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 4h - - 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 4h - - 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 15s ago 4h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 10s ago 4h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 15s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 10s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 15s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 15s ago 4h 445M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 10s ago 4h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 15s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 15s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 10s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 15s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 15s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:39:21.300 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 15s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 15s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 15s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 15s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:39:21.301 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:39:21.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-mon[91136]: from='client.61683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:21.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-mon[91136]: from='client.81608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:21.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-mon[99959]: from='client.61683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:21.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-mon[99959]: from='client.81608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:21.302 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:21.224Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:21.302 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:21.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:21.302 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:21.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:21 smithi146 ceph-mon[103278]: from='client.61683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:21 smithi146 ceph-mon[103278]: from='client.81608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:39:21.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:39:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:39:21.876 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:39:21.876 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:39:21.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:39:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:39:22.184 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:22.185 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:22.185 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:22.185 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:22 smithi146 ceph-mon[103278]: from='client.61695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/615901599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:22 smithi146 ceph-mon[103278]: from='client.81626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:22 smithi146 ceph-mon[103278]: pgmap v7782: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/656594481' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[91136]: from='client.61695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/615901599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[91136]: from='client.81626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[91136]: pgmap v7782: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/656594481' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[99959]: from='client.61695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/615901599' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[99959]: from='client.81626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[99959]: pgmap v7782: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/656594481' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:24 smithi146 ceph-mon[103278]: pgmap v7783: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:24 smithi078 ceph-mon[91136]: pgmap v7783: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:24 smithi078 ceph-mon[99959]: pgmap v7783: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:26 smithi146 ceph-mon[103278]: pgmap v7784: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[99959]: pgmap v7784: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:26 smithi078 ceph-mon[91136]: pgmap v7784: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:28 smithi146 ceph-mon[103278]: pgmap v7785: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:28 smithi078 ceph-mon[91136]: pgmap v7785: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:28 smithi078 ceph-mon[99959]: pgmap v7785: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:30 smithi146 ceph-mon[103278]: pgmap v7786: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:30 smithi078 ceph-mon[91136]: pgmap v7786: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:30 smithi078 ceph-mon[99959]: pgmap v7786: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:39:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:39:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:31.225Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:31.225Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:31.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:32 smithi146 ceph-mon[103278]: pgmap v7787: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:32 smithi078 ceph-mon[99959]: pgmap v7787: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:32 smithi078 ceph-mon[91136]: pgmap v7787: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:34 smithi146 ceph-mon[103278]: pgmap v7788: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:34 smithi078 ceph-mon[91136]: pgmap v7788: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:34 smithi078 ceph-mon[99959]: pgmap v7788: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:36 smithi146 ceph-mon[103278]: pgmap v7789: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:36 smithi078 ceph-mon[91136]: pgmap v7789: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:36 smithi078 ceph-mon[99959]: pgmap v7789: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:38.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:38 smithi078 ceph-mon[99959]: pgmap v7790: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:38.324 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:38 smithi078 ceph-mon[91136]: pgmap v7790: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:38.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:38 smithi146 ceph-mon[103278]: pgmap v7790: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:40.464 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:40 smithi146 ceph-mon[103278]: pgmap v7791: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:40.471 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:40 smithi078 ceph-mon[99959]: pgmap v7791: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:40.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:40 smithi078 ceph-mon[91136]: pgmap v7791: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:40.755 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:39:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:39:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:41.225Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:41.226Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:41.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:42 smithi146 ceph-mon[103278]: pgmap v7792: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:42 smithi078 ceph-mon[91136]: pgmap v7792: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:42 smithi078 ceph-mon[99959]: pgmap v7792: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:43.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:44 smithi146 ceph-mon[103278]: pgmap v7793: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:44 smithi078 ceph-mon[91136]: pgmap v7793: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:44 smithi078 ceph-mon[99959]: pgmap v7793: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:47 smithi146 ceph-mon[103278]: pgmap v7794: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[99959]: pgmap v7794: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:47 smithi078 ceph-mon[91136]: pgmap v7794: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:48 smithi146 ceph-mon[103278]: pgmap v7795: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:48 smithi078 ceph-mon[99959]: pgmap v7795: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:48 smithi078 ceph-mon[91136]: pgmap v7795: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:50 smithi146 ceph-mon[103278]: pgmap v7796: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:50 smithi078 ceph-mon[99959]: pgmap v7796: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:50 smithi078 ceph-mon[91136]: pgmap v7796: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:39:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:39:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:51.226Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:51.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:39:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:39:51.227Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:39:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:39:52.500 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:39:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:52 smithi078 ceph-mon[91136]: pgmap v7797: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:52 smithi078 ceph-mon[99959]: pgmap v7797: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:53.053 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:39:53.053 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 4h - - 2024-09-21T00:39:53.053 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 4h - - 2024-09-21T00:39:53.053 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 2s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 2s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 6s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 6s ago 4h 445M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 2s ago 4h 292M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 6s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 2s ago 4h 14.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 2s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:39:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 2s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 2s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 2s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 2s ago 4h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:39:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:39:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:52 smithi146 ceph-mon[103278]: pgmap v7797: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:39:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:39:53.391 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:39:53.658 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:39:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:39:53.659 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:39:53.967 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:39:53.968 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:39:53.968 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:39:53.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:53.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[99959]: from='client.81638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:53.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[99959]: from='client.61725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:53.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[99959]: from='client.81650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:53.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4023162185' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:53.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:53.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[91136]: from='client.81638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:53.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[91136]: from='client.61725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:53.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[91136]: from='client.81650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:53.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4023162185' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:39:54.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:53 smithi146 ceph-mon[103278]: from='client.81638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:54.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:53 smithi146 ceph-mon[103278]: from='client.61725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:54.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:53 smithi146 ceph-mon[103278]: from='client.81650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:54.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4023162185' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:39:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:54 smithi078 ceph-mon[91136]: from='client.61743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2972224747' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:54 smithi078 ceph-mon[91136]: pgmap v7798: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:54 smithi078 ceph-mon[99959]: from='client.61743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2972224747' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:54 smithi078 ceph-mon[99959]: pgmap v7798: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:54 smithi146 ceph-mon[103278]: from='client.61743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:39:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2972224747' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:39:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:54 smithi146 ceph-mon[103278]: pgmap v7798: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:56 smithi146 ceph-mon[103278]: pgmap v7799: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:56 smithi078 ceph-mon[91136]: pgmap v7799: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:56 smithi078 ceph-mon[99959]: pgmap v7799: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:39:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:58 smithi146 ceph-mon[103278]: pgmap v7800: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:58 smithi078 ceph-mon[99959]: pgmap v7800: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:58 smithi078 ceph-mon[91136]: pgmap v7800: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:39:59.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:39:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:39:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:39:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:39:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:00 smithi078 ceph-mon[99959]: pgmap v7801: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T00:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:00 smithi078 ceph-mon[91136]: pgmap v7801: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:00.372 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:00.372 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:00 smithi146 ceph-mon[103278]: pgmap v7801: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:40:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:40:01.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:01.227Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:01.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:01.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:01.446 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:01.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:02 smithi146 ceph-mon[103278]: pgmap v7802: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:02 smithi078 ceph-mon[91136]: pgmap v7802: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:02 smithi078 ceph-mon[99959]: pgmap v7802: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:04 smithi146 ceph-mon[103278]: pgmap v7803: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:04 smithi078 ceph-mon[91136]: pgmap v7803: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:04 smithi078 ceph-mon[99959]: pgmap v7803: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:06 smithi146 ceph-mon[103278]: pgmap v7804: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:06 smithi078 ceph-mon[91136]: pgmap v7804: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:06 smithi078 ceph-mon[99959]: pgmap v7804: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:08 smithi146 ceph-mon[103278]: pgmap v7805: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:08 smithi078 ceph-mon[91136]: pgmap v7805: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:08 smithi078 ceph-mon[99959]: pgmap v7805: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:10 smithi146 ceph-mon[103278]: pgmap v7806: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:10 smithi078 ceph-mon[91136]: pgmap v7806: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:10 smithi078 ceph-mon[99959]: pgmap v7806: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:40:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:40:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:11.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:11.228Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:11.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:13.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:12 smithi146 ceph-mon[103278]: pgmap v7807: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:12 smithi078 ceph-mon[91136]: pgmap v7807: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:12 smithi078 ceph-mon[99959]: pgmap v7807: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:14 smithi146 ceph-mon[103278]: pgmap v7808: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:14 smithi078 ceph-mon[99959]: pgmap v7808: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:14 smithi078 ceph-mon[91136]: pgmap v7808: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:16 smithi146 ceph-mon[103278]: pgmap v7809: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:16 smithi078 ceph-mon[91136]: pgmap v7809: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:16 smithi078 ceph-mon[99959]: pgmap v7809: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:18.505 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:18 smithi146 ceph-mon[103278]: pgmap v7810: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:18 smithi078 ceph-mon[99959]: pgmap v7810: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:18 smithi078 ceph-mon[91136]: pgmap v7810: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:20 smithi146 ceph-mon[103278]: pgmap v7811: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:20 smithi078 ceph-mon[99959]: pgmap v7811: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:20 smithi078 ceph-mon[91136]: pgmap v7811: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:40:20.962 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:40:20] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T00:40:21.480 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:21.228Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:21.481 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:21.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:21.481 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:21.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:22.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:22 smithi078 ceph-mon[91136]: pgmap v7812: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:22.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:22 smithi078 ceph-mon[99959]: pgmap v7812: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:22 smithi146 ceph-mon[103278]: pgmap v7812: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:23.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:24.265 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:40:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:24 smithi146 ceph-mon[103278]: pgmap v7813: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:24 smithi078 ceph-mon[91136]: pgmap v7813: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:24 smithi078 ceph-mon[99959]: pgmap v7813: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 4h - - 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 4h - - 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 18s ago 4h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 13s ago 4h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 18s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 13s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 18s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 18s ago 4h 444M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 13s ago 4h 291M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:40:24.795 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 18s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 18s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 13s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 18s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 18s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 18s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 18s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 13s ago 4h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 18s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:40:24.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 18s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:40:24.797 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:40:25.103 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:40:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:40:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:40:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:40:25.124 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:40:25.371 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:40:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:40:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:40:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:40:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:40:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:40:25.373 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:40:25.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[99959]: from='client.61755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[99959]: from='client.81680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[99959]: from='client.81686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/115756499' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:25.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[91136]: from='client.61755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[91136]: from='client.81680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[91136]: from='client.81686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/115756499' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:25 smithi146 ceph-mon[103278]: from='client.61755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:25 smithi146 ceph-mon[103278]: from='client.81680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:25 smithi146 ceph-mon[103278]: from='client.81686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/115756499' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:25.679 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:25.679 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:25.679 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:25.679 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:25.679 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:26 smithi146 ceph-mon[103278]: from='client.61779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/365923658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:26 smithi146 ceph-mon[103278]: pgmap v7814: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:26 smithi078 ceph-mon[99959]: from='client.61779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/365923658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:26 smithi078 ceph-mon[99959]: pgmap v7814: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:26 smithi078 ceph-mon[91136]: from='client.61779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/365923658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:26 smithi078 ceph-mon[91136]: pgmap v7814: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:28 smithi146 ceph-mon[103278]: pgmap v7815: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:28 smithi078 ceph-mon[99959]: pgmap v7815: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:28 smithi078 ceph-mon[91136]: pgmap v7815: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:30 smithi146 ceph-mon[103278]: pgmap v7816: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:30 smithi078 ceph-mon[99959]: pgmap v7816: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:30 smithi078 ceph-mon[91136]: pgmap v7816: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:40:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:40:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:31.229Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:31.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:31.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:33.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:33 smithi078 ceph-mon[99959]: pgmap v7817: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:33 smithi078 ceph-mon[91136]: pgmap v7817: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:33 smithi146 ceph-mon[103278]: pgmap v7817: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:35 smithi078 ceph-mon[99959]: pgmap v7818: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:35 smithi078 ceph-mon[91136]: pgmap v7818: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:35 smithi146 ceph-mon[103278]: pgmap v7818: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:36 smithi146 ceph-mon[103278]: pgmap v7819: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:36 smithi078 ceph-mon[91136]: pgmap v7819: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:36 smithi078 ceph-mon[99959]: pgmap v7819: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:38 smithi146 ceph-mon[103278]: pgmap v7820: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:38 smithi078 ceph-mon[91136]: pgmap v7820: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:38 smithi078 ceph-mon[99959]: pgmap v7820: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:40.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:40 smithi146 ceph-mon[103278]: pgmap v7821: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:40.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:40 smithi078 ceph-mon[99959]: pgmap v7821: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:40.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:40 smithi078 ceph-mon[91136]: pgmap v7821: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:40:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:40:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:41.229Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:41.229Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:41.230Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:42.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:42 smithi078 ceph-mon[91136]: pgmap v7822: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:42.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:42 smithi078 ceph-mon[99959]: pgmap v7822: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:42 smithi146 ceph-mon[103278]: pgmap v7822: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:43.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:44 smithi146 ceph-mon[103278]: pgmap v7823: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:44 smithi078 ceph-mon[99959]: pgmap v7823: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:44 smithi078 ceph-mon[91136]: pgmap v7823: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:46.495 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:46 smithi078 ceph-mon[99959]: pgmap v7824: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:46.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:46 smithi078 ceph-mon[91136]: pgmap v7824: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:46 smithi146 ceph-mon[103278]: pgmap v7824: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:48 smithi146 ceph-mon[103278]: pgmap v7825: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:48 smithi078 ceph-mon[91136]: pgmap v7825: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:48 smithi078 ceph-mon[99959]: pgmap v7825: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:40:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:40:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:40:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:51 smithi146 ceph-mon[103278]: pgmap v7826: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[99959]: pgmap v7826: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[91136]: pgmap v7826: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:40:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:51.230Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:51.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:40:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:40:51.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:40:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:53 smithi146 ceph-mon[103278]: pgmap v7827: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:53.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:53 smithi078 ceph-mon[99959]: pgmap v7827: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:53 smithi078 ceph-mon[91136]: pgmap v7827: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:40:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:40:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:55 smithi146 ceph-mon[103278]: pgmap v7828: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:40:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:55 smithi078 ceph-mon[91136]: pgmap v7828: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:40:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:55 smithi078 ceph-mon[99959]: pgmap v7828: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:40:55.971 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:40:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:56 smithi078 ceph-mon[99959]: from='client.61791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:56 smithi078 ceph-mon[91136]: from='client.61791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 4h - - 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 10s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 10s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 10s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 10s ago 4h 447M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 5s ago 4h 293M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:40:56.499 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 10s ago 4h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 10s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 10s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 10s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 10s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 10s ago 4h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 5s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 5s ago 4h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 4h 85.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 4h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 73.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 10s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 10s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:40:56.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:40:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:56 smithi146 ceph-mon[103278]: from='client.61791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:56.809 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:40:56.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:40:56.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:40:56.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:56.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:40:56.810 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:40:57.076 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:40:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:40:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:40:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:40:57.077 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:40:57.077 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:40:57.387 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:40:57.387 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:40:57.387 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:57.387 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[99959]: pgmap v7829: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[99959]: from='client.81716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[99959]: from='client.61803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2343098225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[99959]: from='client.81734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[91136]: pgmap v7829: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[91136]: from='client.81716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[91136]: from='client.61803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2343098225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:57.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:57 smithi078 ceph-mon[91136]: from='client.81734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:57 smithi146 ceph-mon[103278]: pgmap v7829: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:40:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:57 smithi146 ceph-mon[103278]: from='client.81716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:57 smithi146 ceph-mon[103278]: from='client.61803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2343098225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:40:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:57 smithi146 ceph-mon[103278]: from='client.81734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:40:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3493197339' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3493197339' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3493197339' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:40:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:59 smithi146 ceph-mon[103278]: pgmap v7830: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:40:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:40:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:59 smithi078 ceph-mon[99959]: pgmap v7830: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:40:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:40:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:40:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:59 smithi078 ceph-mon[91136]: pgmap v7830: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:40:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:40:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:01.027 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:41:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:41:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:01 smithi078 ceph-mon[99959]: pgmap v7831: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:41:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:01 smithi078 ceph-mon[91136]: pgmap v7831: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:41:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:01.231Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:01.231Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:01.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:01.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:01.478 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:01 smithi146 ceph-mon[103278]: pgmap v7831: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T00:41:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:03 smithi146 ceph-mon[103278]: pgmap v7832: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:41:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:03 smithi078 ceph-mon[91136]: pgmap v7832: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:41:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:03 smithi078 ceph-mon[99959]: pgmap v7832: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T00:41:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:05 smithi146 ceph-mon[103278]: pgmap v7833: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:41:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:05 smithi078 ceph-mon[91136]: pgmap v7833: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:41:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:05 smithi078 ceph-mon[99959]: pgmap v7833: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T00:41:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:07 smithi146 ceph-mon[103278]: pgmap v7834: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T00:41:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[91136]: pgmap v7834: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[99959]: pgmap v7834: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:09 smithi146 ceph-mon[103278]: pgmap v7835: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:41:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:09 smithi078 ceph-mon[91136]: pgmap v7835: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:41:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:09 smithi078 ceph-mon[99959]: pgmap v7835: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T00:41:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:41:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:41:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:11.232Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:11.232Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:11.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:11 smithi146 ceph-mon[103278]: pgmap v7836: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:41:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[91136]: pgmap v7836: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[99959]: pgmap v7836: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:13 smithi146 ceph-mon[103278]: pgmap v7837: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:41:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:13 smithi078 ceph-mon[91136]: pgmap v7837: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:41:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:13 smithi078 ceph-mon[99959]: pgmap v7837: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:41:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:15 smithi146 ceph-mon[103278]: pgmap v7838: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:41:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:15 smithi078 ceph-mon[91136]: pgmap v7838: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:41:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:15 smithi078 ceph-mon[99959]: pgmap v7838: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T00:41:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:17 smithi146 ceph-mon[103278]: pgmap v7839: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:41:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:17 smithi078 ceph-mon[99959]: pgmap v7839: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:41:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:17 smithi078 ceph-mon[91136]: pgmap v7839: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T00:41:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:19 smithi146 ceph-mon[103278]: pgmap v7840: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T00:41:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:19 smithi078 ceph-mon[99959]: pgmap v7840: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T00:41:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:19 smithi078 ceph-mon[91136]: pgmap v7840: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T00:41:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:41:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:41:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:21.233Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:21.233Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:21.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:21.856 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:21 smithi078 ceph-mon[99959]: pgmap v7841: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:41:21.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:21 smithi078 ceph-mon[91136]: pgmap v7841: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:41:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:21 smithi146 ceph-mon[103278]: pgmap v7841: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T00:41:23.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:23 smithi078 ceph-mon[91136]: pgmap v7842: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:41:23.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:23.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:23 smithi078 ceph-mon[99959]: pgmap v7842: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:41:23.938 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:23 smithi146 ceph-mon[103278]: pgmap v7842: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T00:41:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:25 smithi146 ceph-mon[103278]: pgmap v7843: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:41:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:25 smithi078 ceph-mon[99959]: pgmap v7843: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:41:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:25 smithi078 ceph-mon[91136]: pgmap v7843: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:41:27.682 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:41:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:27 smithi146 ceph-mon[103278]: pgmap v7844: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-21T00:41:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[91136]: pgmap v7844: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-21T00:41:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[99959]: pgmap v7844: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 80 op/s 2024-09-21T00:41:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:28.207 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:41:28.207 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 4h - - 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 4h - - 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 1s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 16s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 1s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 16s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 1s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 1s ago 4h 448M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 16s ago 4h 294M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 1s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 1s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 16s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 1s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 1s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:41:28.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 1s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 1s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 16s ago 4h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 16s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 16s ago 4h 86.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 16s ago 4h 82.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 16s ago 4h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 1s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 1s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:41:28.209 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:41:28.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:41:28.521 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[99959]: from='client.61827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[99959]: from='client.81752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2397880686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[91136]: from='client.61827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[91136]: from='client.81752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2397880686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:41:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:41:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:41:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:41:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:41:28.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:41:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:28 smithi146 ceph-mon[103278]: from='client.61827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:28 smithi146 ceph-mon[103278]: from='client.81752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2397880686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:41:29.100 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:41:29.101 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:41:29.101 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:41:29.101 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:41:29.101 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:41:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:29 smithi146 ceph-mon[103278]: pgmap v7845: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:41:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:29 smithi146 ceph-mon[103278]: from='client.81758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:29 smithi146 ceph-mon[103278]: from='client.61851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/240781977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[99959]: pgmap v7845: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[99959]: from='client.81758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[99959]: from='client.61851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/240781977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[91136]: pgmap v7845: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[91136]: from='client.81758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[91136]: from='client.61851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:41:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/240781977' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:41:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:41:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:41:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:31.233Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:31.234Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:31.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:31 smithi146 ceph-mon[103278]: pgmap v7846: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[91136]: pgmap v7846: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[99959]: pgmap v7846: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:33 smithi146 ceph-mon[103278]: pgmap v7847: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T00:41:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:33 smithi078 ceph-mon[99959]: pgmap v7847: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T00:41:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:33 smithi078 ceph-mon[91136]: pgmap v7847: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T00:41:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:35 smithi146 ceph-mon[103278]: pgmap v7848: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:35 smithi078 ceph-mon[91136]: pgmap v7848: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:35 smithi078 ceph-mon[99959]: pgmap v7848: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:37 smithi146 ceph-mon[103278]: pgmap v7849: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:38.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:37 smithi078 ceph-mon[99959]: pgmap v7849: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:38.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:37 smithi078 ceph-mon[91136]: pgmap v7849: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:39 smithi146 ceph-mon[103278]: pgmap v7850: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:39 smithi078 ceph-mon[91136]: pgmap v7850: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:39 smithi078 ceph-mon[99959]: pgmap v7850: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:41:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:41:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:41.234Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:41.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:41.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:41 smithi146 ceph-mon[103278]: pgmap v7851: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:42.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:41 smithi078 ceph-mon[91136]: pgmap v7851: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:41 smithi078 ceph-mon[99959]: pgmap v7851: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:43.996 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:43 smithi078 ceph-mon[91136]: pgmap v7852: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:43.996 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:43.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:43.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:43 smithi078 ceph-mon[99959]: pgmap v7852: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:43.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:43.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:43 smithi146 ceph-mon[103278]: pgmap v7852: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:45 smithi078 ceph-mon[91136]: pgmap v7853: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:45 smithi078 ceph-mon[99959]: pgmap v7853: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:45 smithi146 ceph-mon[103278]: pgmap v7853: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:47 smithi146 ceph-mon[103278]: pgmap v7854: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[91136]: pgmap v7854: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[99959]: pgmap v7854: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:49 smithi146 ceph-mon[103278]: pgmap v7855: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:49 smithi078 ceph-mon[91136]: pgmap v7855: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:49 smithi078 ceph-mon[99959]: pgmap v7855: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:41:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:41:51.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:51.235Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:51.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:51.235Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:41:51.484 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:41:51.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:41:52.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[99959]: pgmap v7856: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:52.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.213 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[91136]: pgmap v7856: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:52.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:51 smithi146 ceph-mon[103278]: pgmap v7856: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:41:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:53 smithi146 ceph-mon[103278]: pgmap v7857: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:53 smithi078 ceph-mon[99959]: pgmap v7857: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:53 smithi078 ceph-mon[91136]: pgmap v7857: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:41:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:55 smithi146 ceph-mon[103278]: pgmap v7858: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:55 smithi078 ceph-mon[91136]: pgmap v7858: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:55 smithi078 ceph-mon[99959]: pgmap v7858: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:41:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:57 smithi146 ceph-mon[103278]: pgmap v7859: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:57 smithi078 ceph-mon[91136]: pgmap v7859: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:57 smithi078 ceph-mon[99959]: pgmap v7859: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:41:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:41:59.398 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 4h - - 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 4h - - 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 13s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 8s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 13s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 8s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 13s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 13s ago 4h 449M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:41:59.924 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 8s ago 4h 295M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 13s ago 4h 304M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 13s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 8s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 13s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 13s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 13s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 13s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 8s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 8s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 8s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 8s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 8s ago 4h 75.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 13s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:41:59.925 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:41:59.926 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 13s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:41:59.926 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:42:00.232 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:41:59 smithi078 ceph-mon[91136]: pgmap v7860: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:00.233 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:41:59 smithi078 ceph-mon[99959]: pgmap v7860: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:42:00.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:42:00.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:42:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:41:59 smithi146 ceph-mon[103278]: pgmap v7860: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:42:00.500 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:42:00.501 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:42:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:42:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:42:00.807 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:42:00.807 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:00.807 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:01.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[91136]: from='client.81782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[91136]: from='client.81788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[91136]: from='client.61875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1670551693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:01.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4156051615' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:01.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[99959]: from='client.81782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[99959]: from='client.81788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[99959]: from='client.61875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1670551693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:01.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4156051615' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:00 smithi146 ceph-mon[103278]: from='client.81782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:00 smithi146 ceph-mon[103278]: from='client.81788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:00 smithi146 ceph-mon[103278]: from='client.61875 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1670551693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4156051615' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:01.236Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:01.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:01.236Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-mon[91136]: pgmap v7861: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-mon[91136]: from='client.81806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-mon[99959]: pgmap v7861: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:01 smithi078 ceph-mon[99959]: from='client.81806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:01 smithi146 ceph-mon[103278]: pgmap v7861: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:01 smithi146 ceph-mon[103278]: from='client.81806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:03 smithi078 ceph-mon[99959]: pgmap v7862: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:03 smithi078 ceph-mon[91136]: pgmap v7862: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:03 smithi146 ceph-mon[103278]: pgmap v7862: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:06.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:05 smithi078 ceph-mon[91136]: pgmap v7863: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:06.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:05 smithi078 ceph-mon[99959]: pgmap v7863: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:05 smithi146 ceph-mon[103278]: pgmap v7863: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:07 smithi146 ceph-mon[103278]: pgmap v7864: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[91136]: pgmap v7864: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[99959]: pgmap v7864: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:09 smithi146 ceph-mon[103278]: pgmap v7865: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:09 smithi078 ceph-mon[91136]: pgmap v7865: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:09 smithi078 ceph-mon[99959]: pgmap v7865: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:42:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:42:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:11.236Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:11.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:11.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:11 smithi146 ceph-mon[103278]: pgmap v7866: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[99959]: pgmap v7866: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[91136]: pgmap v7866: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:13 smithi146 ceph-mon[103278]: pgmap v7867: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:13 smithi078 ceph-mon[91136]: pgmap v7867: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:13 smithi078 ceph-mon[99959]: pgmap v7867: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:15 smithi146 ceph-mon[103278]: pgmap v7868: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:15 smithi078 ceph-mon[91136]: pgmap v7868: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:15 smithi078 ceph-mon[99959]: pgmap v7868: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:17 smithi146 ceph-mon[103278]: pgmap v7869: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:17 smithi078 ceph-mon[99959]: pgmap v7869: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:17 smithi078 ceph-mon[91136]: pgmap v7869: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:20 smithi146 ceph-mon[103278]: pgmap v7870: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:20 smithi078 ceph-mon[91136]: pgmap v7870: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:20 smithi078 ceph-mon[99959]: pgmap v7870: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:42:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:42:21.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:21.237Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:21.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:21.237Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:21.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:21.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:22.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:22 smithi146 ceph-mon[103278]: pgmap v7871: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:22.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:22 smithi078 ceph-mon[91136]: pgmap v7871: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:22 smithi078 ceph-mon[99959]: pgmap v7871: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:24 smithi078 ceph-mon[91136]: pgmap v7872: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:24.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:24 smithi078 ceph-mon[99959]: pgmap v7872: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:24 smithi146 ceph-mon[103278]: pgmap v7872: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:26 smithi146 ceph-mon[103278]: pgmap v7873: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:26 smithi078 ceph-mon[91136]: pgmap v7873: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:26 smithi078 ceph-mon[99959]: pgmap v7873: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:28 smithi146 ceph-mon[103278]: pgmap v7874: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[99959]: pgmap v7874: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[91136]: pgmap v7874: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:30 smithi146 ceph-mon[103278]: pgmap v7875: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:30 smithi078 ceph-mon[99959]: pgmap v7875: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:30 smithi078 ceph-mon[91136]: pgmap v7875: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:42:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:42:31.103 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:42:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:31.237Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:31.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:31.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 4h - - 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 4h - - 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 4s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 4s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 4s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:42:31.631 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 4s ago 4h 450M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 4h 296M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 4s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 4s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 4s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 4s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 4s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 4s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:42:31.632 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 19s ago 4h 75.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:42:31.633 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 4s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:42:31.633 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:42:31.633 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 4s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:42:31.633 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:42:31.955 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:42:31.955 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:42:31.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:42:31.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:42:31.956 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:42:32.227 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:42:32.228 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[91136]: pgmap v7876: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[91136]: from='client.61899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/334841145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[99959]: pgmap v7876: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:32.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[99959]: from='client.61899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/334841145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:32 smithi146 ceph-mon[103278]: pgmap v7876: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:32 smithi146 ceph-mon[103278]: from='client.61899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/334841145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:42:32.539 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:42:32.540 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:42:32.540 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:32.540 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:42:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:33 smithi146 ceph-mon[103278]: from='client.61905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:33 smithi146 ceph-mon[103278]: from='client.61911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2353090813' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:33 smithi078 ceph-mon[91136]: from='client.61905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:33 smithi078 ceph-mon[91136]: from='client.61911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2353090813' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:33 smithi078 ceph-mon[99959]: from='client.61905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:33 smithi078 ceph-mon[99959]: from='client.61911 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2353090813' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:42:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:34 smithi146 ceph-mon[103278]: pgmap v7877: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:34 smithi146 ceph-mon[103278]: from='client.81842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:34 smithi078 ceph-mon[91136]: pgmap v7877: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:34 smithi078 ceph-mon[91136]: from='client.81842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:34 smithi078 ceph-mon[99959]: pgmap v7877: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:34 smithi078 ceph-mon[99959]: from='client.81842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:36 smithi146 ceph-mon[103278]: pgmap v7878: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:36 smithi078 ceph-mon[91136]: pgmap v7878: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:36 smithi078 ceph-mon[99959]: pgmap v7878: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:38 smithi146 ceph-mon[103278]: pgmap v7879: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:38 smithi078 ceph-mon[91136]: pgmap v7879: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:38 smithi078 ceph-mon[99959]: pgmap v7879: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:40 smithi146 ceph-mon[103278]: pgmap v7880: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:40 smithi078 ceph-mon[91136]: pgmap v7880: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:40 smithi078 ceph-mon[99959]: pgmap v7880: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:42:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:42:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:41.238Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:41.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:41.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:42 smithi146 ceph-mon[103278]: pgmap v7881: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:42 smithi078 ceph-mon[91136]: pgmap v7881: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:42 smithi078 ceph-mon[99959]: pgmap v7881: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:44 smithi146 ceph-mon[103278]: pgmap v7882: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:44.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:44 smithi078 ceph-mon[99959]: pgmap v7882: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:44 smithi078 ceph-mon[91136]: pgmap v7882: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:46.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:46 smithi078 ceph-mon[91136]: pgmap v7883: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:46.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:46 smithi078 ceph-mon[99959]: pgmap v7883: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:46 smithi146 ceph-mon[103278]: pgmap v7883: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:48 smithi146 ceph-mon[103278]: pgmap v7884: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[91136]: pgmap v7884: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[99959]: pgmap v7884: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:50 smithi146 ceph-mon[103278]: pgmap v7885: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:50 smithi078 ceph-mon[91136]: pgmap v7885: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:50 smithi078 ceph-mon[99959]: pgmap v7885: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:42:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T00:42:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:51.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:51.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:42:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:42:51.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:42:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:52 smithi146 ceph-mon[103278]: pgmap v7886: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[99959]: pgmap v7886: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[91136]: pgmap v7886: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:42:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:54 smithi146 ceph-mon[103278]: pgmap v7887: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:54 smithi078 ceph-mon[91136]: pgmap v7887: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:54 smithi078 ceph-mon[99959]: pgmap v7887: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:42:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:55 smithi146 ceph-mon[103278]: pgmap v7888: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:55 smithi078 ceph-mon[91136]: pgmap v7888: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:55 smithi078 ceph-mon[99959]: pgmap v7888: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:57 smithi146 ceph-mon[103278]: pgmap v7889: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:57.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:57 smithi078 ceph-mon[99959]: pgmap v7889: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:57 smithi078 ceph-mon[91136]: pgmap v7889: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:42:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:59 smithi146 ceph-mon[103278]: pgmap v7890: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:59 smithi078 ceph-mon[91136]: pgmap v7890: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:42:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:59 smithi078 ceph-mon[99959]: pgmap v7890: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:42:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:43:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:43:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:01 smithi146 ceph-mon[103278]: pgmap v7891: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:01 smithi078 ceph-mon[91136]: pgmap v7891: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:01 smithi078 ceph-mon[99959]: pgmap v7891: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:01.240Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:01.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:01.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:02.835 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:43:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:03 smithi146 ceph-mon[103278]: pgmap v7892: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:03 smithi146 ceph-mon[103278]: from='client.61935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:03.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:03 smithi146 ceph-mon[103278]: from='client.81860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:43:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 4h - - 2024-09-21T00:43:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 4h - - 2024-09-21T00:43:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 16s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 11s ago 4h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 16s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 11s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 16s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 16s ago 4h 451M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 11s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 16s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 16s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 11s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 16s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 16s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 16s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 16s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:43:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 11s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 11s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 11s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 11s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 11s ago 4h 76.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 16s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 16s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:43:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:43:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[99959]: pgmap v7892: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[99959]: from='client.61935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[99959]: from='client.81860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[91136]: pgmap v7892: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[91136]: from='client.61935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:03 smithi078 ceph-mon[91136]: from='client.81860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:03.671 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:03.672 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:43:03.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:43:03.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:43:03.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:43:03.938 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:43:03.939 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:43:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:43:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:43:04.246 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:43:04.246 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:43:04.246 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:43:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:04 smithi146 ceph-mon[103278]: from='client.81866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1000805152' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:04 smithi146 ceph-mon[103278]: from='client.81878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:04 smithi078 ceph-mon[91136]: from='client.81866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1000805152' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:04 smithi078 ceph-mon[91136]: from='client.81878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:04 smithi078 ceph-mon[99959]: from='client.81866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1000805152' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:04 smithi078 ceph-mon[99959]: from='client.81878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:05.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:05 smithi146 ceph-mon[103278]: pgmap v7893: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:05.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2686284409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:05 smithi078 ceph-mon[99959]: pgmap v7893: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2686284409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:05 smithi078 ceph-mon[91136]: pgmap v7893: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2686284409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:07 smithi078 ceph-mon[99959]: pgmap v7894: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:07 smithi078 ceph-mon[91136]: pgmap v7894: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:07 smithi146 ceph-mon[103278]: pgmap v7894: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:09 smithi146 ceph-mon[103278]: pgmap v7895: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:09 smithi078 ceph-mon[91136]: pgmap v7895: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:09 smithi078 ceph-mon[99959]: pgmap v7895: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:43:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:43:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:11 smithi078 ceph-mon[91136]: pgmap v7896: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:11 smithi078 ceph-mon[99959]: pgmap v7896: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:11.240Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:11.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:11.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:11 smithi146 ceph-mon[103278]: pgmap v7896: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:14.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:14 smithi146 ceph-mon[103278]: pgmap v7897: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:14.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:14.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:14 smithi078 ceph-mon[91136]: pgmap v7897: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:14 smithi078 ceph-mon[99959]: pgmap v7897: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:16 smithi146 ceph-mon[103278]: pgmap v7898: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:16 smithi078 ceph-mon[91136]: pgmap v7898: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:16 smithi078 ceph-mon[99959]: pgmap v7898: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:18 smithi146 ceph-mon[103278]: pgmap v7899: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:18 smithi078 ceph-mon[91136]: pgmap v7899: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:18 smithi078 ceph-mon[99959]: pgmap v7899: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:20 smithi146 ceph-mon[103278]: pgmap v7900: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:20 smithi078 ceph-mon[99959]: pgmap v7900: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:20 smithi078 ceph-mon[91136]: pgmap v7900: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:43:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:43:21.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:21.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:21.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:21.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:21.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:21.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:22 smithi146 ceph-mon[103278]: pgmap v7901: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:22 smithi078 ceph-mon[91136]: pgmap v7901: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:22 smithi078 ceph-mon[99959]: pgmap v7901: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:24 smithi146 ceph-mon[103278]: pgmap v7902: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:24 smithi078 ceph-mon[91136]: pgmap v7902: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:24 smithi078 ceph-mon[99959]: pgmap v7902: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:26.352 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:26 smithi078 ceph-mon[99959]: pgmap v7903: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:26.352 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:26 smithi078 ceph-mon[91136]: pgmap v7903: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:26 smithi146 ceph-mon[103278]: pgmap v7903: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:28.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[99959]: pgmap v7904: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:28.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[91136]: pgmap v7904: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:28.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:28 smithi146 ceph-mon[103278]: pgmap v7904: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:30 smithi146 ceph-mon[103278]: pgmap v7905: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:30 smithi078 ceph-mon[91136]: pgmap v7905: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:30 smithi078 ceph-mon[99959]: pgmap v7905: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:43:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:43:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:31.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:31.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:31.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:32 smithi146 ceph-mon[103278]: pgmap v7906: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:32 smithi078 ceph-mon[91136]: pgmap v7906: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:32 smithi078 ceph-mon[99959]: pgmap v7906: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:33 smithi146 ceph-mon[103278]: pgmap v7907: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[91136]: pgmap v7907: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[99959]: pgmap v7907: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:34.544 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 4h - - 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 4h - - 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 7s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 2s ago 4h 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 7s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 2s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 7s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 7s ago 4h 451M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 2s ago 4h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 7s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:43:35.068 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 7s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 2s ago 4h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 7s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 7s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 7s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 7s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 2s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 2s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 2s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 2s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 2s ago 4h 76.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 7s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 7s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:43:35.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:43:35.378 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:43:35.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:43:35.380 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:43:35.380 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[99959]: pgmap v7908: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[99959]: from='client.61971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[99959]: from='client.61977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[99959]: from='client.81902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[91136]: pgmap v7908: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[91136]: from='client.61971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[91136]: from='client.61977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:35 smithi078 ceph-mon[91136]: from='client.81902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:35 smithi146 ceph-mon[103278]: pgmap v7908: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:35 smithi146 ceph-mon[103278]: from='client.61971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:35 smithi146 ceph-mon[103278]: from='client.61977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:35 smithi146 ceph-mon[103278]: from='client.81902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:35.649 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:43:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:43:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:43:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:43:35.649 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:43:35.650 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:43:35.959 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:43:35.959 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:43:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:43:35.959 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:43:35.959 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:43:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3654885789' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:36 smithi146 ceph-mon[103278]: from='client.81914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2384374247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3654885789' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:36 smithi078 ceph-mon[99959]: from='client.81914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2384374247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3654885789' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:43:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:36 smithi078 ceph-mon[91136]: from='client.81914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:43:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2384374247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:43:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:37 smithi146 ceph-mon[103278]: pgmap v7909: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:37 smithi078 ceph-mon[91136]: pgmap v7909: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:37 smithi078 ceph-mon[99959]: pgmap v7909: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:39 smithi146 ceph-mon[103278]: pgmap v7910: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:39 smithi078 ceph-mon[91136]: pgmap v7910: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:39.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:39 smithi078 ceph-mon[99959]: pgmap v7910: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:43:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:43:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:41 smithi078 ceph-mon[99959]: pgmap v7911: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:41 smithi078 ceph-mon[91136]: pgmap v7911: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:41.243Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:41.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:41.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:41 smithi146 ceph-mon[103278]: pgmap v7911: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:43 smithi078 ceph-mon[91136]: pgmap v7912: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:43 smithi078 ceph-mon[99959]: pgmap v7912: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:43 smithi146 ceph-mon[103278]: pgmap v7912: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:45.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:45 smithi146 ceph-mon[103278]: pgmap v7913: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:45 smithi078 ceph-mon[99959]: pgmap v7913: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:45 smithi078 ceph-mon[91136]: pgmap v7913: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:47 smithi146 ceph-mon[103278]: pgmap v7914: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:47.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:47 smithi078 ceph-mon[99959]: pgmap v7914: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:47 smithi078 ceph-mon[91136]: pgmap v7914: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:49 smithi146 ceph-mon[103278]: pgmap v7915: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:49 smithi078 ceph-mon[99959]: pgmap v7915: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:49 smithi078 ceph-mon[91136]: pgmap v7915: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:43:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:43:51.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:51.243Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:51.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:43:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:43:51.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:43:51.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:51 smithi078 ceph-mon[91136]: pgmap v7916: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:51.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:51 smithi078 ceph-mon[99959]: pgmap v7916: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:51 smithi146 ceph-mon[103278]: pgmap v7916: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:53 smithi146 ceph-mon[103278]: pgmap v7917: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[99959]: pgmap v7917: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[91136]: pgmap v7917: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:43:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:43:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:55 smithi146 ceph-mon[103278]: pgmap v7918: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:55 smithi078 ceph-mon[99959]: pgmap v7918: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:55 smithi078 ceph-mon[91136]: pgmap v7918: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:57 smithi146 ceph-mon[103278]: pgmap v7919: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:57 smithi078 ceph-mon[99959]: pgmap v7919: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:57 smithi078 ceph-mon[91136]: pgmap v7919: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:43:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:59 smithi146 ceph-mon[103278]: pgmap v7920: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:43:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:59 smithi078 ceph-mon[99959]: pgmap v7920: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:43:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:43:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:59 smithi078 ceph-mon[91136]: pgmap v7920: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:43:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:43:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:44:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T00:44:01.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:01 smithi078 ceph-mon[99959]: pgmap v7921: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:01 smithi078 ceph-mon[91136]: pgmap v7921: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:01.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:01.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:01.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:01 smithi146 ceph-mon[103278]: pgmap v7921: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:03 smithi146 ceph-mon[103278]: pgmap v7922: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:03 smithi078 ceph-mon[91136]: pgmap v7922: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:03 smithi078 ceph-mon[99959]: pgmap v7922: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:05 smithi146 ceph-mon[103278]: pgmap v7923: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:05.800 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:05 smithi078 ceph-mon[99959]: pgmap v7923: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:05.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:05 smithi078 ceph-mon[91136]: pgmap v7923: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:06.255 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 4h - - 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 4h - - 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 19s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 14s ago 4h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 19s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 14s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:44:06.779 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 19s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 19s ago 4h 452M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 14s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 19s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 19s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 14s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 19s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 19s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 19s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 19s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 14s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 14s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 14s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 14s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:44:06.780 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 14s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:44:06.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 19s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:44:06.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:44:06.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 19s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:44:06.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:44:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:44:07.089 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:44:07.360 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:44:07.361 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:44:07.361 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:44:07.361 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:44:07.691 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:44:07.691 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:07.691 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[91136]: pgmap v7924: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[91136]: from='client.81926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[91136]: from='client.81932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[91136]: from='client.62019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/296864776' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[99959]: pgmap v7924: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[99959]: from='client.81926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[99959]: from='client.81932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[99959]: from='client.62019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/296864776' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:07 smithi146 ceph-mon[103278]: pgmap v7924: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:07 smithi146 ceph-mon[103278]: from='client.81926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:07 smithi146 ceph-mon[103278]: from='client.81932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:07 smithi146 ceph-mon[103278]: from='client.62019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/296864776' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:08 smithi146 ceph-mon[103278]: from='client.62031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2760091077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[91136]: from='client.62031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:08.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2760091077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:08.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[99959]: from='client.62031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:08.777 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2760091077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:08.777 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.777 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:08.777 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:09 smithi078 ceph-mon[91136]: pgmap v7925: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:09 smithi078 ceph-mon[99959]: pgmap v7925: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:09 smithi146 ceph-mon[103278]: pgmap v7925: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:44:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T00:44:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:11.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:11.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:11.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:11 smithi146 ceph-mon[103278]: pgmap v7926: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:11 smithi078 ceph-mon[99959]: pgmap v7926: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:11 smithi078 ceph-mon[91136]: pgmap v7926: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:13 smithi146 ceph-mon[103278]: pgmap v7927: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:13.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[99959]: pgmap v7927: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:13.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[91136]: pgmap v7927: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:15 smithi146 ceph-mon[103278]: pgmap v7928: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:15 smithi078 ceph-mon[99959]: pgmap v7928: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:15 smithi078 ceph-mon[91136]: pgmap v7928: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:17.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:17 smithi146 ceph-mon[103278]: pgmap v7929: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:17.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:17 smithi078 ceph-mon[91136]: pgmap v7929: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:17 smithi078 ceph-mon[99959]: pgmap v7929: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:19 smithi146 ceph-mon[103278]: pgmap v7930: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:19 smithi078 ceph-mon[91136]: pgmap v7930: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:19 smithi078 ceph-mon[99959]: pgmap v7930: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:44:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:44:21.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:21.246Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:21.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:21.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:21.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:21.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:21 smithi146 ceph-mon[103278]: pgmap v7931: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:21.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:21 smithi078 ceph-mon[91136]: pgmap v7931: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:21 smithi078 ceph-mon[99959]: pgmap v7931: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:23 smithi146 ceph-mon[103278]: pgmap v7932: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:23.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:23 smithi078 ceph-mon[91136]: pgmap v7932: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:23.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:23 smithi078 ceph-mon[99959]: pgmap v7932: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:25.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:25 smithi146 ceph-mon[103278]: pgmap v7933: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:25 smithi078 ceph-mon[91136]: pgmap v7933: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:25 smithi078 ceph-mon[99959]: pgmap v7933: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:26.191 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 00:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T00:44:25.849074603Z level=info msg="Completed cleanup jobs" duration=13.097883ms 2024-09-21T00:44:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:27 smithi146 ceph-mon[103278]: pgmap v7934: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:27 smithi078 ceph-mon[91136]: pgmap v7934: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:27 smithi078 ceph-mon[99959]: pgmap v7934: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:29.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:29 smithi078 ceph-mon[91136]: pgmap v7935: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:29.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:29 smithi078 ceph-mon[99959]: pgmap v7935: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:29 smithi146 ceph-mon[103278]: pgmap v7935: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:30.901 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:44:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:44:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:31.247Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:31.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:31.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:31 smithi078 ceph-mon[91136]: pgmap v7936: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:31 smithi078 ceph-mon[99959]: pgmap v7936: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:31 smithi146 ceph-mon[103278]: pgmap v7936: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:33 smithi146 ceph-mon[103278]: pgmap v7937: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[99959]: pgmap v7937: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[91136]: pgmap v7937: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:35 smithi146 ceph-mon[103278]: pgmap v7938: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:35 smithi078 ceph-mon[99959]: pgmap v7938: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:35 smithi078 ceph-mon[91136]: pgmap v7938: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:37 smithi146 ceph-mon[103278]: pgmap v7939: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:37.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:37 smithi078 ceph-mon[99959]: pgmap v7939: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:37.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:37 smithi078 ceph-mon[91136]: pgmap v7939: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:37.994 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:44:38.520 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 4h - - 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 10s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 10s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 10s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 10s ago 4h 452M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 5s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:44:38.521 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 10s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 10s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 10s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 10s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 10s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 10s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 5s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 5s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:44:38.522 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 10s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:44:38.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:44:38.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 10s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:44:38.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:44:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:38 smithi078 ceph-mon[99959]: from='client.81962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:38 smithi078 ceph-mon[91136]: from='client.81962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:38.830 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:44:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:44:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:44:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:44:38.831 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:44:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:38 smithi146 ceph-mon[103278]: from='client.81962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:44:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:44:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:44:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:44:39.102 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:44:39.409 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:44:39.410 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:44:39.410 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[99959]: pgmap v7940: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[99959]: from='client.62049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[99959]: from='client.81974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3150307741' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[99959]: from='client.81986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/829213121' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[91136]: pgmap v7940: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[91136]: from='client.62049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[91136]: from='client.81974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3150307741' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[91136]: from='client.81986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/829213121' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:39 smithi146 ceph-mon[103278]: pgmap v7940: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:39 smithi146 ceph-mon[103278]: from='client.62049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:39 smithi146 ceph-mon[103278]: from='client.81974 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3150307741' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:44:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:39 smithi146 ceph-mon[103278]: from='client.81986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:44:40.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/829213121' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:44:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:44:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:44:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:41.247Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:41.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:41.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:41 smithi146 ceph-mon[103278]: pgmap v7941: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:41 smithi078 ceph-mon[99959]: pgmap v7941: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:41 smithi078 ceph-mon[91136]: pgmap v7941: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:43 smithi146 ceph-mon[103278]: pgmap v7942: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:43 smithi078 ceph-mon[99959]: pgmap v7942: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:43 smithi078 ceph-mon[91136]: pgmap v7942: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:45.873 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:45 smithi146 ceph-mon[103278]: pgmap v7943: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:45 smithi078 ceph-mon[91136]: pgmap v7943: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:45 smithi078 ceph-mon[99959]: pgmap v7943: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:47.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:47 smithi146 ceph-mon[103278]: pgmap v7944: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:47 smithi078 ceph-mon[99959]: pgmap v7944: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:47 smithi078 ceph-mon[91136]: pgmap v7944: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:49.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.393 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.393 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.393 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:50.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:50 smithi078 ceph-mon[99959]: pgmap v7945: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:50.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:50 smithi078 ceph-mon[91136]: pgmap v7945: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:50 smithi146 ceph-mon[103278]: pgmap v7945: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:50.722 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:44:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:44:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:51.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:51.248Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:44:51.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:44:51.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:44:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:52 smithi078 ceph-mon[91136]: pgmap v7946: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:52 smithi078 ceph-mon[99959]: pgmap v7946: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:52 smithi146 ceph-mon[103278]: pgmap v7946: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:53 smithi146 ceph-mon[103278]: pgmap v7947: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[91136]: pgmap v7947: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[99959]: pgmap v7947: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:44:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:55 smithi146 ceph-mon[103278]: pgmap v7948: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:55 smithi078 ceph-mon[91136]: pgmap v7948: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:55 smithi078 ceph-mon[99959]: pgmap v7948: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:44:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:57 smithi146 ceph-mon[103278]: pgmap v7949: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:57 smithi078 ceph-mon[91136]: pgmap v7949: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:57 smithi078 ceph-mon[99959]: pgmap v7949: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:44:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:44:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:44:59 smithi146 ceph-mon[103278]: pgmap v7950: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:44:59 smithi078 ceph-mon[91136]: pgmap v7950: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:44:59 smithi078 ceph-mon[99959]: pgmap v7950: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:45:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:45:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:01.248Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:01.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:01.249Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:01 smithi146 ceph-mon[103278]: pgmap v7951: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:01 smithi078 ceph-mon[99959]: pgmap v7951: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:01 smithi078 ceph-mon[91136]: pgmap v7951: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:03 smithi146 ceph-mon[103278]: pgmap v7952: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:03 smithi078 ceph-mon[99959]: pgmap v7952: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:03 smithi078 ceph-mon[91136]: pgmap v7952: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:05 smithi146 ceph-mon[103278]: pgmap v7953: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:05 smithi078 ceph-mon[91136]: pgmap v7953: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:05 smithi078 ceph-mon[99959]: pgmap v7953: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:07.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:07 smithi146 ceph-mon[103278]: pgmap v7954: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:07 smithi078 ceph-mon[91136]: pgmap v7954: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:07 smithi078 ceph-mon[99959]: pgmap v7954: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:45:10.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:10 smithi078 ceph-mon[99959]: pgmap v7955: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:10.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:10 smithi078 ceph-mon[91136]: pgmap v7955: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:10.246 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:45:10.246 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 4h - - 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 4h - - 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 2s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 17s ago 4h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 2s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 17s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 2s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 2s ago 4h 453M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 17s ago 4h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 2s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 2s ago 4h 15.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 17s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 2s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 2s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 2s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:45:10.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 2s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 17s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 17s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 17s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 17s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 17s ago 4h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 2s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 2s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:45:10.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:45:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:10 smithi146 ceph-mon[103278]: pgmap v7955: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:10.561 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:45:10.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:45:10.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:45:10.563 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:45:10.836 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:45:10.837 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:45:10.837 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:45:10.837 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:45:10.837 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:45:10.837 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:45:10.837 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:45:10.837 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:45:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:45:11.154 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:45:11.154 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:11.155 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:11.155 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:11.155 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[99959]: from='client.62079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[99959]: from='client.62085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1298451068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3164511351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[91136]: from='client.62079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[91136]: from='client.62085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:11.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1298451068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:11.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3164511351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:11.410 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:11.249Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:11.410 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:11.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:11.410 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:11.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:11 smithi146 ceph-mon[103278]: from='client.62079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:11 smithi146 ceph-mon[103278]: from='client.62085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1298451068' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3164511351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:12 smithi146 ceph-mon[103278]: pgmap v7956: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:12 smithi146 ceph-mon[103278]: from='client.82010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:12 smithi146 ceph-mon[103278]: from='client.62103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:12 smithi078 ceph-mon[91136]: pgmap v7956: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:12 smithi078 ceph-mon[91136]: from='client.82010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:12 smithi078 ceph-mon[91136]: from='client.62103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:12 smithi078 ceph-mon[99959]: pgmap v7956: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:12 smithi078 ceph-mon[99959]: from='client.82010 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:12 smithi078 ceph-mon[99959]: from='client.62103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:13 smithi146 ceph-mon[103278]: pgmap v7957: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[99959]: pgmap v7957: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[91136]: pgmap v7957: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:16.091 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:15 smithi146 ceph-mon[103278]: pgmap v7958: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:15 smithi078 ceph-mon[99959]: pgmap v7958: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:15 smithi078 ceph-mon[91136]: pgmap v7958: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:17 smithi146 ceph-mon[103278]: pgmap v7959: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:17 smithi078 ceph-mon[91136]: pgmap v7959: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:17 smithi078 ceph-mon[99959]: pgmap v7959: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:19 smithi146 ceph-mon[103278]: pgmap v7960: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:19 smithi078 ceph-mon[91136]: pgmap v7960: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:19 smithi078 ceph-mon[99959]: pgmap v7960: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:45:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:45:21.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:21.250Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:21.499 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:21.250Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:21.499 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:21.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:22.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:21 smithi078 ceph-mon[91136]: pgmap v7961: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:22.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:21 smithi078 ceph-mon[99959]: pgmap v7961: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:21 smithi146 ceph-mon[103278]: pgmap v7961: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:23 smithi146 ceph-mon[103278]: pgmap v7962: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:23 smithi078 ceph-mon[99959]: pgmap v7962: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:23 smithi078 ceph-mon[91136]: pgmap v7962: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:26.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:25 smithi146 ceph-mon[103278]: pgmap v7963: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:25 smithi078 ceph-mon[99959]: pgmap v7963: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:25 smithi078 ceph-mon[91136]: pgmap v7963: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:27 smithi146 ceph-mon[103278]: pgmap v7964: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:27 smithi078 ceph-mon[91136]: pgmap v7964: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:27 smithi078 ceph-mon[99959]: pgmap v7964: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:29 smithi146 ceph-mon[103278]: pgmap v7965: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[91136]: pgmap v7965: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[99959]: pgmap v7965: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:30.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:45:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:45:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:31 smithi078 ceph-mon[99959]: pgmap v7966: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:31 smithi078 ceph-mon[91136]: pgmap v7966: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:31.251Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:31.251Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:31.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:31 smithi146 ceph-mon[103278]: pgmap v7966: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:33 smithi146 ceph-mon[103278]: pgmap v7967: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[99959]: pgmap v7967: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[91136]: pgmap v7967: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:35 smithi146 ceph-mon[103278]: pgmap v7968: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:35 smithi078 ceph-mon[91136]: pgmap v7968: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:35 smithi078 ceph-mon[99959]: pgmap v7968: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:37 smithi146 ceph-mon[103278]: pgmap v7969: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:37 smithi078 ceph-mon[91136]: pgmap v7969: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:37 smithi078 ceph-mon[99959]: pgmap v7969: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:39 smithi146 ceph-mon[103278]: pgmap v7970: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:39 smithi078 ceph-mon[91136]: pgmap v7970: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:39 smithi078 ceph-mon[99959]: pgmap v7970: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:45:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:45:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:45:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:41.252Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:41.252Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:41.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 4h - - 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 4h - - 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 13s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 9s ago 4h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 13s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 9s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 13s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 13s ago 4h 455M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 9s ago 4h 297M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 13s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:45:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 13s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 9s ago 4h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 13s ago 4h 87.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 13s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 13s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 13s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 9s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 9s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 9s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 9s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 9s ago 4h 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 13s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 13s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:45:41.979 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:45:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:41 smithi146 ceph-mon[103278]: pgmap v7971: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:41 smithi078 ceph-mon[99959]: pgmap v7971: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:41 smithi078 ceph-mon[91136]: pgmap v7971: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:42.291 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:45:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:45:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:45:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:42.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:45:42.292 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:45:42.560 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:45:42.561 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:45:42.870 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:45:42.870 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:42.870 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:45:42.871 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:42.871 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:45:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:42 smithi146 ceph-mon[103278]: from='client.82034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:42 smithi146 ceph-mon[103278]: from='client.82040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:42 smithi146 ceph-mon[103278]: from='client.62127 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1365801003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3140507397' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:43.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[91136]: from='client.82034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[91136]: from='client.82040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[91136]: from='client.62127 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1365801003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3140507397' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[99959]: from='client.82034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[99959]: from='client.82040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[99959]: from='client.62127 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1365801003' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:45:43.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3140507397' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:45:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:43 smithi146 ceph-mon[103278]: pgmap v7972: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:43 smithi146 ceph-mon[103278]: from='client.62139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[91136]: pgmap v7972: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[91136]: from='client.62139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[99959]: pgmap v7972: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[99959]: from='client.62139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:45 smithi146 ceph-mon[103278]: pgmap v7973: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:45 smithi078 ceph-mon[91136]: pgmap v7973: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:45 smithi078 ceph-mon[99959]: pgmap v7973: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:47 smithi146 ceph-mon[103278]: pgmap v7974: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:47 smithi078 ceph-mon[91136]: pgmap v7974: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:47 smithi078 ceph-mon[99959]: pgmap v7974: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:45:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:49 smithi146 ceph-mon[103278]: pgmap v7975: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[91136]: pgmap v7975: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[99959]: pgmap v7975: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:45:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:45:51.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:51 smithi078 ceph-mon[91136]: pgmap v7976: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:51 smithi078 ceph-mon[99959]: pgmap v7976: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:51.253Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:51.253Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:45:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:45:51.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:45:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:51 smithi146 ceph-mon[103278]: pgmap v7976: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[91136]: pgmap v7977: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[99959]: pgmap v7977: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.155 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:54 smithi146 ceph-mon[103278]: pgmap v7977: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:45:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:45:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:56 smithi146 ceph-mon[103278]: pgmap v7978: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:56 smithi078 ceph-mon[91136]: pgmap v7978: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:56 smithi078 ceph-mon[99959]: pgmap v7978: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:45:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:58 smithi146 ceph-mon[103278]: pgmap v7979: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:58 smithi078 ceph-mon[91136]: pgmap v7979: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:58 smithi078 ceph-mon[99959]: pgmap v7979: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:45:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:45:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:00 smithi146 ceph-mon[103278]: pgmap v7980: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:00 smithi078 ceph-mon[91136]: pgmap v7980: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:00 smithi078 ceph-mon[99959]: pgmap v7980: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:46:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:46:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:01.254Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:01.254Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:01.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:02 smithi146 ceph-mon[103278]: pgmap v7981: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:02 smithi078 ceph-mon[99959]: pgmap v7981: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:02 smithi078 ceph-mon[91136]: pgmap v7981: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:04 smithi146 ceph-mon[103278]: pgmap v7982: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:04 smithi078 ceph-mon[91136]: pgmap v7982: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:04 smithi078 ceph-mon[99959]: pgmap v7982: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:06 smithi146 ceph-mon[103278]: pgmap v7983: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:06 smithi078 ceph-mon[91136]: pgmap v7983: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:06 smithi078 ceph-mon[99959]: pgmap v7983: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:08.208 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:08 smithi146 ceph-mon[103278]: pgmap v7984: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:08 smithi078 ceph-mon[91136]: pgmap v7984: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:08 smithi078 ceph-mon[99959]: pgmap v7984: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:09.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:09 smithi146 ceph-mon[103278]: pgmap v7985: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:09.745 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:09.746 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:09.746 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[91136]: pgmap v7985: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[99959]: pgmap v7985: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:46:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:46:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:11.254Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:11.255Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:11.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:11 smithi146 ceph-mon[103278]: pgmap v7986: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:12.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:11 smithi078 ceph-mon[91136]: pgmap v7986: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:12.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:11 smithi078 ceph-mon[99959]: pgmap v7986: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:13.169 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 4h - - 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 4h - - 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 5s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 0s ago 4h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 5s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 0s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 5s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 5s ago 4h 453M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 0s ago 4h 298M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:46:13.720 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 5s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 5s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 0s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 5s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 5s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 5s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 5s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 0s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 0s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 0s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 0s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 0s ago 4h 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 5s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 0s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:46:13.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 5s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:46:13.722 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: pgmap v7987: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: from='client.82070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: pgmap v7987: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: from='client.82070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:13.781 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:13.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:13.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:13.782 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: pgmap v7987: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: from='client.82070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:14.082 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:14.083 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:46:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:46:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:46:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:46:14.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:46:14.084 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:46:14.382 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:46:14.383 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:46:14.656 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:14 smithi078 ceph-mon[99959]: from='client.62157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:14.656 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:14 smithi078 ceph-mon[99959]: from='client.82082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:14.656 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/231177726' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:14.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:14 smithi078 ceph-mon[91136]: from='client.62157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:14.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:14 smithi078 ceph-mon[91136]: from='client.82082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:14.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/231177726' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:14.726 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:46:14.726 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:14.726 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:14.727 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:14.727 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:14 smithi146 ceph-mon[103278]: from='client.62157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:14 smithi146 ceph-mon[103278]: from='client.82082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/231177726' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:15 smithi078 ceph-mon[99959]: pgmap v7988: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:15 smithi078 ceph-mon[99959]: from='client.82094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:15.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1753099226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:15 smithi078 ceph-mon[91136]: pgmap v7988: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:15 smithi078 ceph-mon[91136]: from='client.82094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:15.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1753099226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:15 smithi146 ceph-mon[103278]: pgmap v7988: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:15 smithi146 ceph-mon[103278]: from='client.82094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1753099226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:17.832 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:17 smithi146 ceph-mon[103278]: pgmap v7989: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:17 smithi078 ceph-mon[91136]: pgmap v7989: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:17 smithi078 ceph-mon[99959]: pgmap v7989: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:19 smithi146 ceph-mon[103278]: pgmap v7990: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:19 smithi078 ceph-mon[91136]: pgmap v7990: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:19 smithi078 ceph-mon[99959]: pgmap v7990: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:46:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:46:21.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:21.255Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:21.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:21.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:21 smithi146 ceph-mon[103278]: pgmap v7991: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:22.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:21 smithi078 ceph-mon[99959]: pgmap v7991: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:21 smithi078 ceph-mon[91136]: pgmap v7991: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:23 smithi146 ceph-mon[103278]: pgmap v7992: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:24.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:23 smithi078 ceph-mon[91136]: pgmap v7992: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:24.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:23 smithi078 ceph-mon[99959]: pgmap v7992: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:25 smithi146 ceph-mon[103278]: pgmap v7993: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:25 smithi078 ceph-mon[99959]: pgmap v7993: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:25 smithi078 ceph-mon[91136]: pgmap v7993: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:27 smithi146 ceph-mon[103278]: pgmap v7994: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:27 smithi078 ceph-mon[99959]: pgmap v7994: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:27 smithi078 ceph-mon[91136]: pgmap v7994: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:29.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:29 smithi146 ceph-mon[103278]: pgmap v7995: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:29.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:29.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:29.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[91136]: pgmap v7995: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[99959]: pgmap v7995: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:46:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:46:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:31.256Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:31.256Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:31.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:31 smithi146 ceph-mon[103278]: pgmap v7996: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:31 smithi078 ceph-mon[91136]: pgmap v7996: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:31 smithi078 ceph-mon[99959]: pgmap v7996: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:33 smithi146 ceph-mon[103278]: pgmap v7997: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[91136]: pgmap v7997: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[99959]: pgmap v7997: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:36.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:35 smithi146 ceph-mon[103278]: pgmap v7998: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:35 smithi078 ceph-mon[99959]: pgmap v7998: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:35 smithi078 ceph-mon[91136]: pgmap v7998: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:37 smithi146 ceph-mon[103278]: pgmap v7999: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:37 smithi078 ceph-mon[91136]: pgmap v7999: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:37 smithi078 ceph-mon[99959]: pgmap v7999: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:39 smithi146 ceph-mon[103278]: pgmap v8000: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:39 smithi078 ceph-mon[91136]: pgmap v8000: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:39 smithi078 ceph-mon[99959]: pgmap v8000: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:46:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:46:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:41.256Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:41.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:41.257Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:41 smithi146 ceph-mon[103278]: pgmap v8001: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:42.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:41 smithi078 ceph-mon[91136]: pgmap v8001: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:41 smithi078 ceph-mon[99959]: pgmap v8001: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:43 smithi146 ceph-mon[103278]: pgmap v8002: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:43 smithi078 ceph-mon[91136]: pgmap v8002: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:43 smithi078 ceph-mon[99959]: pgmap v8002: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:46:45.551 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 4h - - 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 4h - - 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 16s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 12s ago 4h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 16s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 12s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 16s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:46:45.552 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 16s ago 4h 454M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 12s ago 4h 299M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 16s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 16s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 12s ago 4h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 16s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 16s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 16s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 16s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 12s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 12s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 12s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:46:45.553 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 12s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:46:45.554 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 12s ago 4h 76.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:46:45.554 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 16s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:46:45.554 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 12s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:46:45.554 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 16s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:46:45.554 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:46:45.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:46:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:46:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:46:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:46:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:46:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:46:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:46:45.867 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:46:45.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:45 smithi078 ceph-mon[91136]: pgmap v8003: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:45.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:45 smithi078 ceph-mon[91136]: from='client.82106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:45.867 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:45 smithi078 ceph-mon[99959]: pgmap v8003: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:45.867 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:45 smithi078 ceph-mon[99959]: from='client.82106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:45 smithi146 ceph-mon[103278]: pgmap v8003: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:45 smithi146 ceph-mon[103278]: from='client.82106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:46.133 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:46:46.133 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:46:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:46:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:46:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:46:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:46:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:46:46.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:46:46.439 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:46:46.439 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:46.439 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:46:46.439 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:46.439 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:46 smithi146 ceph-mon[103278]: from='client.62193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:46 smithi146 ceph-mon[103278]: from='client.82118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1663692708' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:46 smithi146 ceph-mon[103278]: from='client.62211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/919329692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[91136]: from='client.62193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[91136]: from='client.82118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1663692708' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[91136]: from='client.62211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/919329692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[99959]: from='client.62193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[99959]: from='client.82118 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1663692708' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[99959]: from='client.62211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:46:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/919329692' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:46:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:47 smithi146 ceph-mon[103278]: pgmap v8004: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:47 smithi078 ceph-mon[91136]: pgmap v8004: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:47 smithi078 ceph-mon[99959]: pgmap v8004: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:49.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:49 smithi146 ceph-mon[103278]: pgmap v8005: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:49.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:49.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:49.982 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[91136]: pgmap v8005: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[99959]: pgmap v8005: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:46:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:46:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:51.257Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:51.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:46:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:46:51.258Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:46:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:51 smithi146 ceph-mon[103278]: pgmap v8006: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:52.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:51 smithi078 ceph-mon[91136]: pgmap v8006: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:52.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:51 smithi078 ceph-mon[99959]: pgmap v8006: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:54.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[91136]: pgmap v8007: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:54.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:54.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[99959]: pgmap v8007: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:54.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:54.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:53 smithi146 ceph-mon[103278]: pgmap v8007: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:46:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:46:56.162 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:55 smithi078 ceph-mon[91136]: pgmap v8008: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:56.162 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:55 smithi078 ceph-mon[99959]: pgmap v8008: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:55 smithi146 ceph-mon[103278]: pgmap v8008: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:46:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:57 smithi146 ceph-mon[103278]: pgmap v8009: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:57 smithi078 ceph-mon[91136]: pgmap v8009: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:57 smithi078 ceph-mon[99959]: pgmap v8009: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:46:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:46:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:46:59 smithi146 ceph-mon[103278]: pgmap v8010: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:46:59 smithi078 ceph-mon[91136]: pgmap v8010: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:46:59 smithi078 ceph-mon[99959]: pgmap v8010: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:47:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:47:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:01.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:01.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:01.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:01 smithi146 ceph-mon[103278]: pgmap v8011: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:01 smithi078 ceph-mon[91136]: pgmap v8011: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:01 smithi078 ceph-mon[99959]: pgmap v8011: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:03 smithi146 ceph-mon[103278]: pgmap v8012: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:03 smithi078 ceph-mon[99959]: pgmap v8012: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:03 smithi078 ceph-mon[91136]: pgmap v8012: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:05 smithi146 ceph-mon[103278]: pgmap v8013: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:05 smithi078 ceph-mon[91136]: pgmap v8013: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:05 smithi078 ceph-mon[99959]: pgmap v8013: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:07 smithi146 ceph-mon[103278]: pgmap v8014: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:07 smithi078 ceph-mon[91136]: pgmap v8014: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:07 smithi078 ceph-mon[99959]: pgmap v8014: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:10.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:09 smithi146 ceph-mon[103278]: pgmap v8015: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:10.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[91136]: pgmap v8015: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[99959]: pgmap v8015: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:47:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:47:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:11.258Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:11.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:11.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:11 smithi146 ceph-mon[103278]: pgmap v8016: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:11 smithi078 ceph-mon[91136]: pgmap v8016: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:11 smithi078 ceph-mon[99959]: pgmap v8016: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:13 smithi146 ceph-mon[103278]: pgmap v8017: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[91136]: pgmap v8017: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[99959]: pgmap v8017: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:16.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:15 smithi078 ceph-mon[91136]: pgmap v8018: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:16.152 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:15 smithi078 ceph-mon[99959]: pgmap v8018: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:15 smithi146 ceph-mon[103278]: pgmap v8018: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:16.728 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:47:17.259 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:47:17.259 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 4h - - 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 4h - - 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 8s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 3s ago 4h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 8s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 3s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 8s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 8s ago 4h 455M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 3s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 8s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 8s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 3s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 8s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:47:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 8s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 8s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 8s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 3s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 3s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 3s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 3s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 3s ago 4h 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 8s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 3s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 8s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:47:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:47:17.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:47:17.574 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:47:17.844 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:47:17.845 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:47:17.846 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:47:18.153 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:47:18.153 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:18.154 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:18.154 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:18.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[99959]: pgmap v8019: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:18.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[99959]: from='client.62223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:18.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[99959]: from='client.82148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:18.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1903626992' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:18.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[91136]: pgmap v8019: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:18.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[91136]: from='client.62223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:18.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[91136]: from='client.82148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:18.155 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1903626992' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:17 smithi146 ceph-mon[103278]: pgmap v8019: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:17 smithi146 ceph-mon[103278]: from='client.62223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:17 smithi146 ceph-mon[103278]: from='client.82148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1903626992' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:18 smithi146 ceph-mon[103278]: from='client.82154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:18 smithi146 ceph-mon[103278]: from='client.82166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1606431021' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:18 smithi078 ceph-mon[91136]: from='client.82154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:18 smithi078 ceph-mon[91136]: from='client.82166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1606431021' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:18 smithi078 ceph-mon[99959]: from='client.82154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:18 smithi078 ceph-mon[99959]: from='client.82166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1606431021' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:19 smithi146 ceph-mon[103278]: pgmap v8020: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:19 smithi078 ceph-mon[91136]: pgmap v8020: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:19 smithi078 ceph-mon[99959]: pgmap v8020: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:47:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:47:21.486 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:21.259Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:21.486 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:21.259Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:21.486 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:21.260Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:21 smithi146 ceph-mon[103278]: pgmap v8021: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:22.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:21 smithi078 ceph-mon[91136]: pgmap v8021: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:21 smithi078 ceph-mon[99959]: pgmap v8021: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:23 smithi146 ceph-mon[103278]: pgmap v8022: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:23 smithi078 ceph-mon[91136]: pgmap v8022: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:23 smithi078 ceph-mon[99959]: pgmap v8022: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:25 smithi146 ceph-mon[103278]: pgmap v8023: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:25 smithi078 ceph-mon[91136]: pgmap v8023: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:25 smithi078 ceph-mon[99959]: pgmap v8023: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:27 smithi146 ceph-mon[103278]: pgmap v8024: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:27 smithi078 ceph-mon[91136]: pgmap v8024: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:27 smithi078 ceph-mon[99959]: pgmap v8024: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:30.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:30 smithi146 ceph-mon[103278]: pgmap v8025: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:30.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[91136]: pgmap v8025: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[99959]: pgmap v8025: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:47:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:47:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:31.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:31.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:31.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:32 smithi146 ceph-mon[103278]: pgmap v8026: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:32 smithi078 ceph-mon[99959]: pgmap v8026: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:32 smithi078 ceph-mon[91136]: pgmap v8026: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:34 smithi146 ceph-mon[103278]: pgmap v8027: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[91136]: pgmap v8027: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[99959]: pgmap v8027: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:36.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:36 smithi078 ceph-mon[91136]: pgmap v8028: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:36.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:36 smithi078 ceph-mon[99959]: pgmap v8028: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:36 smithi146 ceph-mon[103278]: pgmap v8028: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:38 smithi146 ceph-mon[103278]: pgmap v8029: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:38 smithi078 ceph-mon[91136]: pgmap v8029: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:38 smithi078 ceph-mon[99959]: pgmap v8029: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:40 smithi146 ceph-mon[103278]: pgmap v8030: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:40 smithi078 ceph-mon[91136]: pgmap v8030: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:40 smithi078 ceph-mon[99959]: pgmap v8030: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:47:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:47:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:41.260Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:41.261Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:41.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:42 smithi146 ceph-mon[103278]: pgmap v8031: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:42 smithi078 ceph-mon[91136]: pgmap v8031: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:42 smithi078 ceph-mon[99959]: pgmap v8031: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:44 smithi146 ceph-mon[103278]: pgmap v8032: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:44 smithi078 ceph-mon[99959]: pgmap v8032: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:44 smithi078 ceph-mon[91136]: pgmap v8032: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:46 smithi146 ceph-mon[103278]: pgmap v8033: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:46 smithi078 ceph-mon[91136]: pgmap v8033: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:46 smithi078 ceph-mon[99959]: pgmap v8033: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:48.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:48 smithi078 ceph-mon[99959]: pgmap v8034: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:48.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:48 smithi078 ceph-mon[91136]: pgmap v8034: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:48.459 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:47:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:48 smithi146 ceph-mon[103278]: pgmap v8034: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:48.985 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:47:48.985 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 4h - - 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 4h - - 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 19s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 15s ago 4h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 19s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 15s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 19s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 19s ago 4h 456M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 15s ago 4h 300M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 19s ago 4h 309M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 19s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 15s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 19s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 19s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:47:48.986 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 19s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 19s ago 4h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 15s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 15s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 15s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 15s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 15s ago 4h 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 19s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 15s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:47:48.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 19s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:47:48.988 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:47:49.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:47:49.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:47:49.308 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:47:49.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:47:49.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:47:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:47:49.576 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:47:49.577 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:47:49.886 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:47:49.886 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:47:49.887 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: pgmap v8035: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='client.82178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='client.82184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='client.82190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2755424025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/814311266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: pgmap v8035: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='client.82178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='client.82184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='client.82190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2755424025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/814311266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: pgmap v8035: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='client.82178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='client.82184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='client.82190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2755424025' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:47:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/814311266' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:47:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:47:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:47:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:51 smithi146 ceph-mon[103278]: from='client.62283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:51.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:51 smithi078 ceph-mon[99959]: from='client.62283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:51 smithi078 ceph-mon[91136]: from='client.62283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:47:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:51.262Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:51.262Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:47:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:47:51.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:47:52.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:52 smithi146 ceph-mon[103278]: pgmap v8036: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:52 smithi078 ceph-mon[91136]: pgmap v8036: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:52 smithi078 ceph-mon[99959]: pgmap v8036: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:54 smithi146 ceph-mon[103278]: pgmap v8037: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[91136]: pgmap v8037: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[99959]: pgmap v8037: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:47:56.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:56 smithi078 ceph-mon[99959]: pgmap v8038: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:56.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:56 smithi078 ceph-mon[91136]: pgmap v8038: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:56 smithi146 ceph-mon[103278]: pgmap v8038: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:47:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:58 smithi078 ceph-mon[91136]: pgmap v8039: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:58 smithi078 ceph-mon[99959]: pgmap v8039: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:58 smithi146 ceph-mon[103278]: pgmap v8039: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:47:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:47:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:47:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:47:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:47:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:00 smithi146 ceph-mon[103278]: pgmap v8040: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:00 smithi078 ceph-mon[99959]: pgmap v8040: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:00 smithi078 ceph-mon[91136]: pgmap v8040: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:48:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:48:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:01.263Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:01.263Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:01.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:02.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:02 smithi146 ceph-mon[103278]: pgmap v8041: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:02 smithi078 ceph-mon[99959]: pgmap v8041: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:02 smithi078 ceph-mon[91136]: pgmap v8041: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:04 smithi146 ceph-mon[103278]: pgmap v8042: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:04 smithi078 ceph-mon[99959]: pgmap v8042: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:04 smithi078 ceph-mon[91136]: pgmap v8042: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:06 smithi146 ceph-mon[103278]: pgmap v8043: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:06 smithi078 ceph-mon[99959]: pgmap v8043: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:06 smithi078 ceph-mon[91136]: pgmap v8043: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:08.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:08 smithi146 ceph-mon[103278]: pgmap v8044: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:08.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:08 smithi078 ceph-mon[99959]: pgmap v8044: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:08.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:08 smithi078 ceph-mon[91136]: pgmap v8044: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:10 smithi146 ceph-mon[103278]: pgmap v8045: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[91136]: pgmap v8045: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[99959]: pgmap v8045: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:48:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:48:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:11.263Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:11.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:11.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:12.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:12 smithi146 ceph-mon[103278]: pgmap v8046: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:12 smithi078 ceph-mon[91136]: pgmap v8046: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:12 smithi078 ceph-mon[99959]: pgmap v8046: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:14 smithi146 ceph-mon[103278]: pgmap v8047: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[91136]: pgmap v8047: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[99959]: pgmap v8047: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:16 smithi146 ceph-mon[103278]: pgmap v8048: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:16 smithi078 ceph-mon[91136]: pgmap v8048: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:16 smithi078 ceph-mon[99959]: pgmap v8048: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:18 smithi146 ceph-mon[103278]: pgmap v8049: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:18.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:18 smithi078 ceph-mon[99959]: pgmap v8049: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:18.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:18 smithi078 ceph-mon[91136]: pgmap v8049: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:20.171 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:48:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:20 smithi146 ceph-mon[103278]: pgmap v8050: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:20 smithi078 ceph-mon[91136]: pgmap v8050: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:20 smithi078 ceph-mon[99959]: pgmap v8050: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:20.702 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:48:20.702 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 4h - - 2024-09-21T00:48:20.702 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 4h - - 2024-09-21T00:48:20.702 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 11s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:48:20.702 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 6s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 11s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 6s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 11s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 11s ago 4h 457M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 6s ago 4h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 11s ago 4h 310M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 11s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 6s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 11s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 11s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 11s ago 4h 76.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 11s ago 4h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 6s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 6s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:48:20.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 6s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:48:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 6s ago 4h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:48:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 6s ago 4h 75.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:48:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 11s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:48:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 6s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:48:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 11s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:48:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 6s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:48:21.011 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:48:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:48:21.011 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:48:21.012 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:48:21.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:48:21.013 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:48:21.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-mon[99959]: from='client.62295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:21.264 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3264750276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:21.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-mon[91136]: from='client.62295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:21.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3264750276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:21.284 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:48:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:48:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:48:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:48:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:48:21.285 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:48:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:21 smithi146 ceph-mon[103278]: from='client.62295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3264750276' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:21.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:21.264Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:21.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:21.264Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:21.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:21.598 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:48:21.598 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:21.598 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:21.598 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:21.598 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:22 smithi146 ceph-mon[103278]: pgmap v8051: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:22 smithi146 ceph-mon[103278]: from='client.62301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:22 smithi146 ceph-mon[103278]: from='client.82226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3093728227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[91136]: pgmap v8051: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[91136]: from='client.62301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[91136]: from='client.82226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3093728227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[99959]: pgmap v8051: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[99959]: from='client.62301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[99959]: from='client.82226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3093728227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:23 smithi146 ceph-mon[103278]: from='client.62319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:23 smithi078 ceph-mon[91136]: from='client.62319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:23 smithi078 ceph-mon[99959]: from='client.62319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:24 smithi146 ceph-mon[103278]: pgmap v8052: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:24 smithi078 ceph-mon[91136]: pgmap v8052: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:24 smithi078 ceph-mon[99959]: pgmap v8052: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:26 smithi146 ceph-mon[103278]: pgmap v8053: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:26 smithi078 ceph-mon[99959]: pgmap v8053: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:26 smithi078 ceph-mon[91136]: pgmap v8053: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:28 smithi146 ceph-mon[103278]: pgmap v8054: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:28 smithi078 ceph-mon[91136]: pgmap v8054: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:28 smithi078 ceph-mon[99959]: pgmap v8054: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:29.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:29.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:30 smithi146 ceph-mon[103278]: pgmap v8055: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[91136]: pgmap v8055: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[99959]: pgmap v8055: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:48:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:48:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:31 smithi146 ceph-mon[103278]: pgmap v8056: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:31 smithi078 ceph-mon[99959]: pgmap v8056: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:31 smithi078 ceph-mon[91136]: pgmap v8056: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:31.264Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:31.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:31.265Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:33 smithi146 ceph-mon[103278]: pgmap v8057: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:33 smithi078 ceph-mon[99959]: pgmap v8057: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:33 smithi078 ceph-mon[91136]: pgmap v8057: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:36 smithi146 ceph-mon[103278]: pgmap v8058: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:36 smithi078 ceph-mon[99959]: pgmap v8058: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:36 smithi078 ceph-mon[91136]: pgmap v8058: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:38.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:38 smithi078 ceph-mon[99959]: pgmap v8059: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:38.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:38 smithi078 ceph-mon[91136]: pgmap v8059: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:38 smithi146 ceph-mon[103278]: pgmap v8059: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:40 smithi146 ceph-mon[103278]: pgmap v8060: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:40 smithi078 ceph-mon[91136]: pgmap v8060: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:40 smithi078 ceph-mon[99959]: pgmap v8060: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:48:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:48:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:41.265Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:41.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:41.266Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:42 smithi146 ceph-mon[103278]: pgmap v8061: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:42 smithi078 ceph-mon[91136]: pgmap v8061: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:42 smithi078 ceph-mon[99959]: pgmap v8061: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:44 smithi146 ceph-mon[103278]: pgmap v8062: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:44 smithi078 ceph-mon[91136]: pgmap v8062: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:44 smithi078 ceph-mon[99959]: pgmap v8062: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:46 smithi146 ceph-mon[103278]: pgmap v8063: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:46 smithi078 ceph-mon[91136]: pgmap v8063: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:46 smithi078 ceph-mon[99959]: pgmap v8063: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:48 smithi146 ceph-mon[103278]: pgmap v8064: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:48 smithi078 ceph-mon[91136]: pgmap v8064: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:48 smithi078 ceph-mon[99959]: pgmap v8064: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:50 smithi146 ceph-mon[103278]: pgmap v8065: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[91136]: pgmap v8065: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[99959]: pgmap v8065: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:48:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T00:48:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:51.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:51.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:48:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:48:51.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:48:51.894 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:48:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:52 smithi078 ceph-mon[99959]: pgmap v8066: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:52 smithi078 ceph-mon[91136]: pgmap v8066: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:48:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 4h - - 2024-09-21T00:48:52.441 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 4h - - 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 2s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 18s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 2s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 18s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 2s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 2s ago 4h 458M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 18s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 2s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 2s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 18s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 2s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 2s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 2s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:48:52.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 2s ago 4h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 18s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 18s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 18s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 18s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 18s ago 4h 75.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 2s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 18s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 2s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:48:52.443 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 18s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:48:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:52 smithi146 ceph-mon[103278]: pgmap v8066: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:48:52.795 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:48:53.070 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:48:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:48:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:48:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:48:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:48:53.071 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:48:53.071 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:48:53.386 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:48:53.386 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:48:53.386 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:48:53.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:53 smithi078 ceph-mon[91136]: from='client.62331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:53.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:53 smithi078 ceph-mon[91136]: from='client.82256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:53.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1917569091' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:53.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:53 smithi078 ceph-mon[99959]: from='client.62331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:53.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:53 smithi078 ceph-mon[99959]: from='client.82256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:53.387 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1917569091' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:53.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:53 smithi146 ceph-mon[103278]: from='client.62331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:53.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:53 smithi146 ceph-mon[103278]: from='client.82256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:53.428 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1917569091' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:48:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:54 smithi146 ceph-mon[103278]: pgmap v8067: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:54 smithi146 ceph-mon[103278]: from='client.62343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:54 smithi146 ceph-mon[103278]: from='client.82274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/644497820' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[99959]: pgmap v8067: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[99959]: from='client.62343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[99959]: from='client.82274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/644497820' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[91136]: pgmap v8067: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[91136]: from='client.62343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[91136]: from='client.82274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:48:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/644497820' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:48:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:55 smithi146 ceph-mon[103278]: pgmap v8068: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[99959]: pgmap v8068: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:48:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:55 smithi078 ceph-mon[91136]: pgmap v8068: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:57 smithi146 ceph-mon[103278]: pgmap v8069: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:57 smithi078 ceph-mon[99959]: pgmap v8069: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:57 smithi078 ceph-mon[91136]: pgmap v8069: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:48:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:59 smithi078 ceph-mon[91136]: pgmap v8070: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:59 smithi078 ceph-mon[99959]: pgmap v8070: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:48:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:59 smithi146 ceph-mon[103278]: pgmap v8070: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:48:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:49:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:49:01.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:01 smithi078 ceph-mon[99959]: pgmap v8071: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:01 smithi078 ceph-mon[91136]: pgmap v8071: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:01.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:01.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:01.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:01 smithi146 ceph-mon[103278]: pgmap v8071: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:03 smithi146 ceph-mon[103278]: pgmap v8072: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:03 smithi078 ceph-mon[99959]: pgmap v8072: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:03 smithi078 ceph-mon[91136]: pgmap v8072: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:05 smithi146 ceph-mon[103278]: pgmap v8073: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:05 smithi078 ceph-mon[99959]: pgmap v8073: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:05 smithi078 ceph-mon[91136]: pgmap v8073: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:07 smithi146 ceph-mon[103278]: pgmap v8074: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:07 smithi078 ceph-mon[91136]: pgmap v8074: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:07 smithi078 ceph-mon[99959]: pgmap v8074: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:09.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:09 smithi078 ceph-mon[99959]: pgmap v8075: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:09.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:09 smithi078 ceph-mon[91136]: pgmap v8075: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:09 smithi146 ceph-mon[103278]: pgmap v8075: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:49:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:49:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:11.266Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:11.267Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:11.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:11 smithi146 ceph-mon[103278]: pgmap v8076: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:12.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:11 smithi078 ceph-mon[99959]: pgmap v8076: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:11 smithi078 ceph-mon[91136]: pgmap v8076: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:13 smithi146 ceph-mon[103278]: pgmap v8077: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:13 smithi078 ceph-mon[99959]: pgmap v8077: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:13 smithi078 ceph-mon[91136]: pgmap v8077: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:15.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:15 smithi146 ceph-mon[103278]: pgmap v8078: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:15.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.665 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[99959]: pgmap v8078: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[91136]: pgmap v8078: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:17 smithi146 ceph-mon[103278]: pgmap v8079: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:17.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:17 smithi078 ceph-mon[91136]: pgmap v8079: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:17 smithi078 ceph-mon[99959]: pgmap v8079: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:19 smithi146 ceph-mon[103278]: pgmap v8080: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:19.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:19 smithi078 ceph-mon[91136]: pgmap v8080: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:19.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:19 smithi078 ceph-mon[99959]: pgmap v8080: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:49:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:49:21.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:21 smithi078 ceph-mon[99959]: pgmap v8081: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:21 smithi078 ceph-mon[91136]: pgmap v8081: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:21.267Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:21.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:21.268Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:21 smithi146 ceph-mon[103278]: pgmap v8081: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:23.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:23 smithi078 ceph-mon[99959]: pgmap v8082: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:23.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:23.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:23 smithi078 ceph-mon[91136]: pgmap v8082: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:23.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:23.669 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:49:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:23 smithi146 ceph-mon[103278]: pgmap v8082: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:49:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 4h - - 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 4h - - 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 14s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 9s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 14s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 9s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 14s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 14s ago 4h 458M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 9s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 14s ago 4h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 14s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 9s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 14s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:49:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 14s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 14s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 14s ago 4h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 9s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 9s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 9s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 9s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 9s ago 4h 75.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 14s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 9s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 14s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:49:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 9s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:49:24.509 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:49:24.509 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:49:24.509 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:49:24.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:24.509 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:49:24.510 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:49:24.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:24 smithi078 ceph-mon[99959]: from='client.62367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:24 smithi078 ceph-mon[99959]: from='client.62373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.511 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:24 smithi078 ceph-mon[99959]: from='client.62379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:24 smithi078 ceph-mon[91136]: from='client.62367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:24 smithi078 ceph-mon[91136]: from='client.62373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:24 smithi078 ceph-mon[91136]: from='client.62379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:24 smithi146 ceph-mon[103278]: from='client.62367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:24 smithi146 ceph-mon[103278]: from='client.62373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:24 smithi146 ceph-mon[103278]: from='client.62379 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:49:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:49:24.782 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:49:25.090 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:49:25.090 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:25.090 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:25.090 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:25.090 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:25 smithi146 ceph-mon[103278]: pgmap v8083: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2227606788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:25 smithi146 ceph-mon[103278]: from='client.82310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2153843274' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[99959]: pgmap v8083: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2227606788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[99959]: from='client.82310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2153843274' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[91136]: pgmap v8083: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2227606788' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[91136]: from='client.82310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2153843274' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:27 smithi146 ceph-mon[103278]: pgmap v8084: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:27 smithi078 ceph-mon[99959]: pgmap v8084: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:27 smithi078 ceph-mon[91136]: pgmap v8084: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:29 smithi146 ceph-mon[103278]: pgmap v8085: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:29 smithi078 ceph-mon[91136]: pgmap v8085: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:29 smithi078 ceph-mon[99959]: pgmap v8085: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:30.864 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:49:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:49:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:31.268Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:31.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:31.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:31 smithi146 ceph-mon[103278]: pgmap v8086: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:31 smithi078 ceph-mon[99959]: pgmap v8086: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:31 smithi078 ceph-mon[91136]: pgmap v8086: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:34.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:33 smithi146 ceph-mon[103278]: pgmap v8087: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:34.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:33 smithi078 ceph-mon[91136]: pgmap v8087: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:33 smithi078 ceph-mon[99959]: pgmap v8087: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:35 smithi146 ceph-mon[103278]: pgmap v8088: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[99959]: pgmap v8088: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[91136]: pgmap v8088: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:37 smithi146 ceph-mon[103278]: pgmap v8089: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:37 smithi078 ceph-mon[99959]: pgmap v8089: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:37 smithi078 ceph-mon[91136]: pgmap v8089: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:39.600 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:39 smithi078 ceph-mon[91136]: pgmap v8090: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:39.601 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:39 smithi078 ceph-mon[99959]: pgmap v8090: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:39 smithi146 ceph-mon[103278]: pgmap v8090: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:40.777 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:49:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:49:41.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:41.269Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:41.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:41.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:41.496 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:41.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:41 smithi146 ceph-mon[103278]: pgmap v8091: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:41.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:41 smithi078 ceph-mon[91136]: pgmap v8091: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:41 smithi078 ceph-mon[99959]: pgmap v8091: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:43 smithi078 ceph-mon[91136]: pgmap v8092: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:43 smithi078 ceph-mon[99959]: pgmap v8092: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:43 smithi146 ceph-mon[103278]: pgmap v8092: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:45 smithi078 ceph-mon[91136]: pgmap v8093: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:45 smithi078 ceph-mon[99959]: pgmap v8093: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:45 smithi146 ceph-mon[103278]: pgmap v8093: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:47 smithi078 ceph-mon[91136]: pgmap v8094: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:47 smithi078 ceph-mon[99959]: pgmap v8094: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:47 smithi146 ceph-mon[103278]: pgmap v8094: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:49 smithi078 ceph-mon[91136]: pgmap v8095: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:49 smithi078 ceph-mon[99959]: pgmap v8095: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:49.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:49 smithi146 ceph-mon[103278]: pgmap v8095: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:50.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:49:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:49:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:51.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:51.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:51.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:49:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:49:51.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:49:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:52 smithi146 ceph-mon[103278]: pgmap v8096: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:52 smithi078 ceph-mon[99959]: pgmap v8096: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:52 smithi078 ceph-mon[91136]: pgmap v8096: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:54 smithi146 ceph-mon[103278]: pgmap v8097: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:54 smithi078 ceph-mon[99959]: pgmap v8097: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:54 smithi078 ceph-mon[91136]: pgmap v8097: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:49:55.385 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:49:55.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:55 smithi146 ceph-mon[103278]: pgmap v8098: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:55.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 4h - - 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 4h - - 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 5s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 1s ago 4h 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 5s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 1s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 5s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:49:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 5s ago 4h 459M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 1s ago 4h 301M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 5s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 5s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 1s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 5s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 5s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 5s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 5s ago 4h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 1s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 1s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 1s ago 4h 86.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 1s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:49:55.909 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 1s ago 4h 76.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:49:55.910 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 5s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:49:55.910 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 1s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:49:55.910 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 5s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:49:55.910 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:49:55.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[99959]: pgmap v8098: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:55.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[91136]: pgmap v8098: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:49:55.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:55.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:56.220 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:49:56.221 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:49:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:49:56.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:49:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[91136]: from='client.62403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[91136]: from='client.82328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[91136]: from='client.62415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1060794987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[99959]: from='client.62403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[99959]: from='client.82328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[99959]: from='client.62415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1060794987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:56.798 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:49:56.798 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:56.799 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:49:56.799 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:56.799 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:49:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:56 smithi146 ceph-mon[103278]: from='client.62403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:56 smithi146 ceph-mon[103278]: from='client.82328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:56 smithi146 ceph-mon[103278]: from='client.62415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1060794987' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:49:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:57 smithi146 ceph-mon[103278]: pgmap v8099: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:57 smithi146 ceph-mon[103278]: from='client.82346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3780760226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:57 smithi078 ceph-mon[91136]: pgmap v8099: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:57 smithi078 ceph-mon[91136]: from='client.82346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3780760226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:57 smithi078 ceph-mon[99959]: pgmap v8099: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:49:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:57 smithi078 ceph-mon[99959]: from='client.82346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:49:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3780760226' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:49:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:49:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:00.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:49:59 smithi078 ceph-mon[91136]: pgmap v8100: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:00.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:49:59 smithi078 ceph-mon[99959]: pgmap v8100: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:49:59 smithi146 ceph-mon[103278]: pgmap v8100: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:00.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T00:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:50:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:50:01.522 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:01.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:01.522 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:01.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:01.522 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:01.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:01.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:01 smithi078 ceph-mon[91136]: pgmap v8101: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:01 smithi078 ceph-mon[99959]: pgmap v8101: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:01 smithi146 ceph-mon[103278]: pgmap v8101: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:03 smithi146 ceph-mon[103278]: pgmap v8102: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:03 smithi078 ceph-mon[91136]: pgmap v8102: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:03 smithi078 ceph-mon[99959]: pgmap v8102: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:05 smithi146 ceph-mon[103278]: pgmap v8103: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:05 smithi078 ceph-mon[91136]: pgmap v8103: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:05 smithi078 ceph-mon[99959]: pgmap v8103: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:07 smithi146 ceph-mon[103278]: pgmap v8104: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:07 smithi078 ceph-mon[91136]: pgmap v8104: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:07 smithi078 ceph-mon[99959]: pgmap v8104: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:09 smithi146 ceph-mon[103278]: pgmap v8105: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:09 smithi078 ceph-mon[99959]: pgmap v8105: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:09 smithi078 ceph-mon[91136]: pgmap v8105: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:50:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:50:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:11.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:11.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:11.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:12 smithi146 ceph-mon[103278]: pgmap v8106: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:12 smithi078 ceph-mon[91136]: pgmap v8106: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:12 smithi078 ceph-mon[99959]: pgmap v8106: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:14 smithi146 ceph-mon[103278]: pgmap v8107: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:14 smithi078 ceph-mon[99959]: pgmap v8107: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:14 smithi078 ceph-mon[91136]: pgmap v8107: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[91136]: pgmap v8108: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[99959]: pgmap v8108: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:15 smithi146 ceph-mon[103278]: pgmap v8108: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:16.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:17 smithi146 ceph-mon[103278]: pgmap v8109: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:17 smithi078 ceph-mon[91136]: pgmap v8109: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:17 smithi078 ceph-mon[99959]: pgmap v8109: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:19 smithi078 ceph-mon[91136]: pgmap v8110: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:19 smithi078 ceph-mon[99959]: pgmap v8110: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:20.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:19 smithi146 ceph-mon[103278]: pgmap v8110: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:20.931 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:50:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:50:21.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:21.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:21.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:21.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:21.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:21.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:21 smithi078 ceph-mon[91136]: pgmap v8111: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:21 smithi078 ceph-mon[99959]: pgmap v8111: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:21 smithi146 ceph-mon[103278]: pgmap v8111: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:23 smithi078 ceph-mon[91136]: pgmap v8112: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:23 smithi078 ceph-mon[99959]: pgmap v8112: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:23 smithi146 ceph-mon[103278]: pgmap v8112: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:25 smithi078 ceph-mon[99959]: pgmap v8113: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:25 smithi078 ceph-mon[91136]: pgmap v8113: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:25 smithi146 ceph-mon[103278]: pgmap v8113: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:27.091 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 4h - - 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 4h - - 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 17s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 12s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 17s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 12s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 17s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 17s ago 4h 459M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 12s ago 4h 302M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 17s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 17s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:50:27.624 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 12s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 17s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 17s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 17s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 17s ago 4h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 12s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 12s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 12s ago 4h 86.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 12s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 12s ago 4h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 17s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 12s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 17s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:50:27.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 12s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:50:27.933 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:50:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:50:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:50:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:27.933 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:50:27.934 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:50:27.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:27 smithi078 ceph-mon[99959]: pgmap v8114: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:27.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:27 smithi078 ceph-mon[99959]: from='client.82358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:27.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:27 smithi078 ceph-mon[91136]: pgmap v8114: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:27.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:27 smithi078 ceph-mon[91136]: from='client.82358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:28.203 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:50:28.203 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:50:28.204 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:50:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:27 smithi146 ceph-mon[103278]: pgmap v8114: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:27 smithi146 ceph-mon[103278]: from='client.82358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:28.531 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:50:28.531 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:50:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:50:28.531 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:50:28.531 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:50:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:28 smithi146 ceph-mon[103278]: from='client.62445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:28 smithi146 ceph-mon[103278]: from='client.62451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/193920527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:28 smithi146 ceph-mon[103278]: from='client.62463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4053184351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[99959]: from='client.62445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[99959]: from='client.62451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/193920527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[99959]: from='client.62463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4053184351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[91136]: from='client.62445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[91136]: from='client.62451 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/193920527' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:50:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[91136]: from='client.62463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:50:29.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:29.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4053184351' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:50:30.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:29 smithi078 ceph-mon[91136]: pgmap v8115: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:30.265 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:29 smithi078 ceph-mon[99959]: pgmap v8115: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:29 smithi146 ceph-mon[103278]: pgmap v8115: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:50:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:50:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:31 smithi146 ceph-mon[103278]: pgmap v8116: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[91136]: pgmap v8116: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-mon[99959]: pgmap v8116: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:31.272Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:31.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:31.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:33 smithi146 ceph-mon[103278]: pgmap v8117: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:33.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:33 smithi078 ceph-mon[99959]: pgmap v8117: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:33.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:33 smithi078 ceph-mon[91136]: pgmap v8117: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:36.038 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:35 smithi146 ceph-mon[103278]: pgmap v8118: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:36.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.039 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[99959]: pgmap v8118: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[91136]: pgmap v8118: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:36.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:38.216 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:37 smithi146 ceph-mon[103278]: pgmap v8119: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:37 smithi078 ceph-mon[99959]: pgmap v8119: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:37 smithi078 ceph-mon[91136]: pgmap v8119: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:39 smithi146 ceph-mon[103278]: pgmap v8120: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:39 smithi078 ceph-mon[99959]: pgmap v8120: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:39 smithi078 ceph-mon[91136]: pgmap v8120: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:50:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:50:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:41.272Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:41.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:41.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:41 smithi146 ceph-mon[103278]: pgmap v8121: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:41 smithi078 ceph-mon[99959]: pgmap v8121: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:41 smithi078 ceph-mon[91136]: pgmap v8121: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:44.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:43 smithi078 ceph-mon[99959]: pgmap v8122: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:44.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:44.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:44.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:43 smithi078 ceph-mon[91136]: pgmap v8122: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:44.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:44.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:43 smithi146 ceph-mon[103278]: pgmap v8122: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:45 smithi146 ceph-mon[103278]: pgmap v8123: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:45 smithi078 ceph-mon[91136]: pgmap v8123: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:45 smithi078 ceph-mon[99959]: pgmap v8123: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:47 smithi146 ceph-mon[103278]: pgmap v8124: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:47 smithi078 ceph-mon[99959]: pgmap v8124: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:47 smithi078 ceph-mon[91136]: pgmap v8124: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:49 smithi146 ceph-mon[103278]: pgmap v8125: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:49 smithi078 ceph-mon[99959]: pgmap v8125: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:49 smithi078 ceph-mon[91136]: pgmap v8125: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:50:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:50:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:50:51.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[91136]: pgmap v8126: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[99959]: pgmap v8126: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:51.273Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:51.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:50:51.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:50:51.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:50:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:51 smithi146 ceph-mon[103278]: pgmap v8126: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:53 smithi146 ceph-mon[103278]: pgmap v8127: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:53 smithi078 ceph-mon[91136]: pgmap v8127: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:53 smithi078 ceph-mon[99959]: pgmap v8127: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:50:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[99959]: pgmap v8128: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[91136]: pgmap v8128: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:56 smithi146 ceph-mon[103278]: pgmap v8128: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T00:50:56.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:56.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:50:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:58 smithi078 ceph-mon[99959]: pgmap v8129: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:50:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:58 smithi078 ceph-mon[91136]: pgmap v8129: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:50:58.436 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:58 smithi146 ceph-mon[103278]: pgmap v8129: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T00:50:58.825 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:50:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:50:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:50:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:59.352 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:50:59.352 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 4h - - 2024-09-21T00:50:59.352 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 4h - - 2024-09-21T00:50:59.352 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 8s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:50:59.352 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 4s ago 4h 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 8s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 4s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 8s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 8s ago 4h 460M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 4s ago 4h 303M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 8s ago 4h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 8s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 4s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 8s ago 4h 87.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 8s ago 4h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 8s ago 4h 77.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 8s ago 4h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 4s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 4s ago 4h 80.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:50:59.353 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 4s ago 4h 86.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:50:59.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 4s ago 4h 83.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:50:59.354 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 4s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:50:59.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 8s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:50:59.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 4s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:50:59.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 8s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:50:59.354 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 4s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:50:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:50:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:50:59.679 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:50:59.679 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:50:59.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:50:59.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:59.679 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:50:59.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:50:59.680 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:50:59.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:50:59.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:51:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[99959]: pgmap v8130: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T00:51:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[99959]: from='client.62475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[99959]: from='client.62481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2030650951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[91136]: pgmap v8130: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T00:51:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[91136]: from='client.62475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[91136]: from='client.62481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2030650951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:00.314 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:51:00.314 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:00.314 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:00.314 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:00.314 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:00 smithi146 ceph-mon[103278]: pgmap v8130: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T00:51:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:00 smithi146 ceph-mon[103278]: from='client.62475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:00 smithi146 ceph-mon[103278]: from='client.62481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2030650951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:51:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:51:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:01 smithi146 ceph-mon[103278]: from='client.82406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:01 smithi146 ceph-mon[103278]: from='client.62499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/699780298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-mon[91136]: from='client.82406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-mon[91136]: from='client.62499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/699780298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-mon[99959]: from='client.82406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-mon[99959]: from='client.62499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/699780298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:01.273Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:01.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:01.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:02.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:02 smithi078 ceph-mon[91136]: pgmap v8131: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T00:51:02.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:02 smithi078 ceph-mon[99959]: pgmap v8131: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T00:51:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:02 smithi146 ceph-mon[103278]: pgmap v8131: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T00:51:04.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:04 smithi078 ceph-mon[91136]: pgmap v8132: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T00:51:04.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:04.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:04 smithi078 ceph-mon[99959]: pgmap v8132: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T00:51:04.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:04 smithi146 ceph-mon[103278]: pgmap v8132: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s 2024-09-21T00:51:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:06 smithi146 ceph-mon[103278]: pgmap v8133: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T00:51:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:06 smithi078 ceph-mon[91136]: pgmap v8133: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T00:51:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:06 smithi078 ceph-mon[99959]: pgmap v8133: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T00:51:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:08 smithi146 ceph-mon[103278]: pgmap v8134: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T00:51:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:08 smithi078 ceph-mon[91136]: pgmap v8134: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T00:51:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:08 smithi078 ceph-mon[99959]: pgmap v8134: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T00:51:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:10 smithi146 ceph-mon[103278]: pgmap v8135: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:51:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:10 smithi078 ceph-mon[91136]: pgmap v8135: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:51:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:10 smithi078 ceph-mon[99959]: pgmap v8135: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 0 B/s wr, 87 op/s 2024-09-21T00:51:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:51:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:51:11.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[99959]: pgmap v8136: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[91136]: pgmap v8136: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:11.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:11.274Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:11.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:11.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:11.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:11.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:11 smithi146 ceph-mon[103278]: pgmap v8136: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:51:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:13.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:13 smithi078 ceph-mon[99959]: pgmap v8137: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:51:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:13 smithi078 ceph-mon[91136]: pgmap v8137: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:51:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:13 smithi146 ceph-mon[103278]: pgmap v8137: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T00:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:16 smithi146 ceph-mon[103278]: pgmap v8138: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-21T00:51:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[99959]: pgmap v8138: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-21T00:51:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[91136]: pgmap v8138: 161 pgs: 161 active+clean; 585 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-21T00:51:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:18.420 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:18 smithi146 ceph-mon[103278]: pgmap v8139: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T00:51:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:18 smithi078 ceph-mon[91136]: pgmap v8139: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T00:51:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:18 smithi078 ceph-mon[99959]: pgmap v8139: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T00:51:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:20 smithi146 ceph-mon[103278]: pgmap v8140: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T00:51:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:20 smithi078 ceph-mon[91136]: pgmap v8140: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T00:51:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:20 smithi078 ceph-mon[99959]: pgmap v8140: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T00:51:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:51:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T00:51:21.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:21.275Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:21.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:21.275Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:21.493 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:21.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:22 smithi146 ceph-mon[103278]: pgmap v8141: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:51:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:22 smithi078 ceph-mon[99959]: pgmap v8141: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:51:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:22 smithi078 ceph-mon[91136]: pgmap v8141: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T00:51:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:24 smithi146 ceph-mon[103278]: pgmap v8142: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:51:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:24 smithi078 ceph-mon[91136]: pgmap v8142: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:51:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:24 smithi078 ceph-mon[99959]: pgmap v8142: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T00:51:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:26 smithi146 ceph-mon[103278]: pgmap v8143: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:51:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:26 smithi078 ceph-mon[91136]: pgmap v8143: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:51:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:26 smithi078 ceph-mon[99959]: pgmap v8143: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T00:51:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:28 smithi146 ceph-mon[103278]: pgmap v8144: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:51:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:28 smithi078 ceph-mon[91136]: pgmap v8144: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:51:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:28 smithi078 ceph-mon[99959]: pgmap v8144: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T00:51:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:30.495 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:30 smithi146 ceph-mon[103278]: pgmap v8145: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:51:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:30 smithi078 ceph-mon[91136]: pgmap v8145: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:51:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:30 smithi078 ceph-mon[99959]: pgmap v8145: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T00:51:30.615 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:51:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:51:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 4h - - 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 4h - - 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 0s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 15s ago 4h 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 0s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 15s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 0s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 0s ago 4h 461M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 15s ago 4h 304M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:51:31.147 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 0s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 0s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 15s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 0s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 0s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 0s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 0s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 15s ago 4h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 15s ago 4h 80.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 15s ago 4h 86.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 15s ago 4h 83.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 15s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 0s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 15s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 0s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:51:31.148 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 15s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:51:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:51:31.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:51:31.457 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:31.276Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:31.457 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:31.276Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:31.457 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:31.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:31.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: pgmap v8146: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:51:31.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='client.62511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:31.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:31.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='client.62517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='client.62523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1193541061' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: pgmap v8146: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='client.62511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='client.62517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='client.62523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:31.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1193541061' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:51:31.730 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:51:31.731 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:51:31.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:51:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: pgmap v8146: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T00:51:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='client.62511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='client.62517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='client.62523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1193541061' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:51:32.037 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:51:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:32.038 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:51:32.038 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:32.038 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:51:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:32 smithi146 ceph-mon[103278]: from='client.82454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3456409444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:32 smithi078 ceph-mon[91136]: from='client.82454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3456409444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:32 smithi078 ceph-mon[99959]: from='client.82454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:51:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3456409444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:51:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:33 smithi146 ceph-mon[103278]: pgmap v8147: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:51:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:33 smithi078 ceph-mon[91136]: pgmap v8147: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:51:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:33 smithi078 ceph-mon[99959]: pgmap v8147: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T00:51:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:35 smithi146 ceph-mon[103278]: pgmap v8148: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[91136]: pgmap v8148: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[99959]: pgmap v8148: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:37 smithi146 ceph-mon[103278]: pgmap v8149: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:37 smithi078 ceph-mon[91136]: pgmap v8149: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:37 smithi078 ceph-mon[99959]: pgmap v8149: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:39.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:39 smithi146 ceph-mon[103278]: pgmap v8150: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:39 smithi078 ceph-mon[91136]: pgmap v8150: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:39 smithi078 ceph-mon[99959]: pgmap v8150: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:51:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:51:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:41.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:41.277Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:41.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:41 smithi146 ceph-mon[103278]: pgmap v8151: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:41 smithi078 ceph-mon[99959]: pgmap v8151: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:41 smithi078 ceph-mon[91136]: pgmap v8151: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:43 smithi146 ceph-mon[103278]: pgmap v8152: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:44.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:43 smithi078 ceph-mon[99959]: pgmap v8152: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:44.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:44.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:43 smithi078 ceph-mon[91136]: pgmap v8152: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:44.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:46.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:45 smithi078 ceph-mon[91136]: pgmap v8153: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:46.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:45 smithi078 ceph-mon[99959]: pgmap v8153: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:45 smithi146 ceph-mon[103278]: pgmap v8153: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:47 smithi146 ceph-mon[103278]: pgmap v8154: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:47 smithi078 ceph-mon[91136]: pgmap v8154: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:47 smithi078 ceph-mon[99959]: pgmap v8154: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:49 smithi146 ceph-mon[103278]: pgmap v8155: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:49 smithi078 ceph-mon[91136]: pgmap v8155: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:49 smithi078 ceph-mon[99959]: pgmap v8155: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:51:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:51:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:51.277Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:51.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:51:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:51:51.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:51:52.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[99959]: pgmap v8156: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[91136]: pgmap v8156: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:51 smithi146 ceph-mon[103278]: pgmap v8156: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:53 smithi146 ceph-mon[103278]: pgmap v8157: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:53 smithi078 ceph-mon[91136]: pgmap v8157: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:53 smithi078 ceph-mon[99959]: pgmap v8157: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:51:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:55 smithi146 ceph-mon[103278]: pgmap v8158: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[91136]: pgmap v8158: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[99959]: pgmap v8158: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:51:58.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:57 smithi146 ceph-mon[103278]: pgmap v8159: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:57 smithi078 ceph-mon[91136]: pgmap v8159: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:57 smithi078 ceph-mon[99959]: pgmap v8159: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:51:59.085 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:51:59.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:51:59 smithi146 ceph-mon[103278]: pgmap v8160: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:51:59 smithi078 ceph-mon[99959]: pgmap v8160: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:51:59 smithi078 ceph-mon[91136]: pgmap v8160: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:52:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:52:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:01.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:01.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:01.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:01.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:01 smithi078 ceph-mon[99959]: pgmap v8161: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:01 smithi078 ceph-mon[91136]: pgmap v8161: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:01 smithi146 ceph-mon[103278]: pgmap v8161: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:02.324 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:52:02.855 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:52:02.855 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 4h - - 2024-09-21T00:52:02.855 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 4h - - 2024-09-21T00:52:02.855 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 12s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 7s ago 4h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 12s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 7s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 12s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 12s ago 4h 462M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 7s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 12s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:52:02.856 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 12s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 7s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 12s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 12s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 12s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 12s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 7s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:52:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 7s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 7s ago 4h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 7s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 7s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 12s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 7s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 12s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:52:02.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 7s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:52:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:52:03.165 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:52:03.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:52:03.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:03.165 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:52:03.165 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:52:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:52:03.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:52:03.434 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:52:03.434 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:52:03.434 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:52:03.434 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:52:03.435 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:52:03.747 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:52:03.747 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:52:03.747 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:52:03.747 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:52:03.747 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:52:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: from='client.62547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: pgmap v8162: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: from='client.62553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: from='client.62559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1726769889' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3218281015' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: from='client.62547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: pgmap v8162: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: from='client.62553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: from='client.62559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1726769889' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3218281015' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: from='client.62547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: pgmap v8162: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: from='client.62553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: from='client.62559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1726769889' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3218281015' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:04 smithi078 ceph-mon[91136]: from='client.62571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:04 smithi078 ceph-mon[99959]: from='client.62571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:04 smithi146 ceph-mon[103278]: from='client.62571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:05 smithi078 ceph-mon[91136]: pgmap v8163: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:05 smithi078 ceph-mon[99959]: pgmap v8163: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:05 smithi146 ceph-mon[103278]: pgmap v8163: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:07 smithi146 ceph-mon[103278]: pgmap v8164: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:07 smithi078 ceph-mon[99959]: pgmap v8164: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:07 smithi078 ceph-mon[91136]: pgmap v8164: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:09 smithi146 ceph-mon[103278]: pgmap v8165: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:09 smithi078 ceph-mon[99959]: pgmap v8165: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:09 smithi078 ceph-mon[91136]: pgmap v8165: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:10.928 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:52:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:52:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:11.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:11.278Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:11.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:11 smithi146 ceph-mon[103278]: pgmap v8166: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[99959]: pgmap v8166: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[91136]: pgmap v8166: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:13 smithi146 ceph-mon[103278]: pgmap v8167: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:13 smithi078 ceph-mon[99959]: pgmap v8167: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:13 smithi078 ceph-mon[91136]: pgmap v8167: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:15 smithi146 ceph-mon[103278]: pgmap v8168: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:16.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.348 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.349 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[91136]: pgmap v8168: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:16.349 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.349 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.350 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.350 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[99959]: pgmap v8168: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:16.350 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.350 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:16.350 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:17 smithi146 ceph-mon[103278]: pgmap v8169: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:17 smithi078 ceph-mon[91136]: pgmap v8169: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:17 smithi078 ceph-mon[99959]: pgmap v8169: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:20.256 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:19 smithi146 ceph-mon[103278]: pgmap v8170: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:19 smithi078 ceph-mon[91136]: pgmap v8170: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:20.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:19 smithi078 ceph-mon[99959]: pgmap v8170: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:52:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:52:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:21.278Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:21.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:21.489 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:21.279Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:22.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:21 smithi078 ceph-mon[99959]: pgmap v8171: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:22.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:21 smithi078 ceph-mon[91136]: pgmap v8171: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:21 smithi146 ceph-mon[103278]: pgmap v8171: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:23 smithi146 ceph-mon[103278]: pgmap v8172: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:23 smithi078 ceph-mon[99959]: pgmap v8172: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:23 smithi078 ceph-mon[91136]: pgmap v8172: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:25 smithi146 ceph-mon[103278]: pgmap v8173: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:25 smithi078 ceph-mon[91136]: pgmap v8173: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:25 smithi078 ceph-mon[99959]: pgmap v8173: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:27 smithi146 ceph-mon[103278]: pgmap v8174: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:27 smithi078 ceph-mon[99959]: pgmap v8174: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:27 smithi078 ceph-mon[91136]: pgmap v8174: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:29 smithi146 ceph-mon[103278]: pgmap v8175: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:29 smithi078 ceph-mon[91136]: pgmap v8175: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:29 smithi078 ceph-mon[99959]: pgmap v8175: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:52:30] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:52:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:31.279Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:31.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:31.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:32 smithi146 ceph-mon[103278]: pgmap v8176: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[91136]: pgmap v8176: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[99959]: pgmap v8176: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:34.040 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:52:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:34 smithi146 ceph-mon[103278]: pgmap v8177: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:34 smithi078 ceph-mon[99959]: pgmap v8177: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:34 smithi078 ceph-mon[91136]: pgmap v8177: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:34.591 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:52:34.591 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 4h - - 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 4h - - 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 3s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 3s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 3s ago 4h 463M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 4h 315M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 3s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:52:34.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 4h 87.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 19s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:52:34.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:34.906 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:52:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:52:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:52:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:52:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:52:34.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:52:34.907 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:52:35.178 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:52:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:52:35.179 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:52:35.179 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:52:35.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:35 smithi078 ceph-mon[99959]: from='client.62583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:35.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3089331160' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:35.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:35 smithi078 ceph-mon[91136]: from='client.62583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:35.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3089331160' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:35.485 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:52:35.485 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:52:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:52:35.485 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:52:35.485 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:52:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:35 smithi146 ceph-mon[103278]: from='client.62583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3089331160' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:52:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='client.82508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: pgmap v8178: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='client.82514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='client.82526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1012517223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='client.82508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: pgmap v8178: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='client.82514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='client.82526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1012517223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='client.82508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: pgmap v8178: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='client.82514 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='client.82526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:52:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1012517223' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:52:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:38 smithi146 ceph-mon[103278]: pgmap v8179: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:38 smithi078 ceph-mon[91136]: pgmap v8179: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:38 smithi078 ceph-mon[99959]: pgmap v8179: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:40.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:40 smithi146 ceph-mon[103278]: pgmap v8180: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:40 smithi078 ceph-mon[91136]: pgmap v8180: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:40 smithi078 ceph-mon[99959]: pgmap v8180: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:52:40] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:52:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:41.280Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:41.280Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:41.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:42 smithi146 ceph-mon[103278]: pgmap v8181: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:42.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:42 smithi078 ceph-mon[91136]: pgmap v8181: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:42 smithi078 ceph-mon[99959]: pgmap v8181: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:44 smithi146 ceph-mon[103278]: pgmap v8182: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:44 smithi078 ceph-mon[91136]: pgmap v8182: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:44 smithi078 ceph-mon[99959]: pgmap v8182: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:46.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:46 smithi078 ceph-mon[91136]: pgmap v8183: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:46.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:46 smithi078 ceph-mon[99959]: pgmap v8183: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:46 smithi146 ceph-mon[103278]: pgmap v8183: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:48.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:48 smithi078 ceph-mon[99959]: pgmap v8184: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:48.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:48 smithi078 ceph-mon[91136]: pgmap v8184: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:48 smithi146 ceph-mon[103278]: pgmap v8184: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:50 smithi146 ceph-mon[103278]: pgmap v8185: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:50 smithi078 ceph-mon[91136]: pgmap v8185: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:50 smithi078 ceph-mon[99959]: pgmap v8185: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:52:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T00:52:51.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:51.281Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:51.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:51.281Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:52:51.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:52:51.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:52:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:52 smithi146 ceph-mon[103278]: pgmap v8186: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[99959]: pgmap v8186: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[91136]: pgmap v8186: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:54 smithi146 ceph-mon[103278]: pgmap v8187: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:54 smithi078 ceph-mon[99959]: pgmap v8187: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:54 smithi078 ceph-mon[91136]: pgmap v8187: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:52:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:56 smithi146 ceph-mon[103278]: pgmap v8188: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[91136]: pgmap v8188: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[99959]: pgmap v8188: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:52:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:58 smithi146 ceph-mon[103278]: pgmap v8189: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:58 smithi078 ceph-mon[99959]: pgmap v8189: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:58 smithi078 ceph-mon[91136]: pgmap v8189: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:52:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:52:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:52:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:52:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:52:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:00 smithi146 ceph-mon[103278]: pgmap v8190: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:00 smithi078 ceph-mon[91136]: pgmap v8190: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:00 smithi078 ceph-mon[99959]: pgmap v8190: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:53:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:53:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:01.281Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:01.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:01.282Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:02.274 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:02 smithi146 ceph-mon[103278]: pgmap v8191: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:02 smithi078 ceph-mon[99959]: pgmap v8191: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:02 smithi078 ceph-mon[91136]: pgmap v8191: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:04 smithi146 ceph-mon[103278]: pgmap v8192: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:04 smithi078 ceph-mon[91136]: pgmap v8192: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:04 smithi078 ceph-mon[99959]: pgmap v8192: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:05.778 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:53:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:53:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 4h - - 2024-09-21T00:53:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 4h - - 2024-09-21T00:53:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 15s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:53:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 10s ago 4h 87.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 15s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 10s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 15s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 15s ago 4h 462M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 10s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 15s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 15s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 10s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 15s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 15s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 15s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 15s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:53:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:53:06.308 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:53:06.308 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:53:06.308 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 15s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:53:06.308 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:53:06.308 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 15s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:53:06.308 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:53:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:06 smithi146 ceph-mon[103278]: pgmap v8193: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:06 smithi078 ceph-mon[99959]: pgmap v8193: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:06 smithi078 ceph-mon[91136]: pgmap v8193: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:06.619 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:53:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:53:06.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:53:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:53:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:53:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:53:06.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:53:06.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:53:06.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:53:06.894 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:53:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:53:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:53:06.896 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:53:07.216 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:53:07.217 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:07.217 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:07.217 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:07 smithi146 ceph-mon[103278]: from='client.62619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:07 smithi146 ceph-mon[103278]: from='client.82544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3618104387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:07.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:07 smithi078 ceph-mon[91136]: from='client.62619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:07.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:07 smithi078 ceph-mon[91136]: from='client.82544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3618104387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:07 smithi078 ceph-mon[99959]: from='client.62619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:07 smithi078 ceph-mon[99959]: from='client.82544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3618104387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:08.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[91136]: from='client.82550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:08.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[91136]: pgmap v8194: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:08.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[91136]: from='client.62643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:08.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1846434167' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:08.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[99959]: from='client.82550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:08.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[99959]: pgmap v8194: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:08.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[99959]: from='client.62643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:08.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1846434167' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:08 smithi146 ceph-mon[103278]: from='client.82550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:08 smithi146 ceph-mon[103278]: pgmap v8194: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:08 smithi146 ceph-mon[103278]: from='client.62643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1846434167' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:10 smithi146 ceph-mon[103278]: pgmap v8195: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:10 smithi078 ceph-mon[91136]: pgmap v8195: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:10 smithi078 ceph-mon[99959]: pgmap v8195: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:53:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:53:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:11.282Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:11.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:11.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:12 smithi146 ceph-mon[103278]: pgmap v8196: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[99959]: pgmap v8196: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[91136]: pgmap v8196: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:12.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:14 smithi146 ceph-mon[103278]: pgmap v8197: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:14 smithi078 ceph-mon[99959]: pgmap v8197: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:14 smithi078 ceph-mon[91136]: pgmap v8197: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:16 smithi146 ceph-mon[103278]: pgmap v8198: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[99959]: pgmap v8198: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[91136]: pgmap v8198: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:18 smithi146 ceph-mon[103278]: pgmap v8199: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:18.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:18 smithi078 ceph-mon[99959]: pgmap v8199: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:18.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:18 smithi078 ceph-mon[91136]: pgmap v8199: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:20 smithi146 ceph-mon[103278]: pgmap v8200: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:20 smithi078 ceph-mon[99959]: pgmap v8200: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:20 smithi078 ceph-mon[91136]: pgmap v8200: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:53:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:53:21.359 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:21.283Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:21.359 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:21.283Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:21.359 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:21.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:22 smithi146 ceph-mon[103278]: pgmap v8201: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:22 smithi078 ceph-mon[99959]: pgmap v8201: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:22 smithi078 ceph-mon[91136]: pgmap v8201: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:24 smithi146 ceph-mon[103278]: pgmap v8202: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:24 smithi078 ceph-mon[99959]: pgmap v8202: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:24 smithi078 ceph-mon[91136]: pgmap v8202: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:26 smithi146 ceph-mon[103278]: pgmap v8203: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:26 smithi078 ceph-mon[99959]: pgmap v8203: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:26 smithi078 ceph-mon[91136]: pgmap v8203: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:27.582 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:27 smithi078 ceph-mon[99959]: pgmap v8204: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:27.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:27 smithi078 ceph-mon[91136]: pgmap v8204: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:27 smithi146 ceph-mon[103278]: pgmap v8204: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:29.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:29 smithi078 ceph-mon[91136]: pgmap v8205: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:29.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:29 smithi078 ceph-mon[99959]: pgmap v8205: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:29 smithi146 ceph-mon[103278]: pgmap v8205: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:53:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:53:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[99959]: pgmap v8206: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[91136]: pgmap v8206: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:31.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:31.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:31.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:31 smithi146 ceph-mon[103278]: pgmap v8206: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:33 smithi146 ceph-mon[103278]: pgmap v8207: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:33 smithi078 ceph-mon[91136]: pgmap v8207: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:33 smithi078 ceph-mon[99959]: pgmap v8207: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:35 smithi146 ceph-mon[103278]: pgmap v8208: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:35 smithi078 ceph-mon[91136]: pgmap v8208: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:35 smithi078 ceph-mon[99959]: pgmap v8208: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:37.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.512 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:53:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:38.034 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 4h - - 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 4h - - 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 1s ago 4h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 1s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 6s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 6s ago 4h 463M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 1s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 6s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:53:38.035 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 1s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 1s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 1s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 1s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 1s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 1s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 1s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:53:38.036 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 1s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:53:38.346 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:53:38.346 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:53:38.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:53:38.347 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:53:38.348 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:38 smithi078 ceph-mon[99959]: pgmap v8209: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:38.348 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:38 smithi078 ceph-mon[91136]: pgmap v8209: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:38 smithi146 ceph-mon[103278]: pgmap v8209: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:53:38.618 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:53:38.619 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:53:38.619 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:53:38.619 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:53:38.619 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:53:38.619 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:53:38.930 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:53:38.930 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:53:38.930 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:53:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[91136]: from='client.62655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[91136]: from='client.62661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[91136]: from='client.82586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2586142414' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/557020245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[99959]: from='client.62655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[99959]: from='client.62661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[99959]: from='client.82586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2586142414' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/557020245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:39 smithi146 ceph-mon[103278]: from='client.62655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:39 smithi146 ceph-mon[103278]: from='client.62661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:39 smithi146 ceph-mon[103278]: from='client.82586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2586142414' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:53:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/557020245' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:53:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:40 smithi146 ceph-mon[103278]: pgmap v8210: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:40 smithi146 ceph-mon[103278]: from='client.82598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:40 smithi078 ceph-mon[91136]: pgmap v8210: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:40 smithi078 ceph-mon[91136]: from='client.82598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:40 smithi078 ceph-mon[99959]: pgmap v8210: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:40 smithi078 ceph-mon[99959]: from='client.82598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:53:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:53:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:53:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:41.284Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:41.284Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:41.285Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:42.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:42 smithi146 ceph-mon[103278]: pgmap v8211: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:42 smithi078 ceph-mon[91136]: pgmap v8211: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:42 smithi078 ceph-mon[99959]: pgmap v8211: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:44 smithi146 ceph-mon[103278]: pgmap v8212: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:44 smithi078 ceph-mon[91136]: pgmap v8212: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:44 smithi078 ceph-mon[99959]: pgmap v8212: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:46.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:46 smithi146 ceph-mon[103278]: pgmap v8213: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:46 smithi078 ceph-mon[91136]: pgmap v8213: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:46 smithi078 ceph-mon[99959]: pgmap v8213: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:48.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:48 smithi078 ceph-mon[99959]: pgmap v8214: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:48.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:48 smithi078 ceph-mon[91136]: pgmap v8214: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:48 smithi146 ceph-mon[103278]: pgmap v8214: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:50.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:50 smithi078 ceph-mon[99959]: pgmap v8215: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:50.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:50 smithi078 ceph-mon[91136]: pgmap v8215: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:50 smithi146 ceph-mon[103278]: pgmap v8215: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:53:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:53:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:51.285Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:51.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:53:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:53:51.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:53:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:52 smithi146 ceph-mon[103278]: pgmap v8216: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[91136]: pgmap v8216: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[99959]: pgmap v8216: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:54 smithi146 ceph-mon[103278]: pgmap v8217: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:54 smithi078 ceph-mon[91136]: pgmap v8217: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:54 smithi078 ceph-mon[99959]: pgmap v8217: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:53:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:55 smithi146 ceph-mon[103278]: pgmap v8218: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:55 smithi078 ceph-mon[91136]: pgmap v8218: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:55 smithi078 ceph-mon[99959]: pgmap v8218: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:53:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:53:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:58 smithi146 ceph-mon[103278]: pgmap v8219: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:58 smithi078 ceph-mon[91136]: pgmap v8219: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:58 smithi078 ceph-mon[99959]: pgmap v8219: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:53:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:53:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:53:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:53:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:53:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:00 smithi146 ceph-mon[103278]: pgmap v8220: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:00 smithi078 ceph-mon[91136]: pgmap v8220: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:00 smithi078 ceph-mon[99959]: pgmap v8220: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:54:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:54:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:01.286Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:01.286Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:01.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:02 smithi146 ceph-mon[103278]: pgmap v8221: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:02 smithi078 ceph-mon[99959]: pgmap v8221: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:02 smithi078 ceph-mon[91136]: pgmap v8221: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:04 smithi146 ceph-mon[103278]: pgmap v8222: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:04 smithi078 ceph-mon[99959]: pgmap v8222: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:04 smithi078 ceph-mon[91136]: pgmap v8222: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:06 smithi146 ceph-mon[103278]: pgmap v8223: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:06 smithi078 ceph-mon[91136]: pgmap v8223: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:06 smithi078 ceph-mon[99959]: pgmap v8223: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:08 smithi146 ceph-mon[103278]: pgmap v8224: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:08 smithi078 ceph-mon[99959]: pgmap v8224: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:08 smithi078 ceph-mon[91136]: pgmap v8224: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:09.223 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:54:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:54:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 4h - - 2024-09-21T00:54:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 4h - - 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 18s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 13s ago 4h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 18s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 13s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 18s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 18s ago 4h 463M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 13s ago 4h 305M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 18s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 18s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 13s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 18s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:54:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 18s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 18s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 18s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 13s ago 4h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 18s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 18s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:54:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:54:10.082 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:54:10.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:54:10.082 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:54:10.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:10.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:54:10.083 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:54:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-mon[99959]: pgmap v8225: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-mon[99959]: from='client.62691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4002457207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-mon[91136]: pgmap v8225: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-mon[91136]: from='client.62691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4002457207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:10.371 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:54:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:54:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:54:10.371 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:54:10.372 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:54:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:54:10.373 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:54:10.373 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:54:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:10 smithi146 ceph-mon[103278]: pgmap v8225: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:10 smithi146 ceph-mon[103278]: from='client.62691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4002457207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:10.692 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:54:10.692 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:10.692 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:10.692 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:54:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:54:11.376 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-mon[99959]: from='client.82616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:11.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-mon[99959]: from='client.82622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:11.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3777728388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:11.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-mon[91136]: from='client.82616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:11.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-mon[91136]: from='client.82622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:11.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3777728388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:11.377 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:11.287Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:11.378 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:11.287Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:11.378 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:11.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:11 smithi146 ceph-mon[103278]: from='client.82616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:11 smithi146 ceph-mon[103278]: from='client.82622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3777728388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:12 smithi146 ceph-mon[103278]: from='client.62715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:12 smithi146 ceph-mon[103278]: pgmap v8226: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[91136]: from='client.62715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[91136]: pgmap v8226: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[99959]: from='client.62715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[99959]: pgmap v8226: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:14 smithi146 ceph-mon[103278]: pgmap v8227: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:14 smithi078 ceph-mon[91136]: pgmap v8227: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:14 smithi078 ceph-mon[99959]: pgmap v8227: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:16 smithi146 ceph-mon[103278]: pgmap v8228: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:16 smithi078 ceph-mon[91136]: pgmap v8228: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:16 smithi078 ceph-mon[99959]: pgmap v8228: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:17 smithi146 ceph-mon[103278]: pgmap v8229: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[99959]: pgmap v8229: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:17 smithi078 ceph-mon[91136]: pgmap v8229: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:19 smithi146 ceph-mon[103278]: pgmap v8230: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:19 smithi078 ceph-mon[91136]: pgmap v8230: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:19 smithi078 ceph-mon[99959]: pgmap v8230: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:54:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T00:54:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:21 smithi146 ceph-mon[103278]: pgmap v8231: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:21.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:21 smithi078 ceph-mon[91136]: pgmap v8231: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:21.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:21 smithi078 ceph-mon[99959]: pgmap v8231: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:21.288Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:21.288Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:21.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:23 smithi146 ceph-mon[103278]: pgmap v8232: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:23 smithi078 ceph-mon[99959]: pgmap v8232: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:23 smithi078 ceph-mon[91136]: pgmap v8232: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:25 smithi146 ceph-mon[103278]: pgmap v8233: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:25 smithi078 ceph-mon[91136]: pgmap v8233: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:25 smithi078 ceph-mon[99959]: pgmap v8233: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 00:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T00:54:25.849378242Z level=info msg="Completed cleanup jobs" duration=13.408295ms 2024-09-21T00:54:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:27 smithi146 ceph-mon[103278]: pgmap v8234: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:27 smithi078 ceph-mon[91136]: pgmap v8234: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:27 smithi078 ceph-mon[99959]: pgmap v8234: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:29.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:29 smithi078 ceph-mon[99959]: pgmap v8235: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:29.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:29 smithi078 ceph-mon[91136]: pgmap v8235: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:29 smithi146 ceph-mon[103278]: pgmap v8235: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:30.824 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:54:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:54:31.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:31.289Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:31.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:31.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:31.460 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:31.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:31 smithi146 ceph-mon[103278]: pgmap v8236: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:31.774 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:31 smithi078 ceph-mon[99959]: pgmap v8236: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:31.774 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:31 smithi078 ceph-mon[91136]: pgmap v8236: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:33.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:33 smithi146 ceph-mon[103278]: pgmap v8237: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:33 smithi078 ceph-mon[91136]: pgmap v8237: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:33 smithi078 ceph-mon[99959]: pgmap v8237: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:35 smithi146 ceph-mon[103278]: pgmap v8238: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:35 smithi078 ceph-mon[99959]: pgmap v8238: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:35 smithi078 ceph-mon[91136]: pgmap v8238: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:37 smithi146 ceph-mon[103278]: pgmap v8239: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[99959]: pgmap v8239: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[91136]: pgmap v8239: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:39 smithi146 ceph-mon[103278]: pgmap v8240: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:39.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:39 smithi078 ceph-mon[99959]: pgmap v8240: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:39.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:39 smithi078 ceph-mon[91136]: pgmap v8240: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:40.983 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:54:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:54:40.984 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 4h - - 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 9s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 9s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:54:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 9s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 9s ago 4h 464M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 5s ago 4h 306M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 9s ago 4h 315M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 9s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 9s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 9s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 9s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 9s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 5s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 5s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:54:41.522 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 77.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:54:41.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 9s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:54:41.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:54:41.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 9s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:54:41.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:54:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:41 smithi146 ceph-mon[103278]: pgmap v8241: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:41 smithi146 ceph-mon[103278]: from='client.82646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:41 smithi146 ceph-mon[103278]: from='client.62733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-mon[99959]: pgmap v8241: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-mon[99959]: from='client.82646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-mon[99959]: from='client.62733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-mon[91136]: pgmap v8241: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-mon[91136]: from='client.82646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-mon[91136]: from='client.62733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:41.289Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:41.289Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:41.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:41.837 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:54:41.838 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:54:42.103 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:54:42.104 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:54:42.436 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:54:42.436 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:54:42.436 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:42.436 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:54:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:42 smithi146 ceph-mon[103278]: from='client.82658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2272259355' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:42 smithi146 ceph-mon[103278]: from='client.82670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/514051341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[99959]: from='client.82658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2272259355' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[99959]: from='client.82670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/514051341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[91136]: from='client.82658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2272259355' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:54:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[91136]: from='client.82670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:54:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/514051341' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:54:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:43 smithi146 ceph-mon[103278]: pgmap v8242: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:43 smithi078 ceph-mon[91136]: pgmap v8242: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:43 smithi078 ceph-mon[99959]: pgmap v8242: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:45.725 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:45 smithi146 ceph-mon[103278]: pgmap v8243: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:45 smithi078 ceph-mon[91136]: pgmap v8243: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:45 smithi078 ceph-mon[99959]: pgmap v8243: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:47 smithi146 ceph-mon[103278]: pgmap v8244: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:47 smithi078 ceph-mon[91136]: pgmap v8244: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:47 smithi078 ceph-mon[99959]: pgmap v8244: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:49 smithi146 ceph-mon[103278]: pgmap v8245: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:49 smithi078 ceph-mon[99959]: pgmap v8245: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:49 smithi078 ceph-mon[91136]: pgmap v8245: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:54:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:54:51.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:51 smithi078 ceph-mon[99959]: pgmap v8246: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:51.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:51 smithi078 ceph-mon[91136]: pgmap v8246: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:51.619 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:51.290Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:51.619 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:51.290Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:54:51.619 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:54:51.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:54:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:51 smithi146 ceph-mon[103278]: pgmap v8246: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:53 smithi146 ceph-mon[103278]: pgmap v8247: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:53 smithi078 ceph-mon[99959]: pgmap v8247: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:53 smithi078 ceph-mon[91136]: pgmap v8247: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:54:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:55 smithi146 ceph-mon[103278]: pgmap v8248: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:55 smithi078 ceph-mon[91136]: pgmap v8248: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:55 smithi078 ceph-mon[99959]: pgmap v8248: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:54:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:57 smithi146 ceph-mon[103278]: pgmap v8249: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[91136]: pgmap v8249: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[99959]: pgmap v8249: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:54:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:54:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:54:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:54:59 smithi146 ceph-mon[103278]: pgmap v8250: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:54:59 smithi078 ceph-mon[91136]: pgmap v8250: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:54:59 smithi078 ceph-mon[99959]: pgmap v8250: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:55:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:55:01.598 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:01.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:01.599 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:01.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:55:01.599 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:01.291Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:01 smithi146 ceph-mon[103278]: pgmap v8251: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:01 smithi078 ceph-mon[91136]: pgmap v8251: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:01 smithi078 ceph-mon[99959]: pgmap v8251: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:03 smithi146 ceph-mon[103278]: pgmap v8252: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:03 smithi078 ceph-mon[91136]: pgmap v8252: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:03 smithi078 ceph-mon[99959]: pgmap v8252: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:05.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:05 smithi146 ceph-mon[103278]: pgmap v8253: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:05 smithi078 ceph-mon[99959]: pgmap v8253: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:05 smithi078 ceph-mon[91136]: pgmap v8253: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:07 smithi146 ceph-mon[103278]: pgmap v8254: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:07 smithi078 ceph-mon[91136]: pgmap v8254: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:07 smithi078 ceph-mon[99959]: pgmap v8254: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:09 smithi146 ceph-mon[103278]: pgmap v8255: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:09 smithi078 ceph-mon[91136]: pgmap v8255: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:09 smithi078 ceph-mon[99959]: pgmap v8255: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:10.982 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:55:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:55:11.562 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:11.291Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:11.562 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:11.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:55:11.563 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:11.292Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:11.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:11 smithi078 ceph-mon[99959]: pgmap v8256: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:11.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:11 smithi078 ceph-mon[91136]: pgmap v8256: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:11 smithi146 ceph-mon[103278]: pgmap v8256: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:12.732 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:55:13.287 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:55:13.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 4h - - 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 4h - - 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 1s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 16s ago 4h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 1s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 16s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 1s ago 4h 581M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 1s ago 4h 465M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 16s ago 4h 307M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 1s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 1s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 16s ago 4h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 1s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 1s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 1s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:55:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 1s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 16s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 16s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 16s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 16s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 16s ago 4h 77.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 1s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 16s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 1s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:55:13.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 16s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:55:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:13.602 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:55:13.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:55:13.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:55:13.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:55:13.604 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:55:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:55:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:55:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:55:14.199 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:55:14.199 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:14.199 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:14.199 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:14.200 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[91136]: pgmap v8257: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[91136]: from='client.82682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[91136]: from='client.82688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3365246610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[99959]: pgmap v8257: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:14.200 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[99959]: from='client.82682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:14.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[99959]: from='client.82688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:14.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:14.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:14.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3365246610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:14 smithi146 ceph-mon[103278]: pgmap v8257: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:14 smithi146 ceph-mon[103278]: from='client.82682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:14 smithi146 ceph-mon[103278]: from='client.82688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3365246610' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:15 smithi078 ceph-mon[91136]: from='client.62775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:15 smithi078 ceph-mon[91136]: from='client.82706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2999145513' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:15 smithi078 ceph-mon[99959]: from='client.62775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:15 smithi078 ceph-mon[99959]: from='client.82706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2999145513' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:15 smithi146 ceph-mon[103278]: from='client.62775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:15 smithi146 ceph-mon[103278]: from='client.82706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2999145513' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:16 smithi146 ceph-mon[103278]: pgmap v8258: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:16 smithi078 ceph-mon[91136]: pgmap v8258: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:16 smithi078 ceph-mon[99959]: pgmap v8258: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:17 smithi146 ceph-mon[103278]: pgmap v8259: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[91136]: pgmap v8259: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[99959]: pgmap v8259: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:19 smithi146 ceph-mon[103278]: pgmap v8260: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:19 smithi078 ceph-mon[91136]: pgmap v8260: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:19 smithi078 ceph-mon[99959]: pgmap v8260: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:55:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:55:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:21.292Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:21.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:55:21.726 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:21.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:21 smithi146 ceph-mon[103278]: pgmap v8261: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:21 smithi078 ceph-mon[91136]: pgmap v8261: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:21 smithi078 ceph-mon[99959]: pgmap v8261: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:23 smithi146 ceph-mon[103278]: pgmap v8262: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:24.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:23 smithi078 ceph-mon[99959]: pgmap v8262: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:24.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:23 smithi078 ceph-mon[91136]: pgmap v8262: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:25 smithi146 ceph-mon[103278]: pgmap v8263: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:25 smithi078 ceph-mon[91136]: pgmap v8263: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:25 smithi078 ceph-mon[99959]: pgmap v8263: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:27 smithi146 ceph-mon[103278]: pgmap v8264: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:27 smithi078 ceph-mon[99959]: pgmap v8264: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:27 smithi078 ceph-mon[91136]: pgmap v8264: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:29 smithi146 ceph-mon[103278]: pgmap v8265: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:29 smithi078 ceph-mon[99959]: pgmap v8265: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:29 smithi078 ceph-mon[91136]: pgmap v8265: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:55:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:55:31.658 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:31.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:31.658 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:31.293Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:55:31.658 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:31.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:31.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:31 smithi078 ceph-mon[91136]: pgmap v8266: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:31.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:31 smithi078 ceph-mon[99959]: pgmap v8266: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:31 smithi146 ceph-mon[103278]: pgmap v8266: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:34.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:34 smithi078 ceph-mon[91136]: pgmap v8267: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:34.352 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:34 smithi078 ceph-mon[99959]: pgmap v8267: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:34 smithi146 ceph-mon[103278]: pgmap v8267: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:36 smithi146 ceph-mon[103278]: pgmap v8268: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:36 smithi078 ceph-mon[99959]: pgmap v8268: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:36 smithi078 ceph-mon[91136]: pgmap v8268: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:37 smithi146 ceph-mon[103278]: pgmap v8269: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[91136]: pgmap v8269: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[99959]: pgmap v8269: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T00:55:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:39 smithi146 ceph-mon[103278]: pgmap v8270: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:39 smithi078 ceph-mon[99959]: pgmap v8270: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:39 smithi078 ceph-mon[91136]: pgmap v8270: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:55:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:55:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:41.293Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:41.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:55:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:41.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:41 smithi146 ceph-mon[103278]: pgmap v8271: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:41 smithi078 ceph-mon[91136]: pgmap v8271: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:41 smithi078 ceph-mon[99959]: pgmap v8271: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:43 smithi146 ceph-mon[103278]: pgmap v8272: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:43 smithi078 ceph-mon[91136]: pgmap v8272: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:43 smithi078 ceph-mon[99959]: pgmap v8272: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:44.496 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:55:45.025 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:55:45.025 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 4h - - 2024-09-21T00:55:45.025 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 4h - - 2024-09-21T00:55:45.025 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 12s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:55:45.025 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 8s ago 4h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 12s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 8s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 12s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 12s ago 4h 466M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 8s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 12s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 12s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 8s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 12s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 12s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 12s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 12s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 8s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 8s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:55:45.026 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 8s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:55:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 8s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:55:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 8s ago 4h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:55:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 12s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:55:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 8s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:55:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 12s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:55:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:55:45.221 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:55:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:55:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:55:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:55:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:55:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:55:45.336 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:55:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:55:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:55:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:55:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:55:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:55:45.606 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:55:45.914 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:55:45.915 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:55:45.915 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:55:46.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:45 smithi146 ceph-mon[103278]: pgmap v8273: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:46.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:45 smithi146 ceph-mon[103278]: from='client.62799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:45 smithi146 ceph-mon[103278]: from='client.62805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:45 smithi146 ceph-mon[103278]: from='client.62811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/174590360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[99959]: pgmap v8273: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[99959]: from='client.62799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[99959]: from='client.62805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[99959]: from='client.62811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/174590360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[91136]: pgmap v8273: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[91136]: from='client.62799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[91136]: from='client.62805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[91136]: from='client.62811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/174590360' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:55:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:46 smithi146 ceph-mon[103278]: from='client.62823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1000419399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:47.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:46 smithi078 ceph-mon[91136]: from='client.62823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1000419399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:46 smithi078 ceph-mon[99959]: from='client.62823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:55:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1000419399' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:55:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:47 smithi146 ceph-mon[103278]: pgmap v8274: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:47 smithi078 ceph-mon[91136]: pgmap v8274: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:47 smithi078 ceph-mon[99959]: pgmap v8274: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:49 smithi146 ceph-mon[103278]: pgmap v8275: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:49 smithi078 ceph-mon[91136]: pgmap v8275: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:49 smithi078 ceph-mon[99959]: pgmap v8275: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:55:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:55:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:51.294Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:51.294Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:55:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:55:51.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:55:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:51 smithi146 ceph-mon[103278]: pgmap v8276: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:51 smithi078 ceph-mon[99959]: pgmap v8276: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:51 smithi078 ceph-mon[91136]: pgmap v8276: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:53.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[99959]: pgmap v8277: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:53.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:53 smithi078 ceph-mon[91136]: pgmap v8277: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:53 smithi146 ceph-mon[103278]: pgmap v8277: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:55:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:55 smithi078 ceph-mon[99959]: pgmap v8278: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:55 smithi078 ceph-mon[91136]: pgmap v8278: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:55 smithi146 ceph-mon[103278]: pgmap v8278: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:55:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:57 smithi146 ceph-mon[103278]: pgmap v8279: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[91136]: pgmap v8279: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[99959]: pgmap v8279: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:55:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:55:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:55:59 smithi146 ceph-mon[103278]: pgmap v8280: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:55:59 smithi078 ceph-mon[91136]: pgmap v8280: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:55:59 smithi078 ceph-mon[99959]: pgmap v8280: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:56:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:56:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:01.295Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:01.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:01.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:01 smithi146 ceph-mon[103278]: pgmap v8281: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:01 smithi078 ceph-mon[91136]: pgmap v8281: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:01 smithi078 ceph-mon[99959]: pgmap v8281: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:03 smithi146 ceph-mon[103278]: pgmap v8282: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:03 smithi078 ceph-mon[91136]: pgmap v8282: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:03 smithi078 ceph-mon[99959]: pgmap v8282: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:05 smithi146 ceph-mon[103278]: pgmap v8283: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:05 smithi078 ceph-mon[99959]: pgmap v8283: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:05 smithi078 ceph-mon[91136]: pgmap v8283: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:07 smithi146 ceph-mon[103278]: pgmap v8284: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:07 smithi078 ceph-mon[99959]: pgmap v8284: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:07 smithi078 ceph-mon[91136]: pgmap v8284: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:10 smithi146 ceph-mon[103278]: pgmap v8285: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:10 smithi078 ceph-mon[99959]: pgmap v8285: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:10 smithi078 ceph-mon[91136]: pgmap v8285: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:56:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:56:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:11.295Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:11.295Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:11.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:12 smithi146 ceph-mon[103278]: pgmap v8286: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:12 smithi078 ceph-mon[99959]: pgmap v8286: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:12 smithi078 ceph-mon[91136]: pgmap v8286: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[91136]: pgmap v8287: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[99959]: pgmap v8287: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:13 smithi146 ceph-mon[103278]: pgmap v8287: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:15 smithi146 ceph-mon[103278]: pgmap v8288: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:15.774 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:15 smithi078 ceph-mon[91136]: pgmap v8288: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:15.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:15 smithi078 ceph-mon[99959]: pgmap v8288: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:16.206 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:56:16.731 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:56:16.731 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 4h - - 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 4h - - 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 4s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 4s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 4s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 4s ago 4h 467M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 4h 308M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 4s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 4s ago 4h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:56:16.732 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 4s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 4s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 4s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 4s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 19s ago 4h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 4s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:56:16.733 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 4s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:56:16.734 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:56:16.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:16 smithi078 ceph-mon[99959]: from='client.82754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:16.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:16 smithi078 ceph-mon[91136]: from='client.82754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:16 smithi146 ceph-mon[103278]: from='client.82754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:17.044 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:56:17.045 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:56:17.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:56:17.045 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:56:17.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:56:17.317 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:56:17.630 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:56:17.630 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:17.630 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:17.630 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:17.630 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: from='client.62841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: pgmap v8289: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: from='client.62847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3662355737' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: from='client.62841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: pgmap v8289: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: from='client.62847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3662355737' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:17.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: from='client.62841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: pgmap v8289: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: from='client.62847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3662355737' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:18 smithi146 ceph-mon[103278]: from='client.82778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2051391851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:18 smithi078 ceph-mon[91136]: from='client.82778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2051391851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:18 smithi078 ceph-mon[99959]: from='client.82778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2051391851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:19 smithi146 ceph-mon[103278]: pgmap v8290: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:19 smithi078 ceph-mon[91136]: pgmap v8290: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:19 smithi078 ceph-mon[99959]: pgmap v8290: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:56:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:56:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:21 smithi146 ceph-mon[103278]: pgmap v8291: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:21.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:21 smithi078 ceph-mon[99959]: pgmap v8291: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:21.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:21 smithi078 ceph-mon[91136]: pgmap v8291: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:21.296Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:21.296Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:21.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:23 smithi146 ceph-mon[103278]: pgmap v8292: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:23 smithi078 ceph-mon[99959]: pgmap v8292: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:23.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:23 smithi078 ceph-mon[91136]: pgmap v8292: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:24.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:25 smithi146 ceph-mon[103278]: pgmap v8293: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:25 smithi078 ceph-mon[99959]: pgmap v8293: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:25 smithi078 ceph-mon[91136]: pgmap v8293: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:27.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:27 smithi146 ceph-mon[103278]: pgmap v8294: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:27 smithi078 ceph-mon[99959]: pgmap v8294: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:27 smithi078 ceph-mon[91136]: pgmap v8294: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:28.629 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:29 smithi146 ceph-mon[103278]: pgmap v8295: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:29 smithi078 ceph-mon[99959]: pgmap v8295: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:29 smithi078 ceph-mon[91136]: pgmap v8295: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:56:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:56:31.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:31 smithi078 ceph-mon[99959]: pgmap v8296: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:31 smithi078 ceph-mon[91136]: pgmap v8296: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:31.296Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:31.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:31.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:31 smithi146 ceph-mon[103278]: pgmap v8296: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:33 smithi146 ceph-mon[103278]: pgmap v8297: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[91136]: pgmap v8297: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[99959]: pgmap v8297: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:34.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:34.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:35 smithi146 ceph-mon[103278]: pgmap v8298: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:35 smithi078 ceph-mon[91136]: pgmap v8298: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:35 smithi078 ceph-mon[99959]: pgmap v8298: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:38 smithi146 ceph-mon[103278]: pgmap v8299: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[91136]: pgmap v8299: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[99959]: pgmap v8299: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:40 smithi146 ceph-mon[103278]: pgmap v8300: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:40 smithi078 ceph-mon[91136]: pgmap v8300: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:40 smithi078 ceph-mon[99959]: pgmap v8300: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:56:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:56:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:41.297Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:41.297Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:41.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:42 smithi146 ceph-mon[103278]: pgmap v8301: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:42 smithi078 ceph-mon[99959]: pgmap v8301: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:42 smithi078 ceph-mon[91136]: pgmap v8301: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:44 smithi146 ceph-mon[103278]: pgmap v8302: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:44 smithi078 ceph-mon[99959]: pgmap v8302: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:44 smithi078 ceph-mon[91136]: pgmap v8302: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:46 smithi146 ceph-mon[103278]: pgmap v8303: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:46 smithi078 ceph-mon[91136]: pgmap v8303: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:46 smithi078 ceph-mon[99959]: pgmap v8303: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:47.924 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:56:48.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:48 smithi146 ceph-mon[103278]: pgmap v8304: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:48.449 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 4h - - 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 4h - - 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 15s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 11s ago 4h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 15s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 11s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 15s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 15s ago 4h 468M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 11s ago 4h 309M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 15s ago 4h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 15s ago 4h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:56:48.450 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 11s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 15s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 15s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 15s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 15s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 11s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 11s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 11s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 11s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 11s ago 4h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 15s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 11s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 15s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:56:48.451 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:56:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:48 smithi078 ceph-mon[91136]: pgmap v8304: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:48 smithi078 ceph-mon[99959]: pgmap v8304: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:48.757 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:56:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:56:48.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:56:48.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:56:48.759 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:56:49.023 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:56:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:56:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:56:49.024 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:56:49.024 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:56:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:49 smithi078 ceph-mon[91136]: from='client.82790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:49 smithi078 ceph-mon[91136]: from='client.62877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1970239670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:49 smithi078 ceph-mon[99959]: from='client.82790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:49 smithi078 ceph-mon[99959]: from='client.62877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1970239670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:49.335 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:56:49.336 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:49.336 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:56:49.336 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:49.336 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:56:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:49 smithi146 ceph-mon[103278]: from='client.82790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:49 smithi146 ceph-mon[103278]: from='client.62877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1970239670' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:56:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:50 smithi146 ceph-mon[103278]: from='client.82802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:50 smithi146 ceph-mon[103278]: pgmap v8305: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:50 smithi146 ceph-mon[103278]: from='client.82814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2153019784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[91136]: from='client.82802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[91136]: pgmap v8305: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[91136]: from='client.82814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2153019784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[99959]: from='client.82802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[99959]: pgmap v8305: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[99959]: from='client.82814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:56:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2153019784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:56:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:56:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:56:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:51.298Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:51.298Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:56:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:56:51.299Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:56:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:52 smithi146 ceph-mon[103278]: pgmap v8306: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:52 smithi078 ceph-mon[91136]: pgmap v8306: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:52 smithi078 ceph-mon[99959]: pgmap v8306: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:53 smithi146 ceph-mon[103278]: pgmap v8307: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[91136]: pgmap v8307: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[99959]: pgmap v8307: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:56:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:55 smithi146 ceph-mon[103278]: pgmap v8308: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:55 smithi078 ceph-mon[99959]: pgmap v8308: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:55 smithi078 ceph-mon[91136]: pgmap v8308: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:56:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:57 smithi146 ceph-mon[103278]: pgmap v8309: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[99959]: pgmap v8309: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[91136]: pgmap v8309: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:56:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:56:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:56:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:56:59 smithi146 ceph-mon[103278]: pgmap v8310: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:56:59 smithi078 ceph-mon[99959]: pgmap v8310: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:56:59 smithi078 ceph-mon[91136]: pgmap v8310: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:57:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T00:57:01.717 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:01.299Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:01.717 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:01.300Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:01.717 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:01.300Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:01 smithi146 ceph-mon[103278]: pgmap v8311: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:02.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:01 smithi078 ceph-mon[91136]: pgmap v8311: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:01 smithi078 ceph-mon[99959]: pgmap v8311: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:03 smithi146 ceph-mon[103278]: pgmap v8312: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:03 smithi078 ceph-mon[91136]: pgmap v8312: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:03 smithi078 ceph-mon[99959]: pgmap v8312: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:05 smithi146 ceph-mon[103278]: pgmap v8313: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:05 smithi078 ceph-mon[99959]: pgmap v8313: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:05 smithi078 ceph-mon[91136]: pgmap v8313: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:07.998 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:07 smithi146 ceph-mon[103278]: pgmap v8314: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:07 smithi078 ceph-mon[99959]: pgmap v8314: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:07 smithi078 ceph-mon[91136]: pgmap v8314: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:09 smithi146 ceph-mon[103278]: pgmap v8315: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:09 smithi078 ceph-mon[99959]: pgmap v8315: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:09 smithi078 ceph-mon[91136]: pgmap v8315: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:57:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T00:57:11.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:11.300Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:11.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:11.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:11.740 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:11.301Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:11 smithi146 ceph-mon[103278]: pgmap v8316: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:12.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:11 smithi078 ceph-mon[91136]: pgmap v8316: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:12.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:11 smithi078 ceph-mon[99959]: pgmap v8316: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:13 smithi146 ceph-mon[103278]: pgmap v8317: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[91136]: pgmap v8317: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[99959]: pgmap v8317: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:16.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:15 smithi078 ceph-mon[91136]: pgmap v8318: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:16.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:15 smithi078 ceph-mon[99959]: pgmap v8318: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:15 smithi146 ceph-mon[103278]: pgmap v8318: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:17 smithi146 ceph-mon[103278]: pgmap v8319: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[91136]: pgmap v8319: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[99959]: pgmap v8319: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:19.626 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:57:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:19 smithi078 ceph-mon[99959]: pgmap v8320: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:19 smithi078 ceph-mon[91136]: pgmap v8320: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:20.151 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:57:20.151 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 4h - - 2024-09-21T00:57:20.151 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 4h - - 2024-09-21T00:57:20.151 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 7s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:57:20.151 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 2s ago 4h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 7s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 2s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 7s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 7s ago 4h 469M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 2s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 7s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 7s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 2s ago 4h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 7s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 7s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 7s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 7s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 2s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 2s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 2s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:57:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 2s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:57:20.153 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 2s ago 4h 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:57:20.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 7s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:57:20.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 2s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:57:20.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 7s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:57:20.153 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 2s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:57:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:19 smithi146 ceph-mon[103278]: pgmap v8320: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:57:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:57:20.463 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:57:20.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:57:20.730 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:57:20.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:57:20.731 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:57:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T00:57:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[99959]: from='client.82826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[99959]: from='client.82832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[99959]: from='client.82838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2305932571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[91136]: from='client.82826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[91136]: from='client.82832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[91136]: from='client.82838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2305932571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:21.045 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:57:21.045 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:21.045 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:20 smithi146 ceph-mon[103278]: from='client.82826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:20 smithi146 ceph-mon[103278]: from='client.82832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:20 smithi146 ceph-mon[103278]: from='client.82838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2305932571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:21.301Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:21.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:21.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:21 smithi146 ceph-mon[103278]: pgmap v8321: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:21 smithi146 ceph-mon[103278]: from='client.62931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2002174068' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-mon[99959]: pgmap v8321: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-mon[99959]: from='client.62931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2002174068' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-mon[91136]: pgmap v8321: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-mon[91136]: from='client.62931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2002174068' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:23 smithi146 ceph-mon[103278]: pgmap v8322: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:23 smithi078 ceph-mon[91136]: pgmap v8322: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:23 smithi078 ceph-mon[99959]: pgmap v8322: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:25 smithi146 ceph-mon[103278]: pgmap v8323: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:25 smithi078 ceph-mon[91136]: pgmap v8323: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:25 smithi078 ceph-mon[99959]: pgmap v8323: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:27 smithi146 ceph-mon[103278]: pgmap v8324: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:27 smithi078 ceph-mon[99959]: pgmap v8324: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:27 smithi078 ceph-mon[91136]: pgmap v8324: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:29.154 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:30.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:29 smithi146 ceph-mon[103278]: pgmap v8325: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:29 smithi078 ceph-mon[99959]: pgmap v8325: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:29 smithi078 ceph-mon[91136]: pgmap v8325: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:57:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T00:57:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:31.302Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:31.302Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:31.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:31 smithi146 ceph-mon[103278]: pgmap v8326: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:31 smithi078 ceph-mon[91136]: pgmap v8326: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:31 smithi078 ceph-mon[99959]: pgmap v8326: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:33 smithi146 ceph-mon[103278]: pgmap v8327: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[99959]: pgmap v8327: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[91136]: pgmap v8327: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T00:57:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:35.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:35 smithi146 ceph-mon[103278]: pgmap v8328: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:35 smithi078 ceph-mon[99959]: pgmap v8328: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:35 smithi078 ceph-mon[91136]: pgmap v8328: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:37 smithi146 ceph-mon[103278]: pgmap v8329: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[99959]: pgmap v8329: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[91136]: pgmap v8329: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:39 smithi146 ceph-mon[103278]: pgmap v8330: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:39 smithi078 ceph-mon[91136]: pgmap v8330: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:39 smithi078 ceph-mon[99959]: pgmap v8330: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:57:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T00:57:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:41.302Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:41.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:41.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:41 smithi146 ceph-mon[103278]: pgmap v8331: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:41 smithi078 ceph-mon[91136]: pgmap v8331: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:41 smithi078 ceph-mon[99959]: pgmap v8331: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:43 smithi146 ceph-mon[103278]: pgmap v8332: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:43 smithi078 ceph-mon[99959]: pgmap v8332: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:43 smithi078 ceph-mon[91136]: pgmap v8332: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:45 smithi146 ceph-mon[103278]: pgmap v8333: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:45 smithi078 ceph-mon[99959]: pgmap v8333: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:45 smithi078 ceph-mon[91136]: pgmap v8333: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:47 smithi146 ceph-mon[103278]: pgmap v8334: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:47 smithi078 ceph-mon[99959]: pgmap v8334: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:47 smithi078 ceph-mon[91136]: pgmap v8334: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:49 smithi146 ceph-mon[103278]: pgmap v8335: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:49 smithi078 ceph-mon[99959]: pgmap v8335: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:49 smithi078 ceph-mon[91136]: pgmap v8335: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:57:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:57:51.349 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:57:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:51.303Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:51.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:57:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:57:51.303Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:57:51.939 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:57:51.939 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 4h - - 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 4h - - 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 18s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 14s ago 4h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 18s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 14s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 18s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 18s ago 4h 469M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 14s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 18s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 18s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 14s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 18s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 18s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 18s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:57:51.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 18s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 14s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 14s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 14s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 14s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 14s ago 4h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 18s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 14s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 18s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:57:51.941 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 14s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:57:52.259 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:51 smithi078 ceph-mon[99959]: pgmap v8336: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:52.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:57:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:57:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:57:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:57:52.260 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:57:52.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:51 smithi078 ceph-mon[91136]: pgmap v8336: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:51 smithi146 ceph-mon[103278]: pgmap v8336: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:57:52.535 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:57:52.536 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:57:52.536 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:57:52.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:57:52.842 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:57:52.842 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:57:52.842 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:57:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:53 smithi146 ceph-mon[103278]: from='client.62943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:53 smithi146 ceph-mon[103278]: from='client.62949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:53 smithi146 ceph-mon[103278]: from='client.82874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4157012045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1520127994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[99959]: from='client.62943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[99959]: from='client.62949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[99959]: from='client.82874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4157012045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1520127994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[91136]: from='client.62943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[91136]: from='client.62949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[91136]: from='client.82874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4157012045' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:57:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1520127994' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:57:54.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:54 smithi146 ceph-mon[103278]: pgmap v8337: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:54.409 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:54 smithi146 ceph-mon[103278]: from='client.62967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:54.410 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.410 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.410 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[99959]: pgmap v8337: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[99959]: from='client.62967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[91136]: pgmap v8337: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[91136]: from='client.62967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:55.342 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:57:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:56 smithi146 ceph-mon[103278]: pgmap v8338: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:56 smithi078 ceph-mon[91136]: pgmap v8338: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:56 smithi078 ceph-mon[99959]: pgmap v8338: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:57:58.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[91136]: pgmap v8339: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:58.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[99959]: pgmap v8339: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:58.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:58 smithi146 ceph-mon[103278]: pgmap v8339: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:57:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:57:59.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:57:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:59.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:57:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:57:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:57:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:00 smithi146 ceph-mon[103278]: pgmap v8340: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:00 smithi078 ceph-mon[91136]: pgmap v8340: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:00 smithi078 ceph-mon[99959]: pgmap v8340: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:58:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:58:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:01.303Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:01.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:01.304Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:02 smithi146 ceph-mon[103278]: pgmap v8341: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:02 smithi078 ceph-mon[99959]: pgmap v8341: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:02 smithi078 ceph-mon[91136]: pgmap v8341: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:04 smithi146 ceph-mon[103278]: pgmap v8342: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:04 smithi078 ceph-mon[99959]: pgmap v8342: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:04 smithi078 ceph-mon[91136]: pgmap v8342: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:05.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:06 smithi146 ceph-mon[103278]: pgmap v8343: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:06 smithi078 ceph-mon[91136]: pgmap v8343: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:06 smithi078 ceph-mon[99959]: pgmap v8343: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:08 smithi146 ceph-mon[103278]: pgmap v8344: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:08 smithi078 ceph-mon[91136]: pgmap v8344: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:08 smithi078 ceph-mon[99959]: pgmap v8344: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:10.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:10 smithi146 ceph-mon[103278]: pgmap v8345: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:10 smithi078 ceph-mon[99959]: pgmap v8345: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:10 smithi078 ceph-mon[91136]: pgmap v8345: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:58:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:58:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:11.305Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:11.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:11.305Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:12 smithi146 ceph-mon[103278]: pgmap v8346: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:12 smithi078 ceph-mon[91136]: pgmap v8346: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:12 smithi078 ceph-mon[99959]: pgmap v8346: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:14 smithi146 ceph-mon[103278]: pgmap v8347: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[91136]: pgmap v8347: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[99959]: pgmap v8347: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:16 smithi146 ceph-mon[103278]: pgmap v8348: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:16 smithi078 ceph-mon[91136]: pgmap v8348: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:16 smithi078 ceph-mon[99959]: pgmap v8348: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:18.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[91136]: pgmap v8349: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:18.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[99959]: pgmap v8349: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:18.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:18 smithi146 ceph-mon[103278]: pgmap v8349: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:20.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:20 smithi078 ceph-mon[99959]: pgmap v8350: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:20.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:20 smithi078 ceph-mon[91136]: pgmap v8350: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:20 smithi146 ceph-mon[103278]: pgmap v8350: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:58:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:58:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:21.305Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:21.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:21.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:22 smithi146 ceph-mon[103278]: pgmap v8351: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:22 smithi078 ceph-mon[91136]: pgmap v8351: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:22 smithi078 ceph-mon[99959]: pgmap v8351: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:23.136 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 4h - - 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 4h - - 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 10s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 10s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:58:23.687 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 10s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 10s ago 4h 471M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 5s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 10s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 10s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 10s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 10s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 10s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 10s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 5s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 5s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:58:23.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:58:23.689 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 72.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:58:23.689 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 10s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:58:23.689 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:58:23.689 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 10s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:58:23.689 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:58:24.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:58:24.044 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:58:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:24 smithi146 ceph-mon[103278]: pgmap v8352: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:24 smithi146 ceph-mon[103278]: from='client.82898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/347837794' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:24 smithi078 ceph-mon[91136]: pgmap v8352: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:24 smithi078 ceph-mon[91136]: from='client.82898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/347837794' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:24 smithi078 ceph-mon[99959]: pgmap v8352: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:24 smithi078 ceph-mon[99959]: from='client.82898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/347837794' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:24.631 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:58:24.631 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:24.631 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:24.631 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:24.631 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:25 smithi146 ceph-mon[103278]: from='client.62985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:25 smithi146 ceph-mon[103278]: from='client.82910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1675714452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[91136]: from='client.62985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[91136]: from='client.82910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1675714452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[99959]: from='client.62985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[99959]: from='client.82910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1675714452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:26 smithi146 ceph-mon[103278]: from='client.63003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:26 smithi146 ceph-mon[103278]: pgmap v8353: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:26 smithi078 ceph-mon[99959]: from='client.63003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:26 smithi078 ceph-mon[99959]: pgmap v8353: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:26 smithi078 ceph-mon[91136]: from='client.63003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:26 smithi078 ceph-mon[91136]: pgmap v8353: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:28 smithi146 ceph-mon[103278]: pgmap v8354: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:28 smithi078 ceph-mon[91136]: pgmap v8354: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:28 smithi078 ceph-mon[99959]: pgmap v8354: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:30 smithi146 ceph-mon[103278]: pgmap v8355: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:30 smithi078 ceph-mon[91136]: pgmap v8355: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:30 smithi078 ceph-mon[99959]: pgmap v8355: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:58:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:58:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:31.306Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:31.306Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:31.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:32 smithi146 ceph-mon[103278]: pgmap v8356: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:32 smithi078 ceph-mon[91136]: pgmap v8356: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:32 smithi078 ceph-mon[99959]: pgmap v8356: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:34 smithi146 ceph-mon[103278]: pgmap v8357: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[99959]: pgmap v8357: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[91136]: pgmap v8357: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:36 smithi146 ceph-mon[103278]: pgmap v8358: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:36 smithi078 ceph-mon[99959]: pgmap v8358: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:36 smithi078 ceph-mon[91136]: pgmap v8358: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[91136]: pgmap v8359: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[99959]: pgmap v8359: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:38 smithi146 ceph-mon[103278]: pgmap v8359: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:40.509 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:40 smithi078 ceph-mon[99959]: pgmap v8360: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:40.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:40 smithi078 ceph-mon[91136]: pgmap v8360: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:40 smithi146 ceph-mon[103278]: pgmap v8360: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:40.767 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:58:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T00:58:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:41.307Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:41.307Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:41.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:42 smithi078 ceph-mon[91136]: pgmap v8361: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:42 smithi078 ceph-mon[99959]: pgmap v8361: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:42 smithi146 ceph-mon[103278]: pgmap v8361: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:44 smithi078 ceph-mon[99959]: pgmap v8362: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:44 smithi078 ceph-mon[91136]: pgmap v8362: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:44 smithi146 ceph-mon[103278]: pgmap v8362: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:46 smithi146 ceph-mon[103278]: pgmap v8363: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:46 smithi078 ceph-mon[99959]: pgmap v8363: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:46 smithi078 ceph-mon[91136]: pgmap v8363: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:47 smithi146 ceph-mon[103278]: pgmap v8364: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:47 smithi078 ceph-mon[99959]: pgmap v8364: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:47 smithi078 ceph-mon[91136]: pgmap v8364: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:49 smithi146 ceph-mon[103278]: pgmap v8365: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:49 smithi078 ceph-mon[91136]: pgmap v8365: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:49 smithi078 ceph-mon[99959]: pgmap v8365: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:58:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T00:58:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:51.308Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:51.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:58:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:58:51.308Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:58:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:51 smithi146 ceph-mon[103278]: pgmap v8366: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:51 smithi078 ceph-mon[91136]: pgmap v8366: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:51 smithi078 ceph-mon[99959]: pgmap v8366: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:53 smithi146 ceph-mon[103278]: pgmap v8367: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[99959]: pgmap v8367: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[91136]: pgmap v8367: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:54.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:54.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:54.924 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:58:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:58:55.450 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:58:55.450 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 4h - - 2024-09-21T00:58:55.450 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 4h - - 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 1s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 17s ago 4h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 1s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 17s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 1s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 1s ago 4h 470M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 17s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 1s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 1s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 17s ago 4h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 1s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 1s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 1s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:58:55.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 1s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 17s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 17s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 17s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 17s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 17s ago 4h 73.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 1s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 17s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 1s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:58:55.452 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 17s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:58:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:58:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:58:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:58:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:55.759 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:58:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:58:55.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:55 smithi078 ceph-mon[99959]: pgmap v8368: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:55.761 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:55 smithi078 ceph-mon[99959]: from='client.82934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:55.761 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:55 smithi078 ceph-mon[99959]: from='client.63021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:55.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:55 smithi078 ceph-mon[91136]: pgmap v8368: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:55.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:55 smithi078 ceph-mon[91136]: from='client.82934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:55.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:55 smithi078 ceph-mon[91136]: from='client.63021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:55 smithi146 ceph-mon[103278]: pgmap v8368: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:58:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:55 smithi146 ceph-mon[103278]: from='client.82934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:55 smithi146 ceph-mon[103278]: from='client.63021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:58:56.030 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:58:56.031 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:58:56.031 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:58:56.341 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:58:56.341 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:56.341 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:58:56.341 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:56.341 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:58:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:56 smithi146 ceph-mon[103278]: from='client.63027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1315850300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:56 smithi146 ceph-mon[103278]: from='client.63039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2705088685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[91136]: from='client.63027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1315850300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[91136]: from='client.63039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2705088685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[99959]: from='client.63027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1315850300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[99959]: from='client.63039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2705088685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:58:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:57 smithi146 ceph-mon[103278]: pgmap v8369: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:57 smithi078 ceph-mon[99959]: pgmap v8369: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:57 smithi078 ceph-mon[91136]: pgmap v8369: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:58:59.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:59.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:58:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:58:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:00 smithi146 ceph-mon[103278]: pgmap v8370: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:00 smithi078 ceph-mon[91136]: pgmap v8370: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:00 smithi078 ceph-mon[99959]: pgmap v8370: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:59:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:59:01.647 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:01.309Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:01.647 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:01.309Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:01.647 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:01.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:02 smithi146 ceph-mon[103278]: pgmap v8371: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:02 smithi078 ceph-mon[91136]: pgmap v8371: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:02 smithi078 ceph-mon[99959]: pgmap v8371: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:04 smithi146 ceph-mon[103278]: pgmap v8372: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:04 smithi078 ceph-mon[91136]: pgmap v8372: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:04 smithi078 ceph-mon[99959]: pgmap v8372: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:05.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:06 smithi146 ceph-mon[103278]: pgmap v8373: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:06 smithi078 ceph-mon[91136]: pgmap v8373: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:06 smithi078 ceph-mon[99959]: pgmap v8373: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:08 smithi146 ceph-mon[103278]: pgmap v8374: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:08 smithi078 ceph-mon[91136]: pgmap v8374: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:08 smithi078 ceph-mon[99959]: pgmap v8374: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:10 smithi146 ceph-mon[103278]: pgmap v8375: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:10 smithi078 ceph-mon[99959]: pgmap v8375: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:10 smithi078 ceph-mon[91136]: pgmap v8375: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:59:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T00:59:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:11.310Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:11.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:11.310Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:12.285 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:12 smithi146 ceph-mon[103278]: pgmap v8376: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:12 smithi078 ceph-mon[99959]: pgmap v8376: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:12 smithi078 ceph-mon[91136]: pgmap v8376: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:14 smithi146 ceph-mon[103278]: pgmap v8377: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[99959]: pgmap v8377: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[91136]: pgmap v8377: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:16 smithi146 ceph-mon[103278]: pgmap v8378: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:16 smithi078 ceph-mon[99959]: pgmap v8378: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:16 smithi078 ceph-mon[91136]: pgmap v8378: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:18 smithi146 ceph-mon[103278]: pgmap v8379: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:18 smithi078 ceph-mon[91136]: pgmap v8379: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:18 smithi078 ceph-mon[99959]: pgmap v8379: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:20.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:20 smithi078 ceph-mon[99959]: pgmap v8380: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:20 smithi078 ceph-mon[91136]: pgmap v8380: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:20 smithi146 ceph-mon[103278]: pgmap v8380: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:59:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T00:59:21.484 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:21.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:21.484 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:21.311Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:21.484 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:21.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:22.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:22 smithi078 ceph-mon[91136]: pgmap v8381: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:22.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:22 smithi078 ceph-mon[99959]: pgmap v8381: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:22 smithi146 ceph-mon[103278]: pgmap v8381: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:24 smithi078 ceph-mon[99959]: pgmap v8382: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:24 smithi078 ceph-mon[91136]: pgmap v8382: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:24 smithi146 ceph-mon[103278]: pgmap v8382: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:25.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:26 smithi078 ceph-mon[91136]: pgmap v8383: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:26 smithi078 ceph-mon[99959]: pgmap v8383: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:26.628 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:59:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:26 smithi146 ceph-mon[103278]: pgmap v8383: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:27.152 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 4h - - 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 4h - - 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 13s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 8s ago 4h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 13s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 8s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 13s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 13s ago 4h 471M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 8s ago 4h 312M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 13s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 13s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:59:27.153 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 8s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 13s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 13s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 13s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 13s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 8s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 8s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 8s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 8s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 8s ago 4h 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 13s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 8s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 13s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:59:27.154 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 8s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:59:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:59:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:59:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:59:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:59:27.464 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:59:27.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:59:27.729 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:59:27.729 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:59:27.730 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:59:28.037 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:59:28.037 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:28.037 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:28.037 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:28.037 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:28.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[99959]: pgmap v8384: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[91136]: pgmap v8384: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[91136]: from='client.63051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[91136]: from='client.63057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[91136]: from='client.82982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3893735541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:28.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1475654950' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:28 smithi146 ceph-mon[103278]: pgmap v8384: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:28 smithi146 ceph-mon[103278]: from='client.63051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:28 smithi146 ceph-mon[103278]: from='client.63057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:28 smithi146 ceph-mon[103278]: from='client.82982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3893735541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1475654950' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[99959]: from='client.63051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[99959]: from='client.63057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[99959]: from='client.82982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3893735541' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T00:59:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1475654950' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T00:59:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:29 smithi146 ceph-mon[103278]: from='client.63075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:29 smithi146 ceph-mon[103278]: pgmap v8385: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:29 smithi078 ceph-mon[99959]: from='client.63075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:29 smithi078 ceph-mon[99959]: pgmap v8385: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:29 smithi078 ceph-mon[91136]: from='client.63075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T00:59:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:29 smithi078 ceph-mon[91136]: pgmap v8385: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:59:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:59:31.586 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:31.311Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:31.586 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:31.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:31.586 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:31.312Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:31 smithi146 ceph-mon[103278]: pgmap v8386: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:31 smithi078 ceph-mon[99959]: pgmap v8386: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:31 smithi078 ceph-mon[91136]: pgmap v8386: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:33.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:33 smithi078 ceph-mon[99959]: pgmap v8387: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:33.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:33 smithi078 ceph-mon[91136]: pgmap v8387: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:33 smithi146 ceph-mon[103278]: pgmap v8387: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:35 smithi146 ceph-mon[103278]: pgmap v8388: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:35 smithi078 ceph-mon[99959]: pgmap v8388: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:35 smithi078 ceph-mon[91136]: pgmap v8388: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:37 smithi146 ceph-mon[103278]: pgmap v8389: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:37 smithi078 ceph-mon[99959]: pgmap v8389: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:37 smithi078 ceph-mon[91136]: pgmap v8389: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:39 smithi146 ceph-mon[103278]: pgmap v8390: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:39.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[91136]: pgmap v8390: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:39 smithi078 ceph-mon[99959]: pgmap v8390: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:40.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:59:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T00:59:41.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:41.313Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:41.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:41.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:41.589 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:41.313Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:41.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:41 smithi078 ceph-mon[91136]: pgmap v8391: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:41.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:41 smithi078 ceph-mon[99959]: pgmap v8391: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:41 smithi146 ceph-mon[103278]: pgmap v8391: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:43 smithi078 ceph-mon[91136]: pgmap v8392: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:43 smithi078 ceph-mon[99959]: pgmap v8392: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:43.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:43 smithi146 ceph-mon[103278]: pgmap v8392: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:43.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:45 smithi146 ceph-mon[103278]: pgmap v8393: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:45 smithi078 ceph-mon[99959]: pgmap v8393: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:45 smithi078 ceph-mon[91136]: pgmap v8393: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:47 smithi146 ceph-mon[103278]: pgmap v8394: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:47 smithi078 ceph-mon[91136]: pgmap v8394: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:47 smithi078 ceph-mon[99959]: pgmap v8394: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:49 smithi146 ceph-mon[103278]: pgmap v8395: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:49 smithi078 ceph-mon[91136]: pgmap v8395: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:49 smithi078 ceph-mon[99959]: pgmap v8395: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 00:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:00:59:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T00:59:51.609 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:51.313Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:51.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:51.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T00:59:51.610 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 00:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T00:59:51.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T00:59:51.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:51 smithi146 ceph-mon[103278]: pgmap v8396: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:52.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:51 smithi078 ceph-mon[99959]: pgmap v8396: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:51 smithi078 ceph-mon[91136]: pgmap v8396: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:53 smithi078 ceph-mon[91136]: pgmap v8397: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:53 smithi078 ceph-mon[99959]: pgmap v8397: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:53.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:53 smithi146 ceph-mon[103278]: pgmap v8397: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T00:59:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:55 smithi146 ceph-mon[103278]: pgmap v8398: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:55 smithi078 ceph-mon[91136]: pgmap v8398: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:55 smithi078 ceph-mon[99959]: pgmap v8398: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T00:59:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:57 smithi146 ceph-mon[103278]: pgmap v8399: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:57 smithi078 ceph-mon[99959]: pgmap v8399: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:57 smithi078 ceph-mon[91136]: pgmap v8399: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T00:59:58.331 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T00:59:58.867 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T00:59:58.867 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 4h - - 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 4h - - 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 5s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 0s ago 4h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 5s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 0s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 5s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 5s ago 4h 471M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 0s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 5s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 5s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 0s ago 4h 16.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 5s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 5s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 5s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T00:59:58.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 5s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 0s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 0s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 0s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 0s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 0s ago 4h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 5s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 0s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 5s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T00:59:58.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 0s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T00:59:59.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:59:59.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T00:59:59.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T00:59:59.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T00:59:59.180 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T00:59:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T00:59:59.450 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T00:59:59.450 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T00:59:59.450 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T00:59:59.450 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T00:59:59.450 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T00:59:59.451 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T00:59:59.760 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T00:59:59.760 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:59.760 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T00:59:59.760 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T00:59:59.760 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[99959]: from='client.63087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[99959]: pgmap v8400: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[99959]: from='client.63093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[99959]: from='client.83018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/417395171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/719710339' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[91136]: from='client.63087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[91136]: pgmap v8400: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[91136]: from='client.63093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[91136]: from='client.83018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/417395171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 00:59:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/719710339' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T00:59:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:59 smithi146 ceph-mon[103278]: from='client.63087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:59 smithi146 ceph-mon[103278]: pgmap v8400: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:59 smithi146 ceph-mon[103278]: from='client.63093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:59 smithi146 ceph-mon[103278]: from='client.83018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/417395171' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 00:59:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/719710339' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:00.876 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:00:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:00:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:00 smithi146 ceph-mon[103278]: from='client.63111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:01.016 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 21 01:00:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-21T01:00:00.647Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726869600138 maxt=1726876800000 ulid=01J891FH72PPJDGKD47T2C9KTY duration=37.012824ms 2024-09-21T01:00:01.016 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 21 01:00:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-21T01:00:00.650Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=2.107774ms 2024-09-21T01:00:01.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:00 smithi078 ceph-mon[91136]: from='client.63111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:00 smithi078 ceph-mon[99959]: from='client.63111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:01.625 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:01.313Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:01.625 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:01.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:01.625 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:01.314Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:01 smithi146 ceph-mon[103278]: pgmap v8401: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:01 smithi078 ceph-mon[91136]: pgmap v8401: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:01 smithi078 ceph-mon[99959]: pgmap v8401: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:03 smithi146 ceph-mon[103278]: pgmap v8402: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:03 smithi078 ceph-mon[91136]: pgmap v8402: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:03 smithi078 ceph-mon[99959]: pgmap v8402: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:05 smithi146 ceph-mon[103278]: pgmap v8403: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:05 smithi078 ceph-mon[99959]: pgmap v8403: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:05 smithi078 ceph-mon[91136]: pgmap v8403: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:07 smithi146 ceph-mon[103278]: pgmap v8404: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:07 smithi078 ceph-mon[99959]: pgmap v8404: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:07 smithi078 ceph-mon[91136]: pgmap v8404: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:09 smithi146 ceph-mon[103278]: pgmap v8405: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:09 smithi078 ceph-mon[99959]: pgmap v8405: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:09 smithi078 ceph-mon[91136]: pgmap v8405: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:00:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:00:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:11.314Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:11.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:11.315Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:12.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:11 smithi146 ceph-mon[103278]: pgmap v8406: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:11 smithi078 ceph-mon[91136]: pgmap v8406: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:11 smithi078 ceph-mon[99959]: pgmap v8406: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:14.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:13 smithi146 ceph-mon[103278]: pgmap v8407: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:14.264 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:13 smithi078 ceph-mon[91136]: pgmap v8407: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:13 smithi078 ceph-mon[99959]: pgmap v8407: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:15 smithi146 ceph-mon[103278]: pgmap v8408: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:15 smithi078 ceph-mon[91136]: pgmap v8408: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:15 smithi078 ceph-mon[99959]: pgmap v8408: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:17 smithi146 ceph-mon[103278]: pgmap v8409: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:17 smithi078 ceph-mon[91136]: pgmap v8409: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:17 smithi078 ceph-mon[99959]: pgmap v8409: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:19 smithi146 ceph-mon[103278]: pgmap v8410: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[91136]: pgmap v8410: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[99959]: pgmap v8410: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:00:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:00:21.656 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:21.315Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:21.656 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:21.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:21.657 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:21.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:21 smithi146 ceph-mon[103278]: pgmap v8411: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:22.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:21 smithi078 ceph-mon[99959]: pgmap v8411: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:22.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:21 smithi078 ceph-mon[91136]: pgmap v8411: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:23.903 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:23 smithi078 ceph-mon[91136]: pgmap v8412: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:23.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:23 smithi078 ceph-mon[99959]: pgmap v8412: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:23 smithi146 ceph-mon[103278]: pgmap v8412: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:25 smithi146 ceph-mon[103278]: pgmap v8413: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:25 smithi078 ceph-mon[99959]: pgmap v8413: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:25 smithi078 ceph-mon[91136]: pgmap v8413: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:27 smithi146 ceph-mon[103278]: pgmap v8414: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:27 smithi078 ceph-mon[91136]: pgmap v8414: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:27 smithi078 ceph-mon[99959]: pgmap v8414: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:29 smithi146 ceph-mon[103278]: pgmap v8415: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:29 smithi078 ceph-mon[99959]: pgmap v8415: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:29 smithi078 ceph-mon[91136]: pgmap v8415: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:30.055 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 4h - - 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 4h - - 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 16s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 11s ago 4h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 16s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 11s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 16s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 16s ago 4h 471M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:00:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 11s ago 4h 311M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 16s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 16s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 11s ago 4h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 16s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 16s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 16s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 16s ago 4h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 11s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 11s ago 4h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 11s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 11s ago 4h 83.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 11s ago 4h 107M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 16s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:00:30.581 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 11s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:00:30.582 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 16s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:00:30.582 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 11s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:00:30.894 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:00:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:00:30.896 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:00:30.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:30 smithi078 ceph-mon[99959]: from='client.63123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:30.897 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:00:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T01:00:30.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:30 smithi078 ceph-mon[91136]: from='client.63123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:30 smithi146 ceph-mon[103278]: from='client.63123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.163 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:00:31.163 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:00:31.164 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:00:31.475 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:00:31.476 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:00:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:00:31.476 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:00:31.476 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:31.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:31.476 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:31.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:31.476 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:31.316Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[91136]: from='client.63129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[91136]: pgmap v8416: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[91136]: from='client.83054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/980450644' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[91136]: from='client.63147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4159875486' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[99959]: from='client.63129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[99959]: pgmap v8416: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[99959]: from='client.83054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/980450644' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[99959]: from='client.63147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4159875486' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:31 smithi146 ceph-mon[103278]: from='client.63129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:31 smithi146 ceph-mon[103278]: pgmap v8416: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:31 smithi146 ceph-mon[103278]: from='client.83054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/980450644' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:00:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:31 smithi146 ceph-mon[103278]: from='client.63147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:00:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4159875486' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:00:33.889 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:33 smithi146 ceph-mon[103278]: pgmap v8417: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:33 smithi078 ceph-mon[91136]: pgmap v8417: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:33 smithi078 ceph-mon[99959]: pgmap v8417: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:36 smithi146 ceph-mon[103278]: pgmap v8418: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:36 smithi078 ceph-mon[99959]: pgmap v8418: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:36 smithi078 ceph-mon[91136]: pgmap v8418: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:38 smithi146 ceph-mon[103278]: pgmap v8419: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:38 smithi078 ceph-mon[99959]: pgmap v8419: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:38 smithi078 ceph-mon[91136]: pgmap v8419: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:39 smithi146 ceph-mon[103278]: pgmap v8420: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[91136]: pgmap v8420: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[99959]: pgmap v8420: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:00:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T01:00:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:41.316Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:41.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:41.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:41 smithi146 ceph-mon[103278]: pgmap v8421: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:41 smithi078 ceph-mon[91136]: pgmap v8421: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:41 smithi078 ceph-mon[99959]: pgmap v8421: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:44.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:43 smithi078 ceph-mon[91136]: pgmap v8422: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:44.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:44.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:43 smithi078 ceph-mon[99959]: pgmap v8422: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:44.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:43 smithi146 ceph-mon[103278]: pgmap v8422: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:45.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:45.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:45 smithi146 ceph-mon[103278]: pgmap v8423: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:45 smithi078 ceph-mon[91136]: pgmap v8423: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:45 smithi078 ceph-mon[99959]: pgmap v8423: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:47 smithi146 ceph-mon[103278]: pgmap v8424: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:47 smithi078 ceph-mon[91136]: pgmap v8424: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:47 smithi078 ceph-mon[99959]: pgmap v8424: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:49 smithi146 ceph-mon[103278]: pgmap v8425: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:49 smithi078 ceph-mon[91136]: pgmap v8425: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:49 smithi078 ceph-mon[99959]: pgmap v8425: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:00:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:00:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:00:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:51.317Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:51.317Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:00:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:00:51.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:00:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:51 smithi146 ceph-mon[103278]: pgmap v8426: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:51 smithi078 ceph-mon[99959]: pgmap v8426: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:51 smithi078 ceph-mon[91136]: pgmap v8426: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:53 smithi146 ceph-mon[103278]: pgmap v8427: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:54.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:53 smithi078 ceph-mon[91136]: pgmap v8427: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:54.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:53 smithi078 ceph-mon[99959]: pgmap v8427: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:00:55.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:55.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:56 smithi146 ceph-mon[103278]: pgmap v8428: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:00:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:56 smithi078 ceph-mon[91136]: pgmap v8428: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:00:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:56 smithi078 ceph-mon[99959]: pgmap v8428: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:00:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:58 smithi146 ceph-mon[103278]: pgmap v8429: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:00:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:58 smithi078 ceph-mon[91136]: pgmap v8429: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:00:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:58 smithi078 ceph-mon[99959]: pgmap v8429: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:00:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:00:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:00:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:00 smithi078 ceph-mon[99959]: pgmap v8430: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:01:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:00 smithi078 ceph-mon[91136]: pgmap v8430: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:01:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:00 smithi146 ceph-mon[103278]: pgmap v8430: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:01:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:01:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:01:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:01.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:01.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:01.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:01.765 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 4h - - 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 4h - - 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 8s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 3s ago 4h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 8s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 3s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 8s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:01:02.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 8s ago 4h 472M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 3s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 8s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 8s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 3s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 8s ago 4h 87.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 8s ago 4h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 8s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 8s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 3s ago 4h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 3s ago 4h 80.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 3s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 3s ago 4h 84.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 3s ago 4h 65.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:01:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 8s ago 4h 89.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:01:02.295 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 3s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:01:02.295 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 8s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:01:02.295 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 3s ago 4h 89.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:01:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:02 smithi078 ceph-mon[99959]: pgmap v8431: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T01:01:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:02 smithi078 ceph-mon[91136]: pgmap v8431: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:01:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:01:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:01:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:02 smithi146 ceph-mon[103278]: pgmap v8431: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T01:01:02.871 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:01:02.871 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:01:02.871 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:01:02.871 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:01:02.871 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:01:02.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:01:02.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:01:03.179 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:01:03.179 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:01:03.179 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:01:03.179 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:01:03.179 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[99959]: from='client.63159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[99959]: from='client.83084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[99959]: from='client.83090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3964620044' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4285213220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[91136]: from='client.63159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[91136]: from='client.83084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[91136]: from='client.83090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3964620044' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:03.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4285213220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:03 smithi146 ceph-mon[103278]: from='client.63159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:03 smithi146 ceph-mon[103278]: from='client.83084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:03 smithi146 ceph-mon[103278]: from='client.83090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3964620044' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4285213220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:04.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:04 smithi078 ceph-mon[99959]: pgmap v8432: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T01:01:04.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:04 smithi078 ceph-mon[99959]: from='client.63183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:04.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:04 smithi078 ceph-mon[91136]: pgmap v8432: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T01:01:04.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:04 smithi078 ceph-mon[91136]: from='client.63183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:04 smithi146 ceph-mon[103278]: pgmap v8432: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T01:01:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:04 smithi146 ceph-mon[103278]: from='client.63183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:05.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:06 smithi078 ceph-mon[91136]: pgmap v8433: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T01:01:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:06 smithi078 ceph-mon[99959]: pgmap v8433: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T01:01:06.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:06 smithi146 ceph-mon[103278]: pgmap v8433: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T01:01:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:08 smithi146 ceph-mon[103278]: pgmap v8434: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:01:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:08 smithi078 ceph-mon[91136]: pgmap v8434: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:01:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:08 smithi078 ceph-mon[99959]: pgmap v8434: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:01:10.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:10 smithi078 ceph-mon[99959]: pgmap v8435: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:01:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:10 smithi078 ceph-mon[91136]: pgmap v8435: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:01:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:10 smithi146 ceph-mon[103278]: pgmap v8435: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:01:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:01:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:01:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:11.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:11.318Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:11.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:12 smithi146 ceph-mon[103278]: pgmap v8436: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:01:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:12 smithi078 ceph-mon[91136]: pgmap v8436: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:01:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:12 smithi078 ceph-mon[99959]: pgmap v8436: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:01:13.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:13 smithi146 ceph-mon[103278]: pgmap v8437: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:01:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:13 smithi078 ceph-mon[91136]: pgmap v8437: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:01:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:13 smithi078 ceph-mon[99959]: pgmap v8437: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T01:01:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:15 smithi146 ceph-mon[103278]: pgmap v8438: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:01:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[99959]: pgmap v8438: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:15 smithi078 ceph-mon[91136]: pgmap v8438: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:01:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:17 smithi146 ceph-mon[103278]: pgmap v8439: 161 pgs: 161 active+clean; 585 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-21T01:01:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:17 smithi078 ceph-mon[91136]: pgmap v8439: 161 pgs: 161 active+clean; 585 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-21T01:01:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:17 smithi078 ceph-mon[99959]: pgmap v8439: 161 pgs: 161 active+clean; 585 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-21T01:01:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:20 smithi146 ceph-mon[103278]: pgmap v8440: 161 pgs: 161 active+clean; 585 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:01:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[91136]: pgmap v8440: 161 pgs: 161 active+clean; 585 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:01:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[99959]: pgmap v8440: 161 pgs: 161 active+clean; 585 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:01:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:01:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:01:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:21.318Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:21.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:21.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:22 smithi146 ceph-mon[103278]: pgmap v8441: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T01:01:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:22 smithi078 ceph-mon[99959]: pgmap v8441: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T01:01:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:22 smithi078 ceph-mon[91136]: pgmap v8441: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T01:01:24.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:24 smithi078 ceph-mon[91136]: pgmap v8442: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:01:24.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:24 smithi078 ceph-mon[99959]: pgmap v8442: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:01:24.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:24 smithi146 ceph-mon[103278]: pgmap v8442: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:01:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:25.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:26 smithi146 ceph-mon[103278]: pgmap v8443: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:01:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:26 smithi078 ceph-mon[91136]: pgmap v8443: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:01:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:26 smithi078 ceph-mon[99959]: pgmap v8443: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:01:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:28 smithi146 ceph-mon[103278]: pgmap v8444: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:01:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:28 smithi078 ceph-mon[91136]: pgmap v8444: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:01:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:28 smithi078 ceph-mon[99959]: pgmap v8444: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:01:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:30 smithi146 ceph-mon[103278]: pgmap v8445: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:01:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:30 smithi078 ceph-mon[91136]: pgmap v8445: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:01:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:30 smithi078 ceph-mon[99959]: pgmap v8445: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:01:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:01:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:01:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:31.319Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:31.319Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:31.320Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:32 smithi146 ceph-mon[103278]: pgmap v8446: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:01:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:32 smithi078 ceph-mon[99959]: pgmap v8446: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:01:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:32 smithi078 ceph-mon[91136]: pgmap v8446: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:01:33.471 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:01:33.993 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:01:33.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 4h - - 2024-09-21T01:01:33.993 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 4h - - 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 19s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 14s ago 4h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 19s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 14s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 19s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 19s ago 4h 473M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 14s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 19s ago 4h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 19s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 14s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 19s ago 4h 87.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 19s ago 4h 91.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 19s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:01:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 19s ago 4h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 14s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 14s ago 4h 81.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 14s ago 4h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 14s ago 4h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 14s ago 4h 65.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 19s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 14s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 19s ago 4h 90.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:01:33.995 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 14s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:01:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:34 smithi078 ceph-mon[91136]: pgmap v8447: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:34 smithi078 ceph-mon[99959]: pgmap v8447: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:01:34.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:01:34.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:01:34.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:34 smithi146 ceph-mon[103278]: pgmap v8447: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:01:34.595 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:01:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:01:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:01:34.596 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:01:34.596 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:01:34.901 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:01:34.901 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:01:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:01:34.901 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:01:34.901 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='client.63195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='client.63201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='client.63207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1020415213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1297721340' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='client.63195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='client.63201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='client.63207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1020415213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1297721340' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='client.63195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='client.63201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='client.63207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1020415213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1297721340' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:01:36.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:36 smithi146 ceph-mon[103278]: pgmap v8448: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:36.352 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:36 smithi146 ceph-mon[103278]: from='client.83138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:36 smithi078 ceph-mon[99959]: pgmap v8448: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:36 smithi078 ceph-mon[99959]: from='client.83138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:36 smithi078 ceph-mon[91136]: pgmap v8448: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:36 smithi078 ceph-mon[91136]: from='client.83138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:01:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:38 smithi146 ceph-mon[103278]: pgmap v8449: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:38 smithi078 ceph-mon[91136]: pgmap v8449: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:38 smithi078 ceph-mon[99959]: pgmap v8449: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:40 smithi146 ceph-mon[103278]: pgmap v8450: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[91136]: pgmap v8450: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[99959]: pgmap v8450: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:01:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:01:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:41.320Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:41.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:41.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:42 smithi146 ceph-mon[103278]: pgmap v8451: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:42 smithi078 ceph-mon[91136]: pgmap v8451: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:42 smithi078 ceph-mon[99959]: pgmap v8451: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:44 smithi146 ceph-mon[103278]: pgmap v8452: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:44 smithi078 ceph-mon[91136]: pgmap v8452: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:44 smithi078 ceph-mon[99959]: pgmap v8452: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:45.258 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:45.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:46.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:46 smithi078 ceph-mon[91136]: pgmap v8453: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:46.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:46 smithi078 ceph-mon[99959]: pgmap v8453: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:46 smithi146 ceph-mon[103278]: pgmap v8453: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:48 smithi146 ceph-mon[103278]: pgmap v8454: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:48 smithi078 ceph-mon[91136]: pgmap v8454: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:48 smithi078 ceph-mon[99959]: pgmap v8454: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:50 smithi146 ceph-mon[103278]: pgmap v8455: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:50 smithi078 ceph-mon[91136]: pgmap v8455: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:50 smithi078 ceph-mon[99959]: pgmap v8455: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:01:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:01:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:51.320Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:51.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:01:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:01:51.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:01:52.488 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:52 smithi078 ceph-mon[91136]: pgmap v8456: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:52.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:52 smithi078 ceph-mon[99959]: pgmap v8456: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:52 smithi146 ceph-mon[103278]: pgmap v8456: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:54 smithi146 ceph-mon[103278]: pgmap v8457: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:54 smithi078 ceph-mon[91136]: pgmap v8457: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:54 smithi078 ceph-mon[99959]: pgmap v8457: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:01:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:01:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:56 smithi146 ceph-mon[103278]: pgmap v8458: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:56 smithi078 ceph-mon[91136]: pgmap v8458: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:56 smithi078 ceph-mon[99959]: pgmap v8458: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:01:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:58 smithi146 ceph-mon[103278]: pgmap v8459: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:58 smithi078 ceph-mon[91136]: pgmap v8459: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:58 smithi078 ceph-mon[99959]: pgmap v8459: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:01:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:01:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:01:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:01:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:01:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[91136]: pgmap v8460: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[99959]: pgmap v8460: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:00 smithi146 ceph-mon[103278]: pgmap v8460: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:02:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T01:02:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:01.321Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:01.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:01.321Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:02.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:02 smithi078 ceph-mon[99959]: pgmap v8461: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:02.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:02 smithi078 ceph-mon[91136]: pgmap v8461: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:02 smithi146 ceph-mon[103278]: pgmap v8461: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:04 smithi146 ceph-mon[103278]: pgmap v8462: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:04 smithi078 ceph-mon[91136]: pgmap v8462: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:04 smithi078 ceph-mon[99959]: pgmap v8462: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:05.199 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:02:05.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:05.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:05 smithi078 ceph-mon[99959]: pgmap v8463: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:05.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:05 smithi078 ceph-mon[99959]: from='client.63231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:05.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:05.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:05 smithi078 ceph-mon[91136]: pgmap v8463: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:05.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:05 smithi078 ceph-mon[91136]: from='client.63231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:05.742 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 4h - - 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 4h - - 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 11s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 6s ago 4h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 11s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 6s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 11s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 11s ago 4h 474M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 6s ago 4h 313M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:02:05.743 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 11s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 11s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 6s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 11s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 11s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 11s ago 4h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 11s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 6s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 6s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 6s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 6s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 6s ago 4h 65.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:02:05.744 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 11s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:02:05.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 6s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:02:05.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 11s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:02:05.745 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 6s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:02:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:05 smithi146 ceph-mon[103278]: pgmap v8463: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:05 smithi146 ceph-mon[103278]: from='client.63231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.056 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:02:06.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:02:06.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:02:06.058 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:02:06.330 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:02:06.331 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:02:06.331 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:02:06.331 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:02:06.659 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:02:06.659 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:06.659 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:06.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:06 smithi078 ceph-mon[99959]: from='client.83156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:06 smithi078 ceph-mon[99959]: from='client.83162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3402612549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:06.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:06 smithi078 ceph-mon[91136]: from='client.83156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:06 smithi078 ceph-mon[91136]: from='client.83162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.660 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3402612549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:06 smithi146 ceph-mon[103278]: from='client.83156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:06 smithi146 ceph-mon[103278]: from='client.83162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3402612549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:07.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:07 smithi078 ceph-mon[91136]: from='client.63255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:07.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:07 smithi078 ceph-mon[91136]: pgmap v8464: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:07.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3425565329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:07.577 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:07 smithi078 ceph-mon[99959]: from='client.63255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:07.577 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:07 smithi078 ceph-mon[99959]: pgmap v8464: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:07.577 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3425565329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:07 smithi146 ceph-mon[103278]: from='client.63255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:07 smithi146 ceph-mon[103278]: pgmap v8464: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3425565329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:09 smithi146 ceph-mon[103278]: pgmap v8465: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:09 smithi078 ceph-mon[91136]: pgmap v8465: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:09 smithi078 ceph-mon[99959]: pgmap v8465: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:02:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T01:02:11.643 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:11.321Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:11.643 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:11.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:11.643 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:11.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:11 smithi146 ceph-mon[103278]: pgmap v8466: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:12.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:11 smithi078 ceph-mon[99959]: pgmap v8466: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:11 smithi078 ceph-mon[91136]: pgmap v8466: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:13 smithi146 ceph-mon[103278]: pgmap v8467: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:13 smithi078 ceph-mon[99959]: pgmap v8467: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:13 smithi078 ceph-mon[91136]: pgmap v8467: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:15 smithi146 ceph-mon[103278]: pgmap v8468: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[91136]: pgmap v8468: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[99959]: pgmap v8468: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:17 smithi146 ceph-mon[103278]: pgmap v8469: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:18.024 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:17 smithi078 ceph-mon[99959]: pgmap v8469: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:18.024 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:17 smithi078 ceph-mon[91136]: pgmap v8469: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[99959]: pgmap v8470: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[91136]: pgmap v8470: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:19 smithi146 ceph-mon[103278]: pgmap v8470: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:02:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:02:21.770 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:21.322Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:21.770 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:21.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:21.770 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:21.323Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:21 smithi078 ceph-mon[91136]: pgmap v8471: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:21 smithi078 ceph-mon[99959]: pgmap v8471: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:21 smithi146 ceph-mon[103278]: pgmap v8471: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:23 smithi078 ceph-mon[91136]: pgmap v8472: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:23 smithi078 ceph-mon[99959]: pgmap v8472: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:24.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:23 smithi146 ceph-mon[103278]: pgmap v8472: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:25.079 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:25 smithi078 ceph-mon[99959]: pgmap v8473: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:25 smithi078 ceph-mon[91136]: pgmap v8473: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:25 smithi146 ceph-mon[103278]: pgmap v8473: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:27 smithi078 ceph-mon[91136]: pgmap v8474: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:27 smithi078 ceph-mon[99959]: pgmap v8474: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:27 smithi146 ceph-mon[103278]: pgmap v8474: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:29 smithi146 ceph-mon[103278]: pgmap v8475: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:29 smithi078 ceph-mon[91136]: pgmap v8475: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:29 smithi078 ceph-mon[99959]: pgmap v8475: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:02:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:02:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:31.323Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:31.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:31.324Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:31 smithi146 ceph-mon[103278]: pgmap v8476: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:31 smithi078 ceph-mon[99959]: pgmap v8476: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:31 smithi078 ceph-mon[91136]: pgmap v8476: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:33 smithi078 ceph-mon[99959]: pgmap v8477: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:33 smithi078 ceph-mon[91136]: pgmap v8477: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:33 smithi146 ceph-mon[103278]: pgmap v8477: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:35.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:35.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:35 smithi146 ceph-mon[103278]: pgmap v8478: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[99959]: pgmap v8478: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[91136]: pgmap v8478: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:36.958 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 4h - - 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 4h - - 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 2s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 18s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 2s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 18s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 2s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 2s ago 4h 475M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 18s ago 4h 314M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:02:37.484 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 2s ago 4h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 2s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 18s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 2s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 2s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 2s ago 4h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 2s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 18s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 18s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 18s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 18s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 18s ago 4h 66.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 2s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 18s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:02:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 2s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:02:37.486 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 18s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:02:37.793 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:02:37.794 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:02:37.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:02:37.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:37.794 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:02:37.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:02:37.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:02:37.796 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:02:37.796 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:02:37.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:02:37.796 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:02:38.062 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:02:38.063 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:02:38.063 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:02:38.063 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:02:38.063 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[99959]: pgmap v8479: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:38.063 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[99959]: from='client.83186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:38.063 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[99959]: from='client.83192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:38.063 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1739898086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:38.063 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[91136]: pgmap v8479: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:38.063 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[91136]: from='client.83186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:38.064 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[91136]: from='client.83192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:38.064 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1739898086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:37 smithi146 ceph-mon[103278]: pgmap v8479: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:37 smithi146 ceph-mon[103278]: from='client.83186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:37 smithi146 ceph-mon[103278]: from='client.83192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1739898086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:02:38.372 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:02:38.372 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:38.372 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:02:38.372 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:38.372 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:02:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:38 smithi146 ceph-mon[103278]: from='client.83198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:38 smithi146 ceph-mon[103278]: from='client.83210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2014504182' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:38 smithi078 ceph-mon[91136]: from='client.83198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:38 smithi078 ceph-mon[91136]: from='client.83210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2014504182' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:38 smithi078 ceph-mon[99959]: from='client.83198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:38 smithi078 ceph-mon[99959]: from='client.83210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:02:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2014504182' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:02:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:39 smithi146 ceph-mon[103278]: pgmap v8480: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[99959]: pgmap v8480: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[91136]: pgmap v8480: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:02:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:02:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:41.324Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:41.325Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:41.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:41 smithi146 ceph-mon[103278]: pgmap v8481: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:41 smithi078 ceph-mon[99959]: pgmap v8481: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:41 smithi078 ceph-mon[91136]: pgmap v8481: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:43 smithi078 ceph-mon[99959]: pgmap v8482: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:43 smithi078 ceph-mon[91136]: pgmap v8482: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:43 smithi146 ceph-mon[103278]: pgmap v8482: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:45 smithi146 ceph-mon[103278]: pgmap v8483: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:45 smithi078 ceph-mon[91136]: pgmap v8483: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:45 smithi078 ceph-mon[99959]: pgmap v8483: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:48.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:47 smithi078 ceph-mon[99959]: pgmap v8484: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:48.141 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:47 smithi078 ceph-mon[91136]: pgmap v8484: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:47 smithi146 ceph-mon[103278]: pgmap v8484: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:49 smithi146 ceph-mon[103278]: pgmap v8485: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:49 smithi078 ceph-mon[91136]: pgmap v8485: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:49 smithi078 ceph-mon[99959]: pgmap v8485: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:02:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:02:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:51.325Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:51.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:02:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:02:51.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:02:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:51 smithi146 ceph-mon[103278]: pgmap v8486: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:51 smithi078 ceph-mon[99959]: pgmap v8486: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:51 smithi078 ceph-mon[91136]: pgmap v8486: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:53 smithi146 ceph-mon[103278]: pgmap v8487: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:53 smithi078 ceph-mon[99959]: pgmap v8487: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:53 smithi078 ceph-mon[91136]: pgmap v8487: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:55.241 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:55.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:02:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:56 smithi146 ceph-mon[103278]: pgmap v8488: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.279 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.280 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[91136]: pgmap v8488: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[99959]: pgmap v8488: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:02:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:02:58.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:58 smithi146 ceph-mon[103278]: pgmap v8489: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:58 smithi078 ceph-mon[91136]: pgmap v8489: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:58 smithi078 ceph-mon[99959]: pgmap v8489: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:02:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:02:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:00 smithi146 ceph-mon[103278]: pgmap v8490: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[91136]: pgmap v8490: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[99959]: pgmap v8490: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:03:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:03:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:01.326Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:01.326Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:01.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:01.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:02 smithi146 ceph-mon[103278]: pgmap v8491: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:02 smithi078 ceph-mon[91136]: pgmap v8491: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:02 smithi078 ceph-mon[99959]: pgmap v8491: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:04 smithi146 ceph-mon[103278]: pgmap v8492: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:04 smithi078 ceph-mon[99959]: pgmap v8492: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:04 smithi078 ceph-mon[91136]: pgmap v8492: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:05.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:06 smithi146 ceph-mon[103278]: pgmap v8493: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:06 smithi078 ceph-mon[99959]: pgmap v8493: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:06 smithi078 ceph-mon[91136]: pgmap v8493: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:08 smithi078 ceph-mon[91136]: pgmap v8494: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:08 smithi078 ceph-mon[99959]: pgmap v8494: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:08 smithi146 ceph-mon[103278]: pgmap v8494: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:08.672 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 4h - - 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 4h - - 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 14s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 9s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 14s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 9s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 14s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 14s ago 4h 475M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 9s ago 4h 315M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:03:09.234 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 14s ago 4h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 14s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 9s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 14s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 14s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 14s ago 4h 77.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 14s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 9s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 9s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 9s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 9s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 9s ago 4h 66.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 14s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 9s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:03:09.235 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 14s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:03:09.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 9s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:09.619 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:03:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:03:09.893 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:03:09.893 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:03:09.893 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:03:09.893 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:03:09.893 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:03:09.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:03:10.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[99959]: pgmap v8495: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:10.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[99959]: from='client.63303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[99959]: from='client.63309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[99959]: from='client.83234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2936591259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[91136]: pgmap v8495: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[91136]: from='client.63303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[91136]: from='client.63309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[91136]: from='client.83234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2936591259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:10.211 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:03:10.211 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:10.211 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:10.211 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:10.211 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:10 smithi146 ceph-mon[103278]: pgmap v8495: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:10 smithi146 ceph-mon[103278]: from='client.63303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:10 smithi146 ceph-mon[103278]: from='client.63309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:10 smithi146 ceph-mon[103278]: from='client.83234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2936591259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:03:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:03:11.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-mon[99959]: from='client.83246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:11.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3509528186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:11.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-mon[91136]: from='client.83246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:11.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3509528186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:11 smithi146 ceph-mon[103278]: from='client.83246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3509528186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:11.326Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:11.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:11.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:12 smithi146 ceph-mon[103278]: pgmap v8496: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:12 smithi078 ceph-mon[99959]: pgmap v8496: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:12 smithi078 ceph-mon[91136]: pgmap v8496: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:14 smithi146 ceph-mon[103278]: pgmap v8497: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:14 smithi078 ceph-mon[99959]: pgmap v8497: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:14 smithi078 ceph-mon[91136]: pgmap v8497: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:16 smithi146 ceph-mon[103278]: pgmap v8498: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[99959]: pgmap v8498: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[91136]: pgmap v8498: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:18.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:18 smithi146 ceph-mon[103278]: pgmap v8499: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:18 smithi078 ceph-mon[91136]: pgmap v8499: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:18 smithi078 ceph-mon[99959]: pgmap v8499: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:20 smithi146 ceph-mon[103278]: pgmap v8500: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[99959]: pgmap v8500: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[91136]: pgmap v8500: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:03:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:03:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:21.327Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:21.327Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:21.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:22 smithi146 ceph-mon[103278]: pgmap v8501: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:22 smithi078 ceph-mon[99959]: pgmap v8501: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:22 smithi078 ceph-mon[91136]: pgmap v8501: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:24 smithi146 ceph-mon[103278]: pgmap v8502: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:24 smithi078 ceph-mon[99959]: pgmap v8502: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:24 smithi078 ceph-mon[91136]: pgmap v8502: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:25.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:26 smithi146 ceph-mon[103278]: pgmap v8503: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:26 smithi078 ceph-mon[91136]: pgmap v8503: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:26 smithi078 ceph-mon[99959]: pgmap v8503: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:28.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:28 smithi078 ceph-mon[99959]: pgmap v8504: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:28.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:28 smithi078 ceph-mon[91136]: pgmap v8504: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:28 smithi146 ceph-mon[103278]: pgmap v8504: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:30 smithi146 ceph-mon[103278]: pgmap v8505: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:30 smithi078 ceph-mon[91136]: pgmap v8505: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:30 smithi078 ceph-mon[99959]: pgmap v8505: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:30.904 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:03:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:03:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:31.328Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:31.328Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:31.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:32 smithi146 ceph-mon[103278]: pgmap v8506: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:32 smithi078 ceph-mon[99959]: pgmap v8506: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:32 smithi078 ceph-mon[91136]: pgmap v8506: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:34 smithi146 ceph-mon[103278]: pgmap v8507: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:34 smithi078 ceph-mon[99959]: pgmap v8507: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:34 smithi078 ceph-mon[91136]: pgmap v8507: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:36 smithi146 ceph-mon[103278]: pgmap v8508: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[99959]: pgmap v8508: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[91136]: pgmap v8508: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:38 smithi146 ceph-mon[103278]: pgmap v8509: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:38 smithi078 ceph-mon[99959]: pgmap v8509: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:38 smithi078 ceph-mon[91136]: pgmap v8509: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:40.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[99959]: pgmap v8510: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:40.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[91136]: pgmap v8510: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:40.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.504 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:03:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:40 smithi146 ceph-mon[103278]: pgmap v8510: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:03:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 4h - - 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 4h - - 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 5s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 1s ago 4h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 5s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 1s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 5s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 5s ago 4h 477M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:03:41.030 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 1s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 5s ago 4h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 5s ago 4h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 1s ago 4h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 5s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 5s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 5s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 5s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 1s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 1s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 1s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 1s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 1s ago 4h 66.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 5s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:03:41.031 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 1s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:03:41.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 5s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:03:41.032 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 1s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:03:41.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:03:41.345 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:03:41.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:03:41.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:41.345 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:03:41.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:03:41.346 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:03:41.617 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:03:41.617 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:03:41.617 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:03:41.617 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:03:41.617 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:03:41.617 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:03:41.618 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:03:41.618 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:41.329Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:41.619 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:41.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:41.619 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:41.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:41.927 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:03:41.928 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:03:41.928 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:41.928 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:03:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:42 smithi146 ceph-mon[103278]: from='client.83258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:42 smithi146 ceph-mon[103278]: pgmap v8511: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:42 smithi146 ceph-mon[103278]: from='client.83264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:42 smithi146 ceph-mon[103278]: from='client.83270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1633183827' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1783248179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[91136]: from='client.83258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[91136]: pgmap v8511: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[91136]: from='client.83264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[91136]: from='client.83270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1633183827' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1783248179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[99959]: from='client.83258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[99959]: pgmap v8511: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[99959]: from='client.83264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[99959]: from='client.83270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1633183827' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:03:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1783248179' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:03:43.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:43 smithi078 ceph-mon[91136]: from='client.63363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:43.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:43 smithi078 ceph-mon[99959]: from='client.63363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:43 smithi146 ceph-mon[103278]: from='client.63363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:03:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:44 smithi146 ceph-mon[103278]: pgmap v8512: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:44 smithi078 ceph-mon[99959]: pgmap v8512: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:44 smithi078 ceph-mon[91136]: pgmap v8512: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:46 smithi146 ceph-mon[103278]: pgmap v8513: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:46 smithi078 ceph-mon[91136]: pgmap v8513: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:46 smithi078 ceph-mon[99959]: pgmap v8513: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:48 smithi146 ceph-mon[103278]: pgmap v8514: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:48 smithi078 ceph-mon[99959]: pgmap v8514: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:48 smithi078 ceph-mon[91136]: pgmap v8514: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:50.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:50 smithi078 ceph-mon[91136]: pgmap v8515: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:50.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:50 smithi078 ceph-mon[99959]: pgmap v8515: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:50 smithi146 ceph-mon[103278]: pgmap v8515: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:03:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:03:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:51 smithi078 ceph-mon[99959]: pgmap v8516: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:51 smithi078 ceph-mon[91136]: pgmap v8516: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:51.329Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:51.329Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:03:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:03:51.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:03:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:51 smithi146 ceph-mon[103278]: pgmap v8516: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:53 smithi146 ceph-mon[103278]: pgmap v8517: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:54.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:53 smithi078 ceph-mon[99959]: pgmap v8517: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:54.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:53 smithi078 ceph-mon[91136]: pgmap v8517: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:03:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:55 smithi146 ceph-mon[103278]: pgmap v8518: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[99959]: pgmap v8518: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[91136]: pgmap v8518: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:03:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:03:58.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:57 smithi146 ceph-mon[103278]: pgmap v8519: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:57 smithi078 ceph-mon[91136]: pgmap v8519: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:57 smithi078 ceph-mon[99959]: pgmap v8519: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:03:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:03:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:03:59 smithi146 ceph-mon[103278]: pgmap v8520: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:03:59 smithi078 ceph-mon[91136]: pgmap v8520: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:03:59 smithi078 ceph-mon[99959]: pgmap v8520: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:04:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:04:01.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.330 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:01.330Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:01.330Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:01.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:02 smithi146 ceph-mon[103278]: pgmap v8521: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:02 smithi078 ceph-mon[91136]: pgmap v8521: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:02 smithi078 ceph-mon[99959]: pgmap v8521: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:04 smithi146 ceph-mon[103278]: pgmap v8522: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:04 smithi078 ceph-mon[91136]: pgmap v8522: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:04 smithi078 ceph-mon[99959]: pgmap v8522: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:06 smithi146 ceph-mon[103278]: pgmap v8523: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:06 smithi078 ceph-mon[91136]: pgmap v8523: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:06 smithi078 ceph-mon[99959]: pgmap v8523: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:08 smithi146 ceph-mon[103278]: pgmap v8524: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:08 smithi078 ceph-mon[91136]: pgmap v8524: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:08 smithi078 ceph-mon[99959]: pgmap v8524: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:10.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:10 smithi078 ceph-mon[99959]: pgmap v8525: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:10.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:10 smithi078 ceph-mon[91136]: pgmap v8525: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:10 smithi146 ceph-mon[103278]: pgmap v8525: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:04:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:04:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:11.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:11.331Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:11.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:12.219 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:04:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:12 smithi146 ceph-mon[103278]: pgmap v8526: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:12 smithi078 ceph-mon[99959]: pgmap v8526: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:12 smithi078 ceph-mon[91136]: pgmap v8526: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 4h - - 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 4h - - 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 17s ago 4h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 12s ago 4h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 17s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 12s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 17s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 17s ago 4h 477M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:04:12.757 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 12s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 17s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 17s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 12s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 17s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 17s ago 4h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 17s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 17s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 12s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 12s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 12s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 12s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 12s ago 4h 66.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 17s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:04:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 12s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:04:12.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 17s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:04:12.759 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 12s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:04:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:04:13.078 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:04:13.363 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:04:13.363 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:04:13.363 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:04:13.363 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:04:13.363 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:04:13.364 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:04:13.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:13 smithi078 ceph-mon[99959]: from='client.83294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:13.364 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2153179888' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:13.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:13 smithi078 ceph-mon[91136]: from='client.83294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:13.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2153179888' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:13 smithi146 ceph-mon[103278]: from='client.83294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2153179888' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:13.681 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:04:13.681 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:13.681 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:13.681 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:13.681 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:14 smithi146 ceph-mon[103278]: from='client.63381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:14 smithi146 ceph-mon[103278]: pgmap v8527: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:14 smithi146 ceph-mon[103278]: from='client.83306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1455089499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[91136]: from='client.63381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[91136]: pgmap v8527: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[91136]: from='client.83306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1455089499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[99959]: from='client.63381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[99959]: pgmap v8527: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[99959]: from='client.83306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1455089499' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:15 smithi146 ceph-mon[103278]: from='client.83318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:15 smithi078 ceph-mon[99959]: from='client.83318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:15 smithi078 ceph-mon[91136]: from='client.83318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:16 smithi146 ceph-mon[103278]: pgmap v8528: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[99959]: pgmap v8528: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[91136]: pgmap v8528: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:18.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:18 smithi146 ceph-mon[103278]: pgmap v8529: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:18 smithi078 ceph-mon[91136]: pgmap v8529: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:18 smithi078 ceph-mon[99959]: pgmap v8529: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:20.374 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:20 smithi146 ceph-mon[103278]: pgmap v8530: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:20 smithi078 ceph-mon[91136]: pgmap v8530: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:20 smithi078 ceph-mon[99959]: pgmap v8530: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:04:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:04:21.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.332 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:21.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:21.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:21.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:22 smithi146 ceph-mon[103278]: pgmap v8531: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:22 smithi078 ceph-mon[91136]: pgmap v8531: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:22 smithi078 ceph-mon[99959]: pgmap v8531: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:24 smithi146 ceph-mon[103278]: pgmap v8532: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:24 smithi078 ceph-mon[99959]: pgmap v8532: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:24 smithi078 ceph-mon[91136]: pgmap v8532: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:25.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:26.216 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 01:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T01:04:25.847407933Z level=info msg="Completed cleanup jobs" duration=12.170234ms 2024-09-21T01:04:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:26 smithi146 ceph-mon[103278]: pgmap v8533: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:26 smithi078 ceph-mon[99959]: pgmap v8533: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:26 smithi078 ceph-mon[91136]: pgmap v8533: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:28 smithi146 ceph-mon[103278]: pgmap v8534: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:28 smithi078 ceph-mon[91136]: pgmap v8534: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:28 smithi078 ceph-mon[99959]: pgmap v8534: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:30.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:30 smithi078 ceph-mon[99959]: pgmap v8535: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:30.480 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:30 smithi078 ceph-mon[91136]: pgmap v8535: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:30 smithi146 ceph-mon[103278]: pgmap v8535: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:04:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:04:31.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:31.331Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:31.484 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:31.332Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:31.484 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:31.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:32.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:32 smithi078 ceph-mon[91136]: pgmap v8536: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:32.379 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:32 smithi078 ceph-mon[99959]: pgmap v8536: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:32 smithi146 ceph-mon[103278]: pgmap v8536: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:34 smithi146 ceph-mon[103278]: pgmap v8537: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:34 smithi078 ceph-mon[91136]: pgmap v8537: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:34 smithi078 ceph-mon[99959]: pgmap v8537: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:04:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:36 smithi146 ceph-mon[103278]: pgmap v8538: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[99959]: pgmap v8538: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[91136]: pgmap v8538: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:36.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:38 smithi146 ceph-mon[103278]: pgmap v8539: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:38 smithi078 ceph-mon[99959]: pgmap v8539: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:38 smithi078 ceph-mon[91136]: pgmap v8539: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:40.347 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:40 smithi146 ceph-mon[103278]: pgmap v8540: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:40 smithi078 ceph-mon[99959]: pgmap v8540: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:40 smithi078 ceph-mon[91136]: pgmap v8540: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:04:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:04:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:41 smithi146 ceph-mon[103278]: pgmap v8541: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[99959]: pgmap v8541: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-mon[91136]: pgmap v8541: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:41.333Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:41.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:41.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:41.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:43.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:43 smithi078 ceph-mon[99959]: pgmap v8542: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:43.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:43.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:43 smithi078 ceph-mon[91136]: pgmap v8542: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:43.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:43.974 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:04:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:43 smithi146 ceph-mon[103278]: pgmap v8542: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 4h - - 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 4h - - 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 8s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 4s ago 4h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 8s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:04:44.498 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 4s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 8s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 8s ago 4h 478M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 4s ago 4h 316M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 8s ago 4h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 8s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 4s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 8s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 8s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 8s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 8s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 4s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:04:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 4s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 4s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 4s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 4s ago 4h 66.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 8s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 4s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 8s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:04:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 4s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:04:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:44 smithi078 ceph-mon[91136]: from='client.83330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:44 smithi078 ceph-mon[91136]: from='client.83336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:44 smithi078 ceph-mon[99959]: from='client.83330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:44 smithi078 ceph-mon[99959]: from='client.83336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:44.816 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:04:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:04:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:04:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:04:44.817 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:04:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:44 smithi146 ceph-mon[103278]: from='client.83330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:44 smithi146 ceph-mon[103278]: from='client.83336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:45.086 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:04:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:04:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:04:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:04:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:04:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:04:45.087 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:04:45.397 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:04:45.397 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:04:45.397 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:04:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:45 smithi146 ceph-mon[103278]: from='client.63423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:45 smithi146 ceph-mon[103278]: pgmap v8543: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4197044716' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:45 smithi146 ceph-mon[103278]: from='client.63435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2068163900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[99959]: from='client.63423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[99959]: pgmap v8543: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4197044716' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[99959]: from='client.63435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2068163900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[91136]: from='client.63423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[91136]: pgmap v8543: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4197044716' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[91136]: from='client.63435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:04:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2068163900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:04:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:47 smithi146 ceph-mon[103278]: pgmap v8544: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:47 smithi078 ceph-mon[91136]: pgmap v8544: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:47 smithi078 ceph-mon[99959]: pgmap v8544: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:49 smithi146 ceph-mon[103278]: pgmap v8545: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:49 smithi078 ceph-mon[91136]: pgmap v8545: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:49 smithi078 ceph-mon[99959]: pgmap v8545: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:04:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:04:51.599 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:51.333Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:51.599 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:51.333Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:04:51.599 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:04:51.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:04:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:51 smithi146 ceph-mon[103278]: pgmap v8546: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:52.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:51 smithi078 ceph-mon[99959]: pgmap v8546: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:51 smithi078 ceph-mon[91136]: pgmap v8546: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:53.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:53 smithi078 ceph-mon[91136]: pgmap v8547: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:53.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:53 smithi078 ceph-mon[99959]: pgmap v8547: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:53 smithi146 ceph-mon[103278]: pgmap v8547: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:04:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:55 smithi146 ceph-mon[103278]: pgmap v8548: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:55 smithi078 ceph-mon[91136]: pgmap v8548: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:55 smithi078 ceph-mon[99959]: pgmap v8548: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:04:57.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:04:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:57 smithi146 ceph-mon[103278]: pgmap v8549: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:57 smithi078 ceph-mon[91136]: pgmap v8549: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:57 smithi078 ceph-mon[99959]: pgmap v8549: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:04:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:04:59.838 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:04:59 smithi146 ceph-mon[103278]: pgmap v8550: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:04:59 smithi078 ceph-mon[91136]: pgmap v8550: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:04:59 smithi078 ceph-mon[99959]: pgmap v8550: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:05:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:05:01.734 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:01 smithi146 ceph-mon[103278]: pgmap v8551: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[99959]: pgmap v8551: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:01.334Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:01.334Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:01.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:01 smithi078 ceph-mon[91136]: pgmap v8551: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:03 smithi146 ceph-mon[103278]: pgmap v8552: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:03 smithi078 ceph-mon[91136]: pgmap v8552: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:03 smithi078 ceph-mon[99959]: pgmap v8552: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:05 smithi146 ceph-mon[103278]: pgmap v8553: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:05 smithi078 ceph-mon[91136]: pgmap v8553: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:05 smithi078 ceph-mon[99959]: pgmap v8553: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:07 smithi146 ceph-mon[103278]: pgmap v8554: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:07 smithi078 ceph-mon[91136]: pgmap v8554: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:07 smithi078 ceph-mon[99959]: pgmap v8554: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:09 smithi146 ceph-mon[103278]: pgmap v8555: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:09 smithi078 ceph-mon[91136]: pgmap v8555: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:09 smithi078 ceph-mon[99959]: pgmap v8555: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:05:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:05:11.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:11.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:11.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:11.335Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:11.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:11.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:11 smithi146 ceph-mon[103278]: pgmap v8556: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:12.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:11 smithi078 ceph-mon[91136]: pgmap v8556: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:11 smithi078 ceph-mon[99959]: pgmap v8556: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:13.847 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:13 smithi078 ceph-mon[91136]: pgmap v8557: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:13.847 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:13.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:13 smithi078 ceph-mon[99959]: pgmap v8557: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:13.847 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:13 smithi146 ceph-mon[103278]: pgmap v8557: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:15.690 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:05:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:15 smithi146 ceph-mon[103278]: pgmap v8558: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:15 smithi078 ceph-mon[99959]: pgmap v8558: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:15 smithi078 ceph-mon[91136]: pgmap v8558: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:05:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 4h - - 2024-09-21T01:05:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 4h - - 2024-09-21T01:05:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 0s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:05:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 15s ago 4h 88.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:05:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 0s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 15s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 0s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 0s ago 4h 479M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 15s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 0s ago 4h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 0s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 15s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 0s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 0s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 0s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 0s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:05:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 15s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 15s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 15s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 15s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 15s ago 4h 66.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 0s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 15s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 0s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:05:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 15s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:05:16.534 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:05:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:05:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:05:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:05:16.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:05:16.806 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:05:16.807 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:05:16.807 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:05:16.807 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:05:16.807 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:05:17.122 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:05:17.122 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:17.123 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='client.63447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='client.83372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='client.63459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3224706480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='client.63447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='client.83372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='client.63459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3224706480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='client.63447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='client.83372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='client.63459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3224706480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:17 smithi146 ceph-mon[103278]: pgmap v8559: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:17 smithi146 ceph-mon[103278]: from='client.63471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2587276771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:17 smithi078 ceph-mon[99959]: pgmap v8559: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:17 smithi078 ceph-mon[99959]: from='client.63471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2587276771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:17 smithi078 ceph-mon[91136]: pgmap v8559: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:17 smithi078 ceph-mon[91136]: from='client.63471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2587276771' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:20.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:19 smithi146 ceph-mon[103278]: pgmap v8560: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:19 smithi078 ceph-mon[91136]: pgmap v8560: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:19 smithi078 ceph-mon[99959]: pgmap v8560: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:05:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:05:21.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[91136]: pgmap v8561: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-mon[99959]: pgmap v8561: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:21.335Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:21.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:21.336Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:22.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:22.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:22.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:22.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:21 smithi146 ceph-mon[103278]: pgmap v8561: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:23 smithi146 ceph-mon[103278]: pgmap v8562: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:24.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:23 smithi078 ceph-mon[99959]: pgmap v8562: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:23 smithi078 ceph-mon[91136]: pgmap v8562: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:25 smithi146 ceph-mon[103278]: pgmap v8563: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:25 smithi078 ceph-mon[91136]: pgmap v8563: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:25 smithi078 ceph-mon[99959]: pgmap v8563: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:27 smithi146 ceph-mon[103278]: pgmap v8564: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:27 smithi078 ceph-mon[91136]: pgmap v8564: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:27 smithi078 ceph-mon[99959]: pgmap v8564: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:29 smithi146 ceph-mon[103278]: pgmap v8565: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:29 smithi078 ceph-mon[91136]: pgmap v8565: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:29 smithi078 ceph-mon[99959]: pgmap v8565: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:05:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:05:31.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:31.336Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:31.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:31.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:31.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:31.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:31 smithi146 ceph-mon[103278]: pgmap v8566: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:31 smithi078 ceph-mon[99959]: pgmap v8566: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:31 smithi078 ceph-mon[91136]: pgmap v8566: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:33 smithi146 ceph-mon[103278]: pgmap v8567: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:34.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:33 smithi078 ceph-mon[99959]: pgmap v8567: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:34.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:33 smithi078 ceph-mon[91136]: pgmap v8567: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:35 smithi146 ceph-mon[103278]: pgmap v8568: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:35 smithi078 ceph-mon[99959]: pgmap v8568: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:35 smithi078 ceph-mon[91136]: pgmap v8568: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:38 smithi146 ceph-mon[103278]: pgmap v8569: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:38 smithi078 ceph-mon[91136]: pgmap v8569: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:38 smithi078 ceph-mon[99959]: pgmap v8569: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:40 smithi146 ceph-mon[103278]: pgmap v8570: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:40 smithi078 ceph-mon[91136]: pgmap v8570: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:40 smithi078 ceph-mon[99959]: pgmap v8570: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:05:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:05:41.713 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:41.337Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:41.714 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:41.337Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:41.714 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:41.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:41.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:41 smithi146 ceph-mon[103278]: pgmap v8571: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:41.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:41.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:41.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[91136]: pgmap v8571: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[99959]: pgmap v8571: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:43 smithi146 ceph-mon[103278]: pgmap v8572: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:43 smithi078 ceph-mon[91136]: pgmap v8572: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:43 smithi078 ceph-mon[99959]: pgmap v8572: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:45 smithi146 ceph-mon[103278]: pgmap v8573: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:45 smithi078 ceph-mon[91136]: pgmap v8573: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:45 smithi078 ceph-mon[99959]: pgmap v8573: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:47.410 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 4h - - 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 4h - - 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 11s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 7s ago 4h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 11s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 7s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:05:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 11s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 11s ago 4h 479M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 7s ago 4h 318M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 11s ago 4h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 11s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 7s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 11s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 11s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 11s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 11s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 7s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 7s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 7s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:05:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 7s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:05:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 7s ago 4h 67.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:05:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 11s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:05:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 7s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:05:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 11s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:05:47.938 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 7s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:05:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:47 smithi146 ceph-mon[103278]: pgmap v8574: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:47 smithi078 ceph-mon[99959]: pgmap v8574: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:47 smithi078 ceph-mon[91136]: pgmap v8574: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:05:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:05:48.249 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:05:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:05:48.518 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:05:48.518 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:05:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[99959]: from='client.83402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[99959]: from='client.63489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[99959]: from='client.63495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/486964102' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[91136]: from='client.83402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[91136]: from='client.63489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[91136]: from='client.63495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/486964102' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:48.831 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:05:48.831 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:48.831 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:05:48.831 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:48.831 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:05:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:48 smithi146 ceph-mon[103278]: from='client.83402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:48 smithi146 ceph-mon[103278]: from='client.63489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:48 smithi146 ceph-mon[103278]: from='client.63495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/486964102' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:05:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:49 smithi146 ceph-mon[103278]: from='client.83426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:49 smithi146 ceph-mon[103278]: pgmap v8575: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2790333938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:49 smithi078 ceph-mon[99959]: from='client.83426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:49 smithi078 ceph-mon[99959]: pgmap v8575: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2790333938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:49 smithi078 ceph-mon[91136]: from='client.83426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:05:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:49 smithi078 ceph-mon[91136]: pgmap v8575: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2790333938' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:05:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:05:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:05:51.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:51.337Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:51.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:51.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:05:51.737 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:05:51.338Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:05:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:51 smithi146 ceph-mon[103278]: pgmap v8576: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:51 smithi078 ceph-mon[91136]: pgmap v8576: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:52.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:51 smithi078 ceph-mon[99959]: pgmap v8576: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:53.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:53 smithi078 ceph-mon[91136]: pgmap v8577: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:53.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:53 smithi078 ceph-mon[99959]: pgmap v8577: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:53 smithi146 ceph-mon[103278]: pgmap v8577: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:05:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:55 smithi146 ceph-mon[103278]: pgmap v8578: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:55 smithi078 ceph-mon[91136]: pgmap v8578: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:55 smithi078 ceph-mon[99959]: pgmap v8578: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:05:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:05:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:58 smithi146 ceph-mon[103278]: pgmap v8579: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:58 smithi078 ceph-mon[99959]: pgmap v8579: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:58 smithi078 ceph-mon[91136]: pgmap v8579: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:05:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:05:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:05:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:05:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:05:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:00 smithi146 ceph-mon[103278]: pgmap v8580: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:00 smithi078 ceph-mon[91136]: pgmap v8580: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:00 smithi078 ceph-mon[99959]: pgmap v8580: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:06:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:06:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:01.338Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:01.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:01.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:02.093 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:01 smithi146 ceph-mon[103278]: pgmap v8581: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:02.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[91136]: pgmap v8581: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[99959]: pgmap v8581: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:03 smithi146 ceph-mon[103278]: pgmap v8582: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:03 smithi078 ceph-mon[91136]: pgmap v8582: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:03 smithi078 ceph-mon[99959]: pgmap v8582: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:05 smithi146 ceph-mon[103278]: pgmap v8583: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:05 smithi078 ceph-mon[91136]: pgmap v8583: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:05 smithi078 ceph-mon[99959]: pgmap v8583: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:07 smithi146 ceph-mon[103278]: pgmap v8584: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:07 smithi078 ceph-mon[91136]: pgmap v8584: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:07 smithi078 ceph-mon[99959]: pgmap v8584: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:09 smithi146 ceph-mon[103278]: pgmap v8585: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:09 smithi078 ceph-mon[91136]: pgmap v8585: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:09 smithi078 ceph-mon[99959]: pgmap v8585: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:06:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:06:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:11.339Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:11.339Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:11.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:11 smithi146 ceph-mon[103278]: pgmap v8586: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:11 smithi078 ceph-mon[99959]: pgmap v8586: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:11 smithi078 ceph-mon[91136]: pgmap v8586: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:13 smithi146 ceph-mon[103278]: pgmap v8587: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:13 smithi078 ceph-mon[99959]: pgmap v8587: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:13 smithi078 ceph-mon[91136]: pgmap v8587: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:15.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:15.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:16.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:15 smithi078 ceph-mon[91136]: pgmap v8588: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:16.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:15 smithi078 ceph-mon[99959]: pgmap v8588: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:15 smithi146 ceph-mon[103278]: pgmap v8588: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:17.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.371 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:18 smithi146 ceph-mon[103278]: pgmap v8589: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:18 smithi078 ceph-mon[91136]: pgmap v8589: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:18 smithi078 ceph-mon[99959]: pgmap v8589: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:19.126 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 4h - - 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 4h - - 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 3s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 18s ago 4h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 3s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 18s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:06:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 3s ago 4h 479M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 18s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 3s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 18s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 18s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 18s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 18s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 18s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:06:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 18s ago 4h 67.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:06:19.657 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:06:19.657 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 18s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:06:19.657 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:06:19.657 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 18s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:19.965 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:06:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:06:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:06:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:06:19.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:06:19.966 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:06:20.237 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:06:20.237 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:06:20.237 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:06:20.237 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:06:20.238 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:06:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-mon[99959]: pgmap v8590: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-mon[99959]: from='client.83438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/373513645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-mon[91136]: pgmap v8590: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-mon[91136]: from='client.83438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/373513645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:20.547 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:06:20.547 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:20.547 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:20.547 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:20.547 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:20 smithi146 ceph-mon[103278]: pgmap v8590: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:20 smithi146 ceph-mon[103278]: from='client.83438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/373513645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:06:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:06:21.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='client.63525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='client.63531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='client.83462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1783817988' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:21.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.657 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='client.63525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='client.63531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='client.83462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1783817988' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='client.63525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='client.63531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='client.83462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1783817988' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:21.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:21.340Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:21.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:21.340Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:21.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:21.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:22.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:22 smithi146 ceph-mon[103278]: pgmap v8591: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:22 smithi078 ceph-mon[91136]: pgmap v8591: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:22 smithi078 ceph-mon[99959]: pgmap v8591: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:24 smithi146 ceph-mon[103278]: pgmap v8592: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:24 smithi078 ceph-mon[91136]: pgmap v8592: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:24 smithi078 ceph-mon[99959]: pgmap v8592: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:26 smithi146 ceph-mon[103278]: pgmap v8593: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:26 smithi078 ceph-mon[99959]: pgmap v8593: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:26 smithi078 ceph-mon[91136]: pgmap v8593: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:28 smithi146 ceph-mon[103278]: pgmap v8594: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:28 smithi078 ceph-mon[91136]: pgmap v8594: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:28 smithi078 ceph-mon[99959]: pgmap v8594: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:30 smithi078 ceph-mon[91136]: pgmap v8595: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:30.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:30 smithi078 ceph-mon[99959]: pgmap v8595: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:30 smithi146 ceph-mon[103278]: pgmap v8595: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:06:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:06:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:31 smithi146 ceph-mon[103278]: pgmap v8596: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:31.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:31 smithi078 ceph-mon[99959]: pgmap v8596: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:31 smithi078 ceph-mon[91136]: pgmap v8596: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:31.340Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:31.341Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:31.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:33 smithi146 ceph-mon[103278]: pgmap v8597: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:33 smithi078 ceph-mon[91136]: pgmap v8597: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:33 smithi078 ceph-mon[99959]: pgmap v8597: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:35.861 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:35 smithi078 ceph-mon[99959]: pgmap v8598: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:35.861 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:35 smithi078 ceph-mon[91136]: pgmap v8598: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:35 smithi146 ceph-mon[103278]: pgmap v8598: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:37 smithi146 ceph-mon[103278]: pgmap v8599: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[91136]: pgmap v8599: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:37 smithi078 ceph-mon[99959]: pgmap v8599: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:39 smithi146 ceph-mon[103278]: pgmap v8600: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:39 smithi078 ceph-mon[91136]: pgmap v8600: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:40.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:39 smithi078 ceph-mon[99959]: pgmap v8600: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:06:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:06:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:41.342Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:41.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:41.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:42.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:42 smithi146 ceph-mon[103278]: pgmap v8601: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:42.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[91136]: pgmap v8601: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[99959]: pgmap v8601: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:44.185 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:44 smithi146 ceph-mon[103278]: pgmap v8602: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:44.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:44 smithi078 ceph-mon[99959]: pgmap v8602: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:44 smithi078 ceph-mon[91136]: pgmap v8602: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:46 smithi146 ceph-mon[103278]: pgmap v8603: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:46 smithi078 ceph-mon[91136]: pgmap v8603: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:46 smithi078 ceph-mon[99959]: pgmap v8603: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:48 smithi146 ceph-mon[103278]: pgmap v8604: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:48 smithi078 ceph-mon[91136]: pgmap v8604: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:48 smithi078 ceph-mon[99959]: pgmap v8604: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:50 smithi146 ceph-mon[103278]: pgmap v8605: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:50 smithi078 ceph-mon[91136]: pgmap v8605: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:50 smithi078 ceph-mon[99959]: pgmap v8605: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:50.841 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:06:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:06:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:06:51.371 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:06:51.371 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 4h - - 2024-09-21T01:06:51.371 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 4h - - 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 14s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 10s ago 4h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 14s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 10s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 14s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 14s ago 4h 479M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 10s ago 4h 317M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 14s ago 4h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 14s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 10s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:06:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 14s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 14s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 14s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 14s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 4h 67.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 14s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 14s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:06:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:06:51.683 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:06:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:06:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:06:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:06:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:06:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:06:51.685 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:06:51.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:51.342Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:51.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:51.342Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:06:51.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:06:51.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:06:51.949 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:06:51.950 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:06:51.951 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:06:52.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[99959]: pgmap v8606: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:52.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[99959]: from='client.63555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:52.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[99959]: from='client.63561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:52.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3404151189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:52.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[91136]: pgmap v8606: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:52.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[91136]: from='client.63555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:52.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[91136]: from='client.63561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:52.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3404151189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:52.273 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:06:52.273 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:06:52.273 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:06:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:52 smithi146 ceph-mon[103278]: pgmap v8606: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:52 smithi146 ceph-mon[103278]: from='client.63555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:52 smithi146 ceph-mon[103278]: from='client.63561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3404151189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:06:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:53 smithi146 ceph-mon[103278]: from='client.83486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:53 smithi146 ceph-mon[103278]: from='client.63579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/189455298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:53.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:53 smithi078 ceph-mon[91136]: from='client.83486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:53 smithi078 ceph-mon[91136]: from='client.63579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/189455298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:53 smithi078 ceph-mon[99959]: from='client.83486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:53 smithi078 ceph-mon[99959]: from='client.63579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:06:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/189455298' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:06:54.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:54 smithi146 ceph-mon[103278]: pgmap v8607: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:54 smithi078 ceph-mon[99959]: pgmap v8607: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:54 smithi078 ceph-mon[91136]: pgmap v8607: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:06:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:06:56.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:56 smithi078 ceph-mon[91136]: pgmap v8608: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:56.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:56 smithi078 ceph-mon[99959]: pgmap v8608: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:56 smithi146 ceph-mon[103278]: pgmap v8608: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:06:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:57 smithi146 ceph-mon[103278]: pgmap v8609: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[91136]: pgmap v8609: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:06:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:57 smithi078 ceph-mon[99959]: pgmap v8609: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:06:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:06:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:06:59 smithi146 ceph-mon[103278]: pgmap v8610: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:06:59 smithi078 ceph-mon[91136]: pgmap v8610: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:06:59 smithi078 ceph-mon[99959]: pgmap v8610: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:07:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:07:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:01.342Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:01.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:01.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:02 smithi146 ceph-mon[103278]: pgmap v8611: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[91136]: pgmap v8611: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[99959]: pgmap v8611: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:04.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:04 smithi146 ceph-mon[103278]: pgmap v8612: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:04 smithi078 ceph-mon[99959]: pgmap v8612: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:04 smithi078 ceph-mon[91136]: pgmap v8612: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:05.373 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:06 smithi146 ceph-mon[103278]: pgmap v8613: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:06 smithi078 ceph-mon[91136]: pgmap v8613: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:06 smithi078 ceph-mon[99959]: pgmap v8613: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:08 smithi146 ceph-mon[103278]: pgmap v8614: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:08 smithi078 ceph-mon[91136]: pgmap v8614: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:08 smithi078 ceph-mon[99959]: pgmap v8614: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:10 smithi146 ceph-mon[103278]: pgmap v8615: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:10 smithi078 ceph-mon[91136]: pgmap v8615: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:10 smithi078 ceph-mon[99959]: pgmap v8615: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:07:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:07:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:11.343Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:11.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:11.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:12 smithi146 ceph-mon[103278]: pgmap v8616: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:12 smithi078 ceph-mon[99959]: pgmap v8616: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:12 smithi078 ceph-mon[91136]: pgmap v8616: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:14 smithi078 ceph-mon[99959]: pgmap v8617: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:14 smithi078 ceph-mon[91136]: pgmap v8617: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:14 smithi146 ceph-mon[103278]: pgmap v8617: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:16 smithi078 ceph-mon[91136]: pgmap v8618: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:16 smithi078 ceph-mon[99959]: pgmap v8618: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:16 smithi146 ceph-mon[103278]: pgmap v8618: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:17 smithi146 ceph-mon[103278]: pgmap v8619: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[91136]: pgmap v8619: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[99959]: pgmap v8619: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:19 smithi146 ceph-mon[103278]: pgmap v8620: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:19 smithi078 ceph-mon[91136]: pgmap v8620: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:19 smithi078 ceph-mon[99959]: pgmap v8620: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:07:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:07:21.712 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:21.343Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:21.712 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:21.343Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:21.712 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:21.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:21 smithi146 ceph-mon[103278]: pgmap v8621: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[91136]: pgmap v8621: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[99959]: pgmap v8621: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:22.564 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 4h - - 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 4h - - 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 1s ago 4h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 1s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:07:23.096 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 6s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 6s ago 4h 481M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 1s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 6s ago 4h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 1s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 1s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 1s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:07:23.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 1s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:07:23.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 1s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:07:23.098 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 1s ago 4h 66.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:07:23.098 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:07:23.098 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 1s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:07:23.098 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:07:23.098 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 1s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:07:23.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:07:23.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:07:23.674 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:07:23.675 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:07:23.676 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:07:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:23 smithi146 ceph-mon[103278]: from='client.63591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:23 smithi146 ceph-mon[103278]: pgmap v8622: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:23 smithi146 ceph-mon[103278]: from='client.63597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:23 smithi146 ceph-mon[103278]: from='client.83522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2990636582' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:23.981 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:07:23.981 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:23.981 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:23.981 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:23.981 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[91136]: from='client.63591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[91136]: pgmap v8622: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[91136]: from='client.63597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[91136]: from='client.83522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2990636582' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[99959]: from='client.63591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[99959]: pgmap v8622: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[99959]: from='client.63597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[99959]: from='client.83522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:23.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2990636582' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:24.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:24 smithi146 ceph-mon[103278]: from='client.63615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:24.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:24.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2805162344' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:24 smithi078 ceph-mon[91136]: from='client.63615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2805162344' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:24 smithi078 ceph-mon[99959]: from='client.63615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2805162344' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:25.919 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:25 smithi146 ceph-mon[103278]: pgmap v8623: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:25 smithi078 ceph-mon[91136]: pgmap v8623: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:25 smithi078 ceph-mon[99959]: pgmap v8623: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:27 smithi146 ceph-mon[103278]: pgmap v8624: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:27 smithi078 ceph-mon[99959]: pgmap v8624: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:27 smithi078 ceph-mon[91136]: pgmap v8624: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:29 smithi146 ceph-mon[103278]: pgmap v8625: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:07:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:29 smithi078 ceph-mon[91136]: pgmap v8625: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:07:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:29 smithi078 ceph-mon[99959]: pgmap v8625: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:07:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:07:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:07:31.734 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:31.344Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:31.734 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:31.344Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:31.734 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:31.345Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:31 smithi146 ceph-mon[103278]: pgmap v8626: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T01:07:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:31 smithi078 ceph-mon[91136]: pgmap v8626: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T01:07:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:31 smithi078 ceph-mon[99959]: pgmap v8626: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T01:07:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:33 smithi146 ceph-mon[103278]: pgmap v8627: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:33 smithi078 ceph-mon[91136]: pgmap v8627: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:33 smithi078 ceph-mon[99959]: pgmap v8627: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:35 smithi146 ceph-mon[103278]: pgmap v8628: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:07:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:35 smithi078 ceph-mon[91136]: pgmap v8628: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:07:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:35 smithi078 ceph-mon[99959]: pgmap v8628: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[91136]: pgmap v8629: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[99959]: pgmap v8629: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:37 smithi146 ceph-mon[103278]: pgmap v8629: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:39 smithi078 ceph-mon[91136]: pgmap v8630: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:39 smithi078 ceph-mon[99959]: pgmap v8630: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:39 smithi146 ceph-mon[103278]: pgmap v8630: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:07:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:07:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:41.345Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:41.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:41.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:41 smithi146 ceph-mon[103278]: pgmap v8631: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[91136]: pgmap v8631: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[99959]: pgmap v8631: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T01:07:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:44.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:43 smithi146 ceph-mon[103278]: pgmap v8632: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:44.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:43 smithi078 ceph-mon[91136]: pgmap v8632: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:43 smithi078 ceph-mon[99959]: pgmap v8632: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:44.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:46.139 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:45 smithi146 ceph-mon[103278]: pgmap v8633: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:45 smithi078 ceph-mon[91136]: pgmap v8633: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:45 smithi078 ceph-mon[99959]: pgmap v8633: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:47 smithi146 ceph-mon[103278]: pgmap v8634: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:47 smithi078 ceph-mon[91136]: pgmap v8634: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:47 smithi078 ceph-mon[99959]: pgmap v8634: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:49 smithi146 ceph-mon[103278]: pgmap v8635: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:49 smithi078 ceph-mon[91136]: pgmap v8635: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:49 smithi078 ceph-mon[99959]: pgmap v8635: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T01:07:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:07:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:07:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:51.346Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:51.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:07:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:07:51.346Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:07:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:51 smithi078 ceph-mon[91136]: pgmap v8636: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:51 smithi078 ceph-mon[99959]: pgmap v8636: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:51 smithi146 ceph-mon[103278]: pgmap v8636: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T01:07:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:53 smithi146 ceph-mon[103278]: pgmap v8637: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:54.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:53 smithi078 ceph-mon[91136]: pgmap v8637: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:54.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:53 smithi078 ceph-mon[99959]: pgmap v8637: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:54.274 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 4h - - 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 4h - - 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 17s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 13s ago 4h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 17s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:07:54.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 13s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 17s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 17s ago 4h 482M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 13s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 17s ago 4h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 17s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 13s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 17s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 17s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 17s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 17s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:07:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 13s ago 4h 66.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:07:54.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 17s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:07:54.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:07:54.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 17s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:07:54.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:07:55.109 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:07:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:07:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:07:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:07:55.110 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:07:55.111 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:55.111 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:54 smithi078 ceph-mon[99959]: from='client.83546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:55.111 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:55.111 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:54 smithi078 ceph-mon[91136]: from='client.83546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:07:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:54 smithi146 ceph-mon[103278]: from='client.83546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:55.377 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:07:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:07:55.377 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:07:55.378 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:07:55.687 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:07:55.687 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:55.687 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:07:55.687 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:55.688 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:07:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[91136]: from='client.63633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[91136]: pgmap v8638: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[91136]: from='client.83558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2694699213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/36069763' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[99959]: from='client.63633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[99959]: pgmap v8638: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[99959]: from='client.83558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2694699213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/36069763' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:55 smithi146 ceph-mon[103278]: from='client.63633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:55 smithi146 ceph-mon[103278]: pgmap v8638: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:07:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:55 smithi146 ceph-mon[103278]: from='client.83558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2694699213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:07:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/36069763' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:07:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:56 smithi146 ceph-mon[103278]: from='client.63651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:56 smithi078 ceph-mon[91136]: from='client.63651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:56 smithi078 ceph-mon[99959]: from='client.63651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[91136]: pgmap v8639: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[99959]: pgmap v8639: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:57 smithi146 ceph-mon[103278]: pgmap v8639: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:07:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:07:59.098 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:59.099 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:07:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:07:59 smithi146 ceph-mon[103278]: pgmap v8640: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:07:59 smithi078 ceph-mon[91136]: pgmap v8640: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:07:59 smithi078 ceph-mon[99959]: pgmap v8640: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:08:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:08:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:01.346Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:01.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:01.347Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:01 smithi146 ceph-mon[103278]: pgmap v8641: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[99959]: pgmap v8641: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[91136]: pgmap v8641: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:03 smithi146 ceph-mon[103278]: pgmap v8642: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:03 smithi078 ceph-mon[91136]: pgmap v8642: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:03 smithi078 ceph-mon[99959]: pgmap v8642: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:06.185 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:05 smithi146 ceph-mon[103278]: pgmap v8643: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:05 smithi078 ceph-mon[91136]: pgmap v8643: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:05 smithi078 ceph-mon[99959]: pgmap v8643: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:07 smithi146 ceph-mon[103278]: pgmap v8644: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:07 smithi078 ceph-mon[91136]: pgmap v8644: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:08.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:07 smithi078 ceph-mon[99959]: pgmap v8644: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:10 smithi146 ceph-mon[103278]: pgmap v8645: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:10 smithi078 ceph-mon[99959]: pgmap v8645: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:10 smithi078 ceph-mon[91136]: pgmap v8645: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:08:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:08:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:11.347Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:11.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:11.348Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:12 smithi146 ceph-mon[103278]: pgmap v8646: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:12 smithi078 ceph-mon[91136]: pgmap v8646: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:12 smithi078 ceph-mon[99959]: pgmap v8646: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:14 smithi146 ceph-mon[103278]: pgmap v8647: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:14 smithi078 ceph-mon[91136]: pgmap v8647: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:14 smithi078 ceph-mon[99959]: pgmap v8647: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:16 smithi146 ceph-mon[103278]: pgmap v8648: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:16 smithi078 ceph-mon[91136]: pgmap v8648: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:16 smithi078 ceph-mon[99959]: pgmap v8648: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:18 smithi146 ceph-mon[103278]: pgmap v8649: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[91136]: pgmap v8649: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[99959]: pgmap v8649: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:20 smithi078 ceph-mon[91136]: pgmap v8650: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:20 smithi078 ceph-mon[99959]: pgmap v8650: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:20 smithi146 ceph-mon[103278]: pgmap v8650: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:08:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:08:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:21.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:21.349Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:21.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:22 smithi146 ceph-mon[103278]: pgmap v8651: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[99959]: pgmap v8651: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[91136]: pgmap v8651: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:24 smithi078 ceph-mon[91136]: pgmap v8652: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:24 smithi078 ceph-mon[99959]: pgmap v8652: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:24 smithi146 ceph-mon[103278]: pgmap v8652: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:25.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:25.981 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:08:26.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:26 smithi146 ceph-mon[103278]: pgmap v8653: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:26 smithi078 ceph-mon[91136]: pgmap v8653: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:26 smithi078 ceph-mon[99959]: pgmap v8653: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:26.510 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 4h - - 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 4h - - 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 9s ago 4h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 4s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 9s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 4s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 9s ago 4h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 9s ago 5h 483M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 4s ago 4h 319M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 9s ago 4h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 9s ago 4h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 4s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 9s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:08:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 9s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 9s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 9s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 4s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 4s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 4s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 4s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 4s ago 4h 66.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 9s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 4s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 9s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:08:26.512 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 4s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:08:26.823 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:08:26.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:08:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:08:26.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:08:26.825 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:08:27.094 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:08:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:08:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:08:27.095 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:08:27.404 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:08:27.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:27.404 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:27.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:27.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:27 smithi078 ceph-mon[99959]: from='client.63663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:27.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:27 smithi078 ceph-mon[99959]: from='client.63669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:27.404 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1940052806' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:27 smithi078 ceph-mon[91136]: from='client.63663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:27 smithi078 ceph-mon[91136]: from='client.63669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:27.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1940052806' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:27.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:27 smithi146 ceph-mon[103278]: from='client.63663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:27.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:27 smithi146 ceph-mon[103278]: from='client.63669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:27.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1940052806' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:28 smithi146 ceph-mon[103278]: from='client.83594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:28 smithi146 ceph-mon[103278]: pgmap v8654: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:28 smithi146 ceph-mon[103278]: from='client.63687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3528464157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[91136]: from='client.83594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[91136]: pgmap v8654: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[91136]: from='client.63687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3528464157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[99959]: from='client.83594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[99959]: pgmap v8654: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[99959]: from='client.63687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3528464157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:30 smithi146 ceph-mon[103278]: pgmap v8655: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:30 smithi078 ceph-mon[91136]: pgmap v8655: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:30 smithi078 ceph-mon[99959]: pgmap v8655: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:30.969 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:08:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:08:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:31.349Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:31.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:31.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:32 smithi146 ceph-mon[103278]: pgmap v8656: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:32 smithi078 ceph-mon[91136]: pgmap v8656: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:32 smithi078 ceph-mon[99959]: pgmap v8656: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:34 smithi078 ceph-mon[91136]: pgmap v8657: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:34 smithi078 ceph-mon[99959]: pgmap v8657: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:34 smithi146 ceph-mon[103278]: pgmap v8657: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:36 smithi146 ceph-mon[103278]: pgmap v8658: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:36 smithi078 ceph-mon[91136]: pgmap v8658: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:36 smithi078 ceph-mon[99959]: pgmap v8658: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:38.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[99959]: pgmap v8659: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:38.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[91136]: pgmap v8659: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:38.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:38 smithi146 ceph-mon[103278]: pgmap v8659: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:40 smithi078 ceph-mon[91136]: pgmap v8660: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:40 smithi078 ceph-mon[99959]: pgmap v8660: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:40 smithi146 ceph-mon[103278]: pgmap v8660: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:40.749 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:08:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:08:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:41.350Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:41.350Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:41.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:41.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:42 smithi146 ceph-mon[103278]: pgmap v8661: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[91136]: pgmap v8661: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[99959]: pgmap v8661: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:44 smithi146 ceph-mon[103278]: pgmap v8662: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:44 smithi078 ceph-mon[91136]: pgmap v8662: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:44 smithi078 ceph-mon[99959]: pgmap v8662: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:46 smithi146 ceph-mon[103278]: pgmap v8663: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:46 smithi078 ceph-mon[91136]: pgmap v8663: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:46 smithi078 ceph-mon[99959]: pgmap v8663: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:48 smithi146 ceph-mon[103278]: pgmap v8664: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:48 smithi078 ceph-mon[91136]: pgmap v8664: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:48 smithi078 ceph-mon[99959]: pgmap v8664: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:50 smithi146 ceph-mon[103278]: pgmap v8665: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:50 smithi078 ceph-mon[91136]: pgmap v8665: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:50 smithi078 ceph-mon[99959]: pgmap v8665: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:08:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:08:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:51.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:51.351Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:08:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:08:51.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:08:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:52 smithi146 ceph-mon[103278]: pgmap v8666: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:52 smithi078 ceph-mon[99959]: pgmap v8666: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:52 smithi078 ceph-mon[91136]: pgmap v8666: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:08:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:54 smithi146 ceph-mon[103278]: pgmap v8667: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:54 smithi078 ceph-mon[99959]: pgmap v8667: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:54 smithi078 ceph-mon[91136]: pgmap v8667: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:08:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:56 smithi078 ceph-mon[99959]: pgmap v8668: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:56 smithi078 ceph-mon[91136]: pgmap v8668: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:56 smithi146 ceph-mon[103278]: pgmap v8668: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:08:57.686 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:08:58.209 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:08:58.209 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 4h - - 2024-09-21T01:08:58.209 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 4h - - 2024-09-21T01:08:58.209 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 0s ago 4h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:08:58.209 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 16s ago 4h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:08:58.209 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 0s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 16s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 0s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 0s ago 5h 483M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 16s ago 4h 320M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 0s ago 4h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 0s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 16s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 0s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 0s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 0s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 0s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 16s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 16s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 16s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:08:58.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 16s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:08:58.211 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 16s ago 4h 66.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:08:58.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 0s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:08:58.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 16s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:08:58.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 0s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:08:58.211 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 16s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:08:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:08:58.521 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:08:58.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[91136]: pgmap v8669: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:58.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[99959]: pgmap v8669: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:58.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:58 smithi146 ceph-mon[103278]: pgmap v8669: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:08:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:08:58.799 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:08:58.800 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:08:59.115 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:08:59.115 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:08:59.115 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:08:59.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[91136]: from='client.83618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[91136]: from='client.83624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[91136]: from='client.63711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:59.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1524202156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:59.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/640714297' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:59 smithi146 ceph-mon[103278]: from='client.83618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:59 smithi146 ceph-mon[103278]: from='client.83624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:59 smithi146 ceph-mon[103278]: from='client.63711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1524202156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:08:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/640714297' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[99959]: from='client.83618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[99959]: from='client.83624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[99959]: from='client.63711 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1524202156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:08:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:08:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/640714297' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:00.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:00 smithi078 ceph-mon[99959]: pgmap v8670: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:00.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:00 smithi078 ceph-mon[99959]: from='client.63723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:00.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:00 smithi078 ceph-mon[91136]: pgmap v8670: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:00.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:00 smithi078 ceph-mon[91136]: from='client.63723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:00 smithi146 ceph-mon[103278]: pgmap v8670: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:00 smithi146 ceph-mon[103278]: from='client.63723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:00.775 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:09:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:09:01.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:01 smithi078 ceph-mon[91136]: pgmap v8671: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:01.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:01 smithi078 ceph-mon[99959]: pgmap v8671: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:01.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:01.351Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:01.436 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:01.352Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:01.436 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:01.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:01 smithi146 ceph-mon[103278]: pgmap v8671: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:03.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:04 smithi078 ceph-mon[99959]: pgmap v8672: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:04 smithi078 ceph-mon[91136]: pgmap v8672: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:04 smithi146 ceph-mon[103278]: pgmap v8672: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:06 smithi078 ceph-mon[99959]: pgmap v8673: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:06 smithi078 ceph-mon[91136]: pgmap v8673: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:06.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:06 smithi146 ceph-mon[103278]: pgmap v8673: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:08 smithi146 ceph-mon[103278]: pgmap v8674: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:08 smithi078 ceph-mon[99959]: pgmap v8674: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:08 smithi078 ceph-mon[91136]: pgmap v8674: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:10 smithi146 ceph-mon[103278]: pgmap v8675: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:10 smithi078 ceph-mon[99959]: pgmap v8675: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:10 smithi078 ceph-mon[91136]: pgmap v8675: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:09:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:09:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:11.353Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:11.353Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:11.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:12.347 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:12 smithi078 ceph-mon[99959]: pgmap v8676: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:12.347 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:12 smithi078 ceph-mon[91136]: pgmap v8676: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:12 smithi146 ceph-mon[103278]: pgmap v8676: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:14 smithi078 ceph-mon[99959]: pgmap v8677: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:14 smithi078 ceph-mon[91136]: pgmap v8677: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:14 smithi146 ceph-mon[103278]: pgmap v8677: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:16 smithi146 ceph-mon[103278]: pgmap v8678: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:16 smithi078 ceph-mon[91136]: pgmap v8678: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:16 smithi078 ceph-mon[99959]: pgmap v8678: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:18 smithi146 ceph-mon[103278]: pgmap v8679: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[99959]: pgmap v8679: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[91136]: pgmap v8679: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:09:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:20 smithi146 ceph-mon[103278]: pgmap v8680: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:20 smithi078 ceph-mon[91136]: pgmap v8680: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:20 smithi078 ceph-mon[99959]: pgmap v8680: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:20.884 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:09:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:09:21.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:21.353Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:21.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:21.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:22.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:22 smithi078 ceph-mon[99959]: pgmap v8681: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:22.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:22 smithi078 ceph-mon[91136]: pgmap v8681: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:22 smithi146 ceph-mon[103278]: pgmap v8681: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:23.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:24.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:24 smithi146 ceph-mon[103278]: pgmap v8682: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:24 smithi078 ceph-mon[91136]: pgmap v8682: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:24 smithi078 ceph-mon[99959]: pgmap v8682: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:25.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:26 smithi146 ceph-mon[103278]: pgmap v8683: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:26 smithi078 ceph-mon[91136]: pgmap v8683: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:26 smithi078 ceph-mon[99959]: pgmap v8683: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:28.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:28 smithi146 ceph-mon[103278]: pgmap v8684: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:28 smithi078 ceph-mon[99959]: pgmap v8684: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:28 smithi078 ceph-mon[91136]: pgmap v8684: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:29.414 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:09:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:29.938 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 5h - - 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 4h - - 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 12s ago 4h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 7s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 12s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 7s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 12s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 12s ago 5h 484M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 7s ago 4h 321M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 12s ago 4h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 12s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 7s ago 4h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 12s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 12s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 12s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:09:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 12s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 7s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 7s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 7s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 7s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 7s ago 4h 66.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 12s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 7s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 12s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:09:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 7s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:09:30.250 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:09:30.250 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:09:30.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:09:30.251 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:09:30.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:30 smithi078 ceph-mon[99959]: pgmap v8685: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:30.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:30 smithi078 ceph-mon[91136]: pgmap v8685: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:30 smithi146 ceph-mon[103278]: pgmap v8685: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:30.519 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:09:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:09:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:09:30.521 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:09:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:09:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:09:30.832 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:09:30.832 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:09:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:09:30.833 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:09:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:09:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:31 smithi146 ceph-mon[103278]: from='client.63735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:31 smithi146 ceph-mon[103278]: from='client.63741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:31 smithi146 ceph-mon[103278]: from='client.63747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/510110118' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3015959048' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[99959]: from='client.63735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[99959]: from='client.63741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[99959]: from='client.63747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/510110118' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3015959048' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[91136]: from='client.63735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[91136]: from='client.63741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[91136]: from='client.63747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/510110118' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3015959048' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:09:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:31.353Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:31.354Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:31.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:31.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:32 smithi146 ceph-mon[103278]: from='client.83678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:32 smithi146 ceph-mon[103278]: pgmap v8686: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:32 smithi078 ceph-mon[91136]: from='client.83678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:32 smithi078 ceph-mon[91136]: pgmap v8686: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:32 smithi078 ceph-mon[99959]: from='client.83678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:09:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:32 smithi078 ceph-mon[99959]: pgmap v8686: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:34 smithi078 ceph-mon[91136]: pgmap v8687: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:34 smithi078 ceph-mon[99959]: pgmap v8687: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:34 smithi146 ceph-mon[103278]: pgmap v8687: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:35.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:36 smithi146 ceph-mon[103278]: pgmap v8688: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:36 smithi078 ceph-mon[99959]: pgmap v8688: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:36 smithi078 ceph-mon[91136]: pgmap v8688: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:38 smithi146 ceph-mon[103278]: pgmap v8689: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[99959]: pgmap v8689: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[91136]: pgmap v8689: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:40 smithi146 ceph-mon[103278]: pgmap v8690: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:40 smithi078 ceph-mon[91136]: pgmap v8690: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:40 smithi078 ceph-mon[99959]: pgmap v8690: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:09:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:09:41.727 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:41.355Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:41.727 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:41.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:41.727 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:41.355Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:42 smithi146 ceph-mon[103278]: pgmap v8691: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:42 smithi078 ceph-mon[91136]: pgmap v8691: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:42 smithi078 ceph-mon[99959]: pgmap v8691: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:43.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:44 smithi078 ceph-mon[99959]: pgmap v8692: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:44 smithi078 ceph-mon[91136]: pgmap v8692: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:44 smithi146 ceph-mon[103278]: pgmap v8692: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:46 smithi146 ceph-mon[103278]: pgmap v8693: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:46 smithi078 ceph-mon[91136]: pgmap v8693: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:46 smithi078 ceph-mon[99959]: pgmap v8693: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:48.536 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:48 smithi146 ceph-mon[103278]: pgmap v8694: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:48 smithi078 ceph-mon[99959]: pgmap v8694: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:48 smithi078 ceph-mon[91136]: pgmap v8694: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:50 smithi146 ceph-mon[103278]: pgmap v8695: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:50 smithi078 ceph-mon[99959]: pgmap v8695: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:50 smithi078 ceph-mon[91136]: pgmap v8695: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:09:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:09:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:51.356Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:51.356Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:09:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:09:51.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:09:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:52 smithi146 ceph-mon[103278]: pgmap v8696: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:52 smithi078 ceph-mon[91136]: pgmap v8696: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:52 smithi078 ceph-mon[99959]: pgmap v8696: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:54.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:54 smithi146 ceph-mon[103278]: pgmap v8697: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:54 smithi078 ceph-mon[91136]: pgmap v8697: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:54 smithi078 ceph-mon[99959]: pgmap v8697: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:09:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:56 smithi146 ceph-mon[103278]: pgmap v8698: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:56 smithi078 ceph-mon[91136]: pgmap v8698: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:56 smithi078 ceph-mon[99959]: pgmap v8698: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:09:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:58 smithi146 ceph-mon[103278]: pgmap v8699: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[91136]: pgmap v8699: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[99959]: pgmap v8699: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:09:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:09:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:09:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:09:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:09:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T01:09:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:00 smithi078 ceph-mon[99959]: pgmap v8700: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:00 smithi078 ceph-mon[91136]: pgmap v8700: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:00 smithi146 ceph-mon[103278]: pgmap v8700: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:10:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:10:01.124 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:10:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:01.357Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:01.357Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:01.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 5h - - 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 3s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 19s ago 4h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 3s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 19s ago 4h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:10:01.651 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 3s ago 5h 489M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 5h 322M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 3s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 19s ago 4h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 4h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 19s ago 4h 67.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:10:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:10:01.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:10:01.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:10:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:10:01.960 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:10:02.232 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:10:02.233 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:10:02.233 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:10:02.233 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:10:02.233 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:10:02.233 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:10:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:02 smithi078 ceph-mon[99959]: pgmap v8701: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:02 smithi078 ceph-mon[99959]: from='client.63771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1055245080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:02 smithi078 ceph-mon[91136]: pgmap v8701: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:02 smithi078 ceph-mon[91136]: from='client.63771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1055245080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:02.569 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:02.569 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:10:02.569 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:10:02.569 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:10:02.569 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:10:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:02 smithi146 ceph-mon[103278]: pgmap v8701: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:02 smithi146 ceph-mon[103278]: from='client.63771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1055245080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='client.63777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='client.63783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='client.63795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1542782713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='client.63777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='client.63783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='client.63795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1542782713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='client.63777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='client.63783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='client.63795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1542782713' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:04.428 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:04 smithi078 ceph-mon[99959]: pgmap v8702: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:04.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:04 smithi078 ceph-mon[91136]: pgmap v8702: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:04 smithi146 ceph-mon[103278]: pgmap v8702: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:06 smithi146 ceph-mon[103278]: pgmap v8703: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:06 smithi078 ceph-mon[91136]: pgmap v8703: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:06 smithi078 ceph-mon[99959]: pgmap v8703: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:08.662 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:08 smithi146 ceph-mon[103278]: pgmap v8704: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:08 smithi078 ceph-mon[91136]: pgmap v8704: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:08 smithi078 ceph-mon[99959]: pgmap v8704: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:10.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:10 smithi146 ceph-mon[103278]: pgmap v8705: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:10 smithi078 ceph-mon[91136]: pgmap v8705: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:10.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:10 smithi078 ceph-mon[99959]: pgmap v8705: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:10:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:10:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:11.358Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:11.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:11.358Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:12 smithi146 ceph-mon[103278]: pgmap v8706: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:12 smithi078 ceph-mon[91136]: pgmap v8706: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:12 smithi078 ceph-mon[99959]: pgmap v8706: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:14 smithi146 ceph-mon[103278]: pgmap v8707: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:14 smithi078 ceph-mon[99959]: pgmap v8707: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:14 smithi078 ceph-mon[91136]: pgmap v8707: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:16 smithi146 ceph-mon[103278]: pgmap v8708: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:16 smithi078 ceph-mon[91136]: pgmap v8708: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:16 smithi078 ceph-mon[99959]: pgmap v8708: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:18 smithi146 ceph-mon[103278]: pgmap v8709: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[99959]: pgmap v8709: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[91136]: pgmap v8709: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:20 smithi146 ceph-mon[103278]: pgmap v8710: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:20 smithi078 ceph-mon[91136]: pgmap v8710: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:20 smithi078 ceph-mon[99959]: pgmap v8710: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:10:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:10:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:21.358Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:21.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:21.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:22 smithi146 ceph-mon[103278]: pgmap v8711: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:22 smithi078 ceph-mon[99959]: pgmap v8711: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:22 smithi078 ceph-mon[91136]: pgmap v8711: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[91136]: pgmap v8712: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:23 smithi078 ceph-mon[99959]: pgmap v8712: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:23 smithi146 ceph-mon[103278]: pgmap v8712: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:24.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:25 smithi078 ceph-mon[91136]: pgmap v8713: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:25 smithi078 ceph-mon[99959]: pgmap v8713: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:25.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:25 smithi146 ceph-mon[103278]: pgmap v8713: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:27 smithi146 ceph-mon[103278]: pgmap v8714: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:27 smithi078 ceph-mon[99959]: pgmap v8714: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:27 smithi078 ceph-mon[91136]: pgmap v8714: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:29.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:30.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:29 smithi146 ceph-mon[103278]: pgmap v8715: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:29 smithi078 ceph-mon[91136]: pgmap v8715: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:29 smithi078 ceph-mon[99959]: pgmap v8715: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:10:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:10:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:31.359Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:31.359Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:31.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:31 smithi146 ceph-mon[103278]: pgmap v8716: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:31 smithi078 ceph-mon[91136]: pgmap v8716: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:31 smithi078 ceph-mon[99959]: pgmap v8716: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:32.864 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:10:33.385 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:10:33.385 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 5h - - 2024-09-21T01:10:33.385 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T01:10:33.385 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 15s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 10s ago 4h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 15s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 15s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 15s ago 5h 486M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 10s ago 5h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 15s ago 5h 331M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 15s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 10s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 15s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 15s ago 4h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 15s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 15s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:10:33.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 4h 67.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 15s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 15s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:10:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:10:33.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:10:33.696 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:10:33.696 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:10:33.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:10:33.696 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:10:33.962 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:10:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:10:33.962 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:10:33.963 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:10:33.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[91136]: pgmap v8717: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:33.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[91136]: from='client.83726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[91136]: from='client.83732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[91136]: from='client.63819 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1914797594' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[99959]: pgmap v8717: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[99959]: from='client.83726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[99959]: from='client.83732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[99959]: from='client.63819 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:33.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1914797594' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:33 smithi146 ceph-mon[103278]: pgmap v8717: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:33 smithi146 ceph-mon[103278]: from='client.83726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:33 smithi146 ceph-mon[103278]: from='client.83732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:33 smithi146 ceph-mon[103278]: from='client.63819 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1914797594' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:10:34.272 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:10:34.272 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:10:34.272 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:10:34.272 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:10:34.272 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:10:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:34 smithi146 ceph-mon[103278]: from='client.63831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1873949370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:34 smithi078 ceph-mon[91136]: from='client.63831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1873949370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:34 smithi078 ceph-mon[99959]: from='client.63831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:10:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1873949370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:10:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:35 smithi146 ceph-mon[103278]: pgmap v8718: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:35 smithi078 ceph-mon[99959]: pgmap v8718: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:35 smithi078 ceph-mon[91136]: pgmap v8718: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:37 smithi146 ceph-mon[103278]: pgmap v8719: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:37 smithi078 ceph-mon[99959]: pgmap v8719: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:37 smithi078 ceph-mon[91136]: pgmap v8719: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:40 smithi146 ceph-mon[103278]: pgmap v8720: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:40 smithi078 ceph-mon[91136]: pgmap v8720: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:40 smithi078 ceph-mon[99959]: pgmap v8720: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:10:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:10:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:41.359Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:41.360Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:41.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:42 smithi146 ceph-mon[103278]: pgmap v8721: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:42 smithi078 ceph-mon[99959]: pgmap v8721: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:42 smithi078 ceph-mon[91136]: pgmap v8721: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:43.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[91136]: pgmap v8722: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[99959]: pgmap v8722: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:43.964 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:43 smithi146 ceph-mon[103278]: pgmap v8722: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:44.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:44.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:45 smithi078 ceph-mon[99959]: pgmap v8723: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:45 smithi078 ceph-mon[91136]: pgmap v8723: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:45 smithi146 ceph-mon[103278]: pgmap v8723: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:47 smithi146 ceph-mon[103278]: pgmap v8724: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:47 smithi078 ceph-mon[91136]: pgmap v8724: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:47 smithi078 ceph-mon[99959]: pgmap v8724: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:49.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:49 smithi146 ceph-mon[103278]: pgmap v8725: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:49 smithi078 ceph-mon[91136]: pgmap v8725: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:49 smithi078 ceph-mon[99959]: pgmap v8725: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:10:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:10:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:10:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:51.360Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:51.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:10:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:10:51.361Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:10:52.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:51 smithi146 ceph-mon[103278]: pgmap v8726: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:52.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:51 smithi078 ceph-mon[99959]: pgmap v8726: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:52.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:51 smithi078 ceph-mon[91136]: pgmap v8726: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:54.082 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:53 smithi078 ceph-mon[99959]: pgmap v8727: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:54.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:53 smithi078 ceph-mon[91136]: pgmap v8727: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:53 smithi146 ceph-mon[103278]: pgmap v8727: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:10:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:55.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:10:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:55 smithi146 ceph-mon[103278]: pgmap v8728: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:10:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:55 smithi078 ceph-mon[99959]: pgmap v8728: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:10:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:55 smithi078 ceph-mon[91136]: pgmap v8728: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:10:58.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:57 smithi078 ceph-mon[99959]: pgmap v8729: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:10:58.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:57 smithi078 ceph-mon[91136]: pgmap v8729: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:10:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:57 smithi146 ceph-mon[103278]: pgmap v8729: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:10:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:00 smithi146 ceph-mon[103278]: pgmap v8730: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:11:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:00 smithi078 ceph-mon[91136]: pgmap v8730: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:11:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:00 smithi078 ceph-mon[99959]: pgmap v8730: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:11:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:11:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:11:01.718 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:01.361Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:01.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:01.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:01.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:01.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:02 smithi146 ceph-mon[103278]: pgmap v8731: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:11:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:02 smithi078 ceph-mon[91136]: pgmap v8731: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:11:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:02 smithi078 ceph-mon[99959]: pgmap v8731: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:11:04.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[91136]: pgmap v8732: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:11:04.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.092 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[99959]: pgmap v8732: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:11:04.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:03 smithi146 ceph-mon[103278]: pgmap v8732: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:11:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:04.578 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:11:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 5h - - 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 5h - - 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 6s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 2s ago 4h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 2s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:11:05.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 6s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 6s ago 5h 487M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 2s ago 5h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 6s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 6s ago 4h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 2s ago 4h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 5h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 4h 77.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 4h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 2s ago 4h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 2s ago 4h 81.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 2s ago 4h 88.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 2s ago 4h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 2s ago 4h 67.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:11:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:11:05.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 2s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:11:05.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:11:05.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 2s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:11:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:11:05.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:11:05.437 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:11:05.731 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:11:05.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:11:05.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[99959]: from='client.63843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:05.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[99959]: pgmap v8733: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:11:05.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[99959]: from='client.83768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:05.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[99959]: from='client.83774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:05.962 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2915747204' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:05.962 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[91136]: from='client.63843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:05.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[91136]: pgmap v8733: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:11:05.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[91136]: from='client.83768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:05.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[91136]: from='client.83774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:05.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2915747204' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:06.062 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:11:06.062 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:06.062 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:06.062 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:05 smithi146 ceph-mon[103278]: from='client.63843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:05 smithi146 ceph-mon[103278]: pgmap v8733: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:11:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:05 smithi146 ceph-mon[103278]: from='client.83768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:05 smithi146 ceph-mon[103278]: from='client.83774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2915747204' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:07.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:06 smithi078 ceph-mon[99959]: from='client.83786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:07.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3627766834' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:07.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:06 smithi078 ceph-mon[91136]: from='client.83786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:07.067 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3627766834' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:06 smithi146 ceph-mon[103278]: from='client.83786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3627766834' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:07 smithi146 ceph-mon[103278]: pgmap v8734: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:11:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:07 smithi078 ceph-mon[91136]: pgmap v8734: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:11:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:07 smithi078 ceph-mon[99959]: pgmap v8734: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:11:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:09 smithi146 ceph-mon[103278]: pgmap v8735: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:11:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:09 smithi078 ceph-mon[91136]: pgmap v8735: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:11:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:09 smithi078 ceph-mon[99959]: pgmap v8735: 161 pgs: 161 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:11:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:11:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:11:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:11.362Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:11.362Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:11.363Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:11 smithi146 ceph-mon[103278]: pgmap v8736: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:11:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:11 smithi078 ceph-mon[99959]: pgmap v8736: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:11:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:11 smithi078 ceph-mon[91136]: pgmap v8736: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:11:14.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:13 smithi078 ceph-mon[99959]: pgmap v8737: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T01:11:14.102 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:14.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:13 smithi078 ceph-mon[91136]: pgmap v8737: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T01:11:14.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:13 smithi146 ceph-mon[103278]: pgmap v8737: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T01:11:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:15 smithi146 ceph-mon[103278]: pgmap v8738: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T01:11:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:15 smithi078 ceph-mon[99959]: pgmap v8738: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T01:11:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:15 smithi078 ceph-mon[91136]: pgmap v8738: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T01:11:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:17 smithi146 ceph-mon[103278]: pgmap v8739: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:11:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:17 smithi078 ceph-mon[99959]: pgmap v8739: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:11:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:17 smithi078 ceph-mon[91136]: pgmap v8739: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:11:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:20.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:20 smithi078 ceph-mon[99959]: pgmap v8740: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:11:20.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:20 smithi078 ceph-mon[91136]: pgmap v8740: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:11:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:20 smithi146 ceph-mon[103278]: pgmap v8740: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T01:11:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:11:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:11:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:21.363Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:21.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:21.364Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:22 smithi146 ceph-mon[103278]: pgmap v8741: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T01:11:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:22 smithi078 ceph-mon[91136]: pgmap v8741: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T01:11:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:22 smithi078 ceph-mon[99959]: pgmap v8741: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T01:11:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:23 smithi146 ceph-mon[103278]: pgmap v8742: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:11:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[99959]: pgmap v8742: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:11:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[91136]: pgmap v8742: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 108 op/s 2024-09-21T01:11:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:25.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:25.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:25 smithi146 ceph-mon[103278]: pgmap v8743: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:11:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:25 smithi078 ceph-mon[99959]: pgmap v8743: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:11:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:25 smithi078 ceph-mon[91136]: pgmap v8743: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:11:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:27 smithi146 ceph-mon[103278]: pgmap v8744: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:11:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:27 smithi078 ceph-mon[91136]: pgmap v8744: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:11:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:27 smithi078 ceph-mon[99959]: pgmap v8744: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:11:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:29 smithi146 ceph-mon[103278]: pgmap v8745: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:11:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:29 smithi078 ceph-mon[91136]: pgmap v8745: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:11:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:29 smithi078 ceph-mon[99959]: pgmap v8745: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:11:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:11:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:11:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:31.364Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:31.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:31.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:31 smithi146 ceph-mon[103278]: pgmap v8746: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:11:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:31 smithi078 ceph-mon[91136]: pgmap v8746: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:11:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:31 smithi078 ceph-mon[99959]: pgmap v8746: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T01:11:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:33 smithi146 ceph-mon[103278]: pgmap v8747: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:33 smithi078 ceph-mon[91136]: pgmap v8747: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:33 smithi078 ceph-mon[99959]: pgmap v8747: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:35.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:35 smithi146 ceph-mon[103278]: pgmap v8748: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:35 smithi078 ceph-mon[91136]: pgmap v8748: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:35 smithi078 ceph-mon[99959]: pgmap v8748: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:36.359 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:11:36.888 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:11:36.888 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 5h - - 2024-09-21T01:11:36.888 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 5h - - 2024-09-21T01:11:36.888 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 18s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:11:36.888 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 13s ago 4h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:11:36.888 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 18s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 13s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 18s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 18s ago 5h 487M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 13s ago 5h 324M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 18s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 18s ago 4h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 13s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 18s ago 5h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 18s ago 5h 92.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 18s ago 5h 78.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 18s ago 4h 91.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 4h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 4h 82.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 4h 88.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 4h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 13s ago 4h 69.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:11:36.889 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 18s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:11:36.890 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:11:36.890 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 18s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:11:36.890 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:11:37.201 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:11:37.201 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:11:37.201 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:11:37.202 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:11:37.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:36 smithi078 ceph-mon[91136]: from='client.63879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:37.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:36 smithi078 ceph-mon[99959]: from='client.63879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:36 smithi146 ceph-mon[103278]: from='client.63879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:11:37.471 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:11:37.472 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:11:37.472 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:11:37.472 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:11:37.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:11:37.780 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:11:37.780 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:37.780 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:11:37.780 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:37.780 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:11:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[99959]: from='client.63885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[99959]: pgmap v8749: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[99959]: from='client.63891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4032115049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3943194165' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[91136]: from='client.63885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[91136]: pgmap v8749: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[91136]: from='client.63891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4032115049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3943194165' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:37 smithi146 ceph-mon[103278]: from='client.63885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:37 smithi146 ceph-mon[103278]: pgmap v8749: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:37 smithi146 ceph-mon[103278]: from='client.63891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4032115049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:11:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3943194165' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:11:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:38 smithi146 ceph-mon[103278]: from='client.63903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[91136]: from='client.63903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[99959]: from='client.63903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:11:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:39 smithi146 ceph-mon[103278]: pgmap v8750: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:39 smithi078 ceph-mon[99959]: pgmap v8750: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:39 smithi078 ceph-mon[91136]: pgmap v8750: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:11:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:11:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:41.365Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:41.365Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:41.366Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:41 smithi146 ceph-mon[103278]: pgmap v8751: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:41 smithi078 ceph-mon[91136]: pgmap v8751: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:41 smithi078 ceph-mon[99959]: pgmap v8751: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:44 smithi146 ceph-mon[103278]: pgmap v8752: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[91136]: pgmap v8752: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[99959]: pgmap v8752: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:46.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:46 smithi078 ceph-mon[99959]: pgmap v8753: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:46.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:46 smithi078 ceph-mon[91136]: pgmap v8753: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:46 smithi146 ceph-mon[103278]: pgmap v8753: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:48.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:48 smithi078 ceph-mon[99959]: pgmap v8754: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:48.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:48 smithi078 ceph-mon[91136]: pgmap v8754: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:48 smithi146 ceph-mon[103278]: pgmap v8754: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:50 smithi078 ceph-mon[91136]: pgmap v8755: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:50 smithi078 ceph-mon[99959]: pgmap v8755: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:50 smithi146 ceph-mon[103278]: pgmap v8755: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:11:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:11:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:51.366Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:51.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:11:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:11:51.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:11:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:52 smithi146 ceph-mon[103278]: pgmap v8756: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:52 smithi078 ceph-mon[99959]: pgmap v8756: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:52 smithi078 ceph-mon[91136]: pgmap v8756: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:54 smithi146 ceph-mon[103278]: pgmap v8757: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:54 smithi078 ceph-mon[99959]: pgmap v8757: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:54 smithi078 ceph-mon[91136]: pgmap v8757: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:11:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:56 smithi146 ceph-mon[103278]: pgmap v8758: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:56 smithi078 ceph-mon[99959]: pgmap v8758: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:56 smithi078 ceph-mon[91136]: pgmap v8758: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:11:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:58 smithi146 ceph-mon[103278]: pgmap v8759: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:58 smithi078 ceph-mon[99959]: pgmap v8759: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:58 smithi078 ceph-mon[91136]: pgmap v8759: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:11:59.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:59.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:59.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:11:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:11:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:11:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:11:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:00 smithi146 ceph-mon[103278]: pgmap v8760: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:00 smithi078 ceph-mon[91136]: pgmap v8760: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:00 smithi078 ceph-mon[99959]: pgmap v8760: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:12:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:12:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:01.366Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:01.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:01.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:02 smithi146 ceph-mon[103278]: pgmap v8761: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:02 smithi078 ceph-mon[99959]: pgmap v8761: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:02 smithi078 ceph-mon[91136]: pgmap v8761: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:04 smithi146 ceph-mon[103278]: pgmap v8762: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[91136]: pgmap v8762: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[99959]: pgmap v8762: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:05.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:06 smithi146 ceph-mon[103278]: pgmap v8763: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:06 smithi078 ceph-mon[91136]: pgmap v8763: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:06 smithi078 ceph-mon[99959]: pgmap v8763: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:12:08.451 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:08 smithi078 ceph-mon[99959]: pgmap v8764: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:08.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:08 smithi078 ceph-mon[91136]: pgmap v8764: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:08 smithi146 ceph-mon[103278]: pgmap v8764: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 5h - - 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 5h - - 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 10s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 5s ago 4h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 10s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:12:08.636 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 5s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 10s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 10s ago 5h 488M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 5s ago 5h 325M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 10s ago 5h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 10s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 5s ago 4h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 10s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 10s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 10s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 10s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 5s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 5s ago 4h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 4h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 4h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:12:08.637 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 5s ago 4h 69.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:12:08.638 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 10s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:12:08.638 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:12:08.638 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 10s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:12:08.638 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:12:08.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:08.956 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:12:08.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:12:08.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:12:08.956 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:12:09.227 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:12:09.227 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:12:09.228 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:12:09.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:09 smithi078 ceph-mon[99959]: from='client.83834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:09.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:09 smithi078 ceph-mon[99959]: from='client.83840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:09.229 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1137060323' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:09.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:09 smithi078 ceph-mon[91136]: from='client.83834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:09.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:09 smithi078 ceph-mon[91136]: from='client.83840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:09.229 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1137060323' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:09 smithi146 ceph-mon[103278]: from='client.83834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:09 smithi146 ceph-mon[103278]: from='client.83840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1137060323' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:09.542 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:12:09.543 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:09.543 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:09.543 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:10 smithi146 ceph-mon[103278]: from='client.83846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:10 smithi146 ceph-mon[103278]: pgmap v8765: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:10 smithi146 ceph-mon[103278]: from='client.83858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1832117904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[99959]: from='client.83846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[99959]: pgmap v8765: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[99959]: from='client.83858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1832117904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[91136]: from='client.83846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[91136]: pgmap v8765: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[91136]: from='client.83858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1832117904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:12:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:12:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:11.367Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:11.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:11.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:12 smithi146 ceph-mon[103278]: pgmap v8766: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:12 smithi078 ceph-mon[99959]: pgmap v8766: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:12 smithi078 ceph-mon[91136]: pgmap v8766: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:14 smithi146 ceph-mon[103278]: pgmap v8767: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:14 smithi078 ceph-mon[99959]: pgmap v8767: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:14 smithi078 ceph-mon[91136]: pgmap v8767: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:16 smithi146 ceph-mon[103278]: pgmap v8768: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:16 smithi078 ceph-mon[91136]: pgmap v8768: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:16 smithi078 ceph-mon[99959]: pgmap v8768: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:18 smithi146 ceph-mon[103278]: pgmap v8769: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:18 smithi078 ceph-mon[91136]: pgmap v8769: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:18 smithi078 ceph-mon[99959]: pgmap v8769: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:19 smithi146 ceph-mon[103278]: pgmap v8770: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[91136]: pgmap v8770: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:19 smithi078 ceph-mon[99959]: pgmap v8770: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:12:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:12:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:21.367Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:21.367Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:21.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:22.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:21 smithi078 ceph-mon[91136]: pgmap v8771: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:22.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:21 smithi078 ceph-mon[99959]: pgmap v8771: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:21 smithi146 ceph-mon[103278]: pgmap v8771: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[99959]: pgmap v8772: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[91136]: pgmap v8772: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:24 smithi146 ceph-mon[103278]: pgmap v8772: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:24.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:24.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:26 smithi146 ceph-mon[103278]: pgmap v8773: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:26 smithi078 ceph-mon[91136]: pgmap v8773: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:26 smithi078 ceph-mon[99959]: pgmap v8773: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:28 smithi078 ceph-mon[91136]: pgmap v8774: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:28 smithi078 ceph-mon[99959]: pgmap v8774: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:28 smithi146 ceph-mon[103278]: pgmap v8774: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:30.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:30 smithi078 ceph-mon[99959]: pgmap v8775: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:30 smithi078 ceph-mon[91136]: pgmap v8775: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:30 smithi146 ceph-mon[103278]: pgmap v8775: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:12:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:12:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:31.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:31.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:31.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:32 smithi146 ceph-mon[103278]: pgmap v8776: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:32 smithi078 ceph-mon[91136]: pgmap v8776: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:32 smithi078 ceph-mon[99959]: pgmap v8776: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:34 smithi078 ceph-mon[91136]: pgmap v8777: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:34 smithi078 ceph-mon[99959]: pgmap v8777: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:34 smithi146 ceph-mon[103278]: pgmap v8777: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:35.534 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:36 smithi146 ceph-mon[103278]: pgmap v8778: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:36 smithi078 ceph-mon[91136]: pgmap v8778: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:36 smithi078 ceph-mon[99959]: pgmap v8778: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:38 smithi146 ceph-mon[103278]: pgmap v8779: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:38 smithi078 ceph-mon[99959]: pgmap v8779: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:38 smithi078 ceph-mon[91136]: pgmap v8779: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:39.834 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:12:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:39 smithi146 ceph-mon[103278]: pgmap v8780: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[91136]: pgmap v8780: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[99959]: pgmap v8780: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 5h - - 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 5h - - 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 1s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 17s ago 4h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 1s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:12:40.363 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 17s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 1s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 1s ago 5h 487M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 17s ago 5h 323M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 1s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 1s ago 4h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 17s ago 4h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 1s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 1s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 1s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 1s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 17s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 17s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 17s ago 4h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 17s ago 4h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 17s ago 4h 66.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 1s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:12:40.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 17s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:12:40.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 1s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:12:40.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 17s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:40.673 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:12:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:12:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:12:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:12:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:12:40.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:12:40.674 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:12:40.674 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:12:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:12:40.942 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:12:40.942 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:12:40.942 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:12:40.942 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:12:40.942 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:12:40.943 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:12:40.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[91136]: from='client.63951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:40.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[91136]: from='client.83876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:40.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[91136]: from='client.63963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:40.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/603616340' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:40.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[99959]: from='client.63951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:40.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[99959]: from='client.83876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:40.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[99959]: from='client.63963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:40.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/603616340' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:41.250 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:12:41.250 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:41.250 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:12:41.251 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:41.251 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:12:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:40 smithi146 ceph-mon[103278]: from='client.63951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:40 smithi146 ceph-mon[103278]: from='client.83876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:40 smithi146 ceph-mon[103278]: from='client.63963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/603616340' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:12:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:41.368Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:41.368Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:41.369Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:41 smithi146 ceph-mon[103278]: pgmap v8781: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:41 smithi146 ceph-mon[103278]: from='client.63975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2820664284' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-mon[99959]: pgmap v8781: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-mon[99959]: from='client.63975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2820664284' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-mon[91136]: pgmap v8781: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-mon[91136]: from='client.63975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:12:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2820664284' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[99959]: pgmap v8782: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[91136]: pgmap v8782: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:43 smithi146 ceph-mon[103278]: pgmap v8782: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:45 smithi146 ceph-mon[103278]: pgmap v8783: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:45 smithi078 ceph-mon[91136]: pgmap v8783: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:45 smithi078 ceph-mon[99959]: pgmap v8783: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:48.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:47 smithi078 ceph-mon[91136]: pgmap v8784: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:48.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:47 smithi078 ceph-mon[99959]: pgmap v8784: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:47 smithi146 ceph-mon[103278]: pgmap v8784: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:49 smithi146 ceph-mon[103278]: pgmap v8785: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:49 smithi078 ceph-mon[91136]: pgmap v8785: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:49 smithi078 ceph-mon[99959]: pgmap v8785: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:50.630 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:12:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:12:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:51.369Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:51.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:12:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:12:51.370Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:12:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:51 smithi146 ceph-mon[103278]: pgmap v8786: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:52.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:51 smithi078 ceph-mon[91136]: pgmap v8786: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:52.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:51 smithi078 ceph-mon[99959]: pgmap v8786: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:53.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:53 smithi146 ceph-mon[103278]: pgmap v8787: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:54.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:53 smithi078 ceph-mon[91136]: pgmap v8787: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:54.190 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:53 smithi078 ceph-mon[99959]: pgmap v8787: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:55.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:54 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:54 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:55.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:54 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:12:56.130 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:55 smithi146 ceph-mon[103278]: pgmap v8788: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:55 smithi078 ceph-mon[91136]: pgmap v8788: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:55 smithi078 ceph-mon[99959]: pgmap v8788: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:12:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:57 smithi146 ceph-mon[103278]: pgmap v8789: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:57 smithi078 ceph-mon[91136]: pgmap v8789: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:57 smithi078 ceph-mon[99959]: pgmap v8789: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:12:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:12:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:59 smithi146 ceph-mon[103278]: pgmap v8790: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:12:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[91136]: pgmap v8790: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[99959]: pgmap v8790: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:12:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:13:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:13:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:01.370Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:01.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:01.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:01 smithi146 ceph-mon[103278]: pgmap v8791: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:01 smithi078 ceph-mon[91136]: pgmap v8791: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:01 smithi078 ceph-mon[99959]: pgmap v8791: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:04.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[91136]: pgmap v8792: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:04.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[99959]: pgmap v8792: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:04.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:03 smithi146 ceph-mon[103278]: pgmap v8792: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:05 smithi146 ceph-mon[103278]: pgmap v8793: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:05 smithi078 ceph-mon[91136]: pgmap v8793: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:05 smithi078 ceph-mon[99959]: pgmap v8793: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:07 smithi146 ceph-mon[103278]: pgmap v8794: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:07 smithi078 ceph-mon[91136]: pgmap v8794: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:07 smithi078 ceph-mon[99959]: pgmap v8794: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:10.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:09 smithi078 ceph-mon[91136]: pgmap v8795: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:10.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:09 smithi078 ceph-mon[99959]: pgmap v8795: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:09 smithi146 ceph-mon[103278]: pgmap v8795: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:13:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:13:11.536 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:13:11.709 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:11.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:11.709 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:11.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:11.709 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:11.371Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 5h - - 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 5h - - 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 13s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 8s ago 4h 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 13s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 8s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 13s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 13s ago 5h 490M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 8s ago 5h 326M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:13:12.096 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 13s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 13s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 8s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 13s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 13s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 13s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 13s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 8s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 8s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 8s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 8s ago 4h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 8s ago 4h 68.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 13s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 8s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 13s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:13:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 8s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:13:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:11 smithi146 ceph-mon[103278]: pgmap v8796: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:13:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:11 smithi078 ceph-mon[91136]: pgmap v8796: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:13:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:11 smithi078 ceph-mon[99959]: pgmap v8796: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:13:12.416 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:13:12.416 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:13:12.417 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:13:12.714 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:13:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:13:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:13:12.715 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:13:13.034 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:13:13.034 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:13.035 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:12 smithi146 ceph-mon[103278]: from='client.83906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:12 smithi146 ceph-mon[103278]: from='client.63993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:12 smithi146 ceph-mon[103278]: from='client.63999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2565221096' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[91136]: from='client.83906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[91136]: from='client.63993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[91136]: from='client.63999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2565221096' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[99959]: from='client.83906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[99959]: from='client.63993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[99959]: from='client.63999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2565221096' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:14.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:13 smithi146 ceph-mon[103278]: from='client.83930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:14.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:13 smithi146 ceph-mon[103278]: pgmap v8797: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:14.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/117198840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:14.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[91136]: from='client.83930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[91136]: pgmap v8797: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/117198840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[99959]: from='client.83930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[99959]: pgmap v8797: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/117198840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:15.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:15 smithi146 ceph-mon[103278]: pgmap v8798: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:15 smithi078 ceph-mon[91136]: pgmap v8798: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:15 smithi078 ceph-mon[99959]: pgmap v8798: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:17 smithi146 ceph-mon[103278]: pgmap v8799: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:17 smithi078 ceph-mon[91136]: pgmap v8799: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:17 smithi078 ceph-mon[99959]: pgmap v8799: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:20 smithi146 ceph-mon[103278]: pgmap v8800: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[91136]: pgmap v8800: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[99959]: pgmap v8800: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:13:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:13:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:21.371Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:21.372Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:21.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:22 smithi146 ceph-mon[103278]: pgmap v8801: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:22 smithi078 ceph-mon[91136]: pgmap v8801: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:22 smithi078 ceph-mon[99959]: pgmap v8801: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:24 smithi146 ceph-mon[103278]: pgmap v8802: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[91136]: pgmap v8802: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[99959]: pgmap v8802: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:26 smithi146 ceph-mon[103278]: pgmap v8803: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:26 smithi078 ceph-mon[91136]: pgmap v8803: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:26 smithi078 ceph-mon[99959]: pgmap v8803: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:28 smithi146 ceph-mon[103278]: pgmap v8804: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:28 smithi078 ceph-mon[91136]: pgmap v8804: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:28 smithi078 ceph-mon[99959]: pgmap v8804: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:29.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:29.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:30.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:30 smithi078 ceph-mon[91136]: pgmap v8805: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:30.509 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:30 smithi078 ceph-mon[99959]: pgmap v8805: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:30 smithi146 ceph-mon[103278]: pgmap v8805: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:30.762 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:13:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:13:31.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:31.372Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:31.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:31.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:31.745 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:31.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:32 smithi146 ceph-mon[103278]: pgmap v8806: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:32 smithi078 ceph-mon[99959]: pgmap v8806: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:32 smithi078 ceph-mon[91136]: pgmap v8806: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:34 smithi146 ceph-mon[103278]: pgmap v8807: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:34 smithi078 ceph-mon[91136]: pgmap v8807: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:34 smithi078 ceph-mon[99959]: pgmap v8807: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:35.338 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:36.406 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:36 smithi146 ceph-mon[103278]: pgmap v8808: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:36 smithi078 ceph-mon[91136]: pgmap v8808: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:36 smithi078 ceph-mon[99959]: pgmap v8808: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:38 smithi146 ceph-mon[103278]: pgmap v8809: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:38 smithi078 ceph-mon[91136]: pgmap v8809: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:38 smithi078 ceph-mon[99959]: pgmap v8809: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:40 smithi146 ceph-mon[103278]: pgmap v8810: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[91136]: pgmap v8810: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[99959]: pgmap v8810: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:13:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:13:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:41.373Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:41.373Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:41.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:42 smithi146 ceph-mon[103278]: pgmap v8811: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:42 smithi078 ceph-mon[91136]: pgmap v8811: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:42 smithi078 ceph-mon[99959]: pgmap v8811: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:43.328 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:13:43.861 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:13:43.861 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 5h - - 2024-09-21T01:13:43.861 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 5h - - 2024-09-21T01:13:43.861 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 4s ago 4h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 0s ago 4h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 4s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 0s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 4s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 4s ago 5h 490M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 0s ago 5h 327M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 4s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 4s ago 4h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 0s ago 4h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 4s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 4s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 4s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:13:43.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 4s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 0s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 0s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 0s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 0s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 0s ago 4h 68.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 4s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 0s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 4s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:13:43.863 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 0s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:13:44.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:13:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:13:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:13:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:13:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:13:44.441 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:13:44.441 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:13:44.441 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:13:44.442 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: pgmap v8812: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: from='client.64023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/703152321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: pgmap v8812: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: from='client.64023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:44.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/703152321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: pgmap v8812: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: from='client.64023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:13:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/703152321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:13:44.752 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:13:44.752 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:13:44.752 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:13:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:45 smithi146 ceph-mon[103278]: from='client.64029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:45 smithi146 ceph-mon[103278]: from='client.83954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2865155901' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[91136]: from='client.64029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[91136]: from='client.83954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2865155901' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[99959]: from='client.64029 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[99959]: from='client.83954 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2865155901' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:13:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:46 smithi146 ceph-mon[103278]: from='client.83966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:46 smithi146 ceph-mon[103278]: pgmap v8813: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:46 smithi078 ceph-mon[91136]: from='client.83966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:46 smithi078 ceph-mon[91136]: pgmap v8813: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:46 smithi078 ceph-mon[99959]: from='client.83966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:13:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:46 smithi078 ceph-mon[99959]: pgmap v8813: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:48 smithi146 ceph-mon[103278]: pgmap v8814: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:48 smithi078 ceph-mon[91136]: pgmap v8814: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:48 smithi078 ceph-mon[99959]: pgmap v8814: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:50.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:50 smithi078 ceph-mon[99959]: pgmap v8815: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:50.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:50 smithi078 ceph-mon[91136]: pgmap v8815: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:50 smithi146 ceph-mon[103278]: pgmap v8815: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:13:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:13:51.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:51.373Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:51.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:51.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:13:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:13:51.374Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:13:52.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:52 smithi078 ceph-mon[91136]: pgmap v8816: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:52 smithi078 ceph-mon[99959]: pgmap v8816: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:52 smithi146 ceph-mon[103278]: pgmap v8816: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:54 smithi146 ceph-mon[103278]: pgmap v8817: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:54 smithi078 ceph-mon[99959]: pgmap v8817: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:54 smithi078 ceph-mon[91136]: pgmap v8817: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:13:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:56 smithi146 ceph-mon[103278]: pgmap v8818: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:56 smithi078 ceph-mon[99959]: pgmap v8818: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:56 smithi078 ceph-mon[91136]: pgmap v8818: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:13:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:58 smithi146 ceph-mon[103278]: pgmap v8819: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:58 smithi078 ceph-mon[99959]: pgmap v8819: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:58 smithi078 ceph-mon[91136]: pgmap v8819: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:13:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:13:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:13:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:13:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:13:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[99959]: pgmap v8820: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[91136]: pgmap v8820: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:00 smithi146 ceph-mon[103278]: pgmap v8820: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:14:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:14:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:01.374Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:01.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:01.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:02.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:02 smithi078 ceph-mon[99959]: pgmap v8821: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:02.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:02 smithi078 ceph-mon[91136]: pgmap v8821: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:02 smithi146 ceph-mon[103278]: pgmap v8821: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[91136]: pgmap v8822: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[99959]: pgmap v8822: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:04 smithi146 ceph-mon[103278]: pgmap v8822: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:06 smithi146 ceph-mon[103278]: pgmap v8823: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:06 smithi078 ceph-mon[99959]: pgmap v8823: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:06 smithi078 ceph-mon[91136]: pgmap v8823: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:08 smithi146 ceph-mon[103278]: pgmap v8824: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:08 smithi078 ceph-mon[99959]: pgmap v8824: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:08 smithi078 ceph-mon[91136]: pgmap v8824: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:10 smithi146 ceph-mon[103278]: pgmap v8825: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:10 smithi078 ceph-mon[99959]: pgmap v8825: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:10 smithi078 ceph-mon[91136]: pgmap v8825: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:14:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:14:11.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:11.375Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:11.375Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:11.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:11.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:12 smithi078 ceph-mon[91136]: pgmap v8826: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:12 smithi078 ceph-mon[99959]: pgmap v8826: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:12 smithi146 ceph-mon[103278]: pgmap v8826: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:14 smithi078 ceph-mon[91136]: pgmap v8827: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:14 smithi078 ceph-mon[99959]: pgmap v8827: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:14 smithi146 ceph-mon[103278]: pgmap v8827: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:15.048 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:14:15.576 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:14:15.576 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 5h - - 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 16s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (4h) 11s ago 4h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 16s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 11s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 16s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 16s ago 5h 491M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 11s ago 5h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 16s ago 5h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (4h) 16s ago 4h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (4h) 11s ago 4h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 16s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 16s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 16s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:14:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 16s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 11s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 11s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 11s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 11s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 11s ago 5h 66.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 16s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 11s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 16s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:14:15.578 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 11s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:14:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:15.884 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:14:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:14:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:14:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:14:15.885 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:14:16.153 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:14:16.153 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:14:16.153 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:14:16.153 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:14:16.153 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:14:16.153 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:14:16.154 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:14:16.464 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:14:16.464 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:16.465 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:16.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[91136]: pgmap v8828: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:16.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[91136]: from='client.64059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:16.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[91136]: from='client.64065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:16.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4262572825' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:16.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[99959]: pgmap v8828: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:16.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[99959]: from='client.64059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:16.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[99959]: from='client.64065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:16.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4262572825' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:16.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:16 smithi146 ceph-mon[103278]: pgmap v8828: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:16.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:16 smithi146 ceph-mon[103278]: from='client.64059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:16.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:16 smithi146 ceph-mon[103278]: from='client.64065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:16.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4262572825' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:17.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:17 smithi146 ceph-mon[103278]: from='client.83990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:17.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:17 smithi146 ceph-mon[103278]: from='client.84002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:17.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/233807115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:17.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:17 smithi078 ceph-mon[99959]: from='client.83990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:17.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:17 smithi078 ceph-mon[99959]: from='client.84002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/233807115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:17 smithi078 ceph-mon[91136]: from='client.83990 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:17 smithi078 ceph-mon[91136]: from='client.84002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/233807115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:18 smithi146 ceph-mon[103278]: pgmap v8829: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:18 smithi078 ceph-mon[99959]: pgmap v8829: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:18 smithi078 ceph-mon[91136]: pgmap v8829: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:20 smithi146 ceph-mon[103278]: pgmap v8830: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:14:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[91136]: pgmap v8830: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[99959]: pgmap v8830: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:21.375Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:21.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:21.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:22.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:22 smithi078 ceph-mon[99959]: pgmap v8831: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:22.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:22 smithi078 ceph-mon[91136]: pgmap v8831: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:22 smithi146 ceph-mon[103278]: pgmap v8831: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[99959]: pgmap v8832: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[91136]: pgmap v8832: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:24 smithi146 ceph-mon[103278]: pgmap v8832: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 01:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T01:14:25.849164776Z level=info msg="Completed cleanup jobs" duration=13.08878ms 2024-09-21T01:14:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:26 smithi146 ceph-mon[103278]: pgmap v8833: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:26 smithi078 ceph-mon[99959]: pgmap v8833: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:26 smithi078 ceph-mon[91136]: pgmap v8833: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:28 smithi146 ceph-mon[103278]: pgmap v8834: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:28 smithi078 ceph-mon[99959]: pgmap v8834: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:28 smithi078 ceph-mon[91136]: pgmap v8834: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:30 smithi146 ceph-mon[103278]: pgmap v8835: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:14:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:14:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:30 smithi078 ceph-mon[99959]: pgmap v8835: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:30 smithi078 ceph-mon[91136]: pgmap v8835: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:31.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:31.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:31.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:31.376Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:31.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:31.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:32.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:32 smithi078 ceph-mon[99959]: pgmap v8836: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:32.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:32 smithi078 ceph-mon[91136]: pgmap v8836: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:32 smithi146 ceph-mon[103278]: pgmap v8836: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:34 smithi146 ceph-mon[103278]: pgmap v8837: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:34 smithi078 ceph-mon[91136]: pgmap v8837: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:34 smithi078 ceph-mon[99959]: pgmap v8837: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:36 smithi146 ceph-mon[103278]: pgmap v8838: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:36 smithi078 ceph-mon[91136]: pgmap v8838: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:36 smithi078 ceph-mon[99959]: pgmap v8838: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:37.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:37 smithi146 ceph-mon[103278]: pgmap v8839: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:37 smithi078 ceph-mon[91136]: pgmap v8839: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:37 smithi078 ceph-mon[99959]: pgmap v8839: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:39 smithi146 ceph-mon[103278]: pgmap v8840: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[99959]: pgmap v8840: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[91136]: pgmap v8840: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:14:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:14:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:41.376Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:41.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:41.377Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:41 smithi146 ceph-mon[103278]: pgmap v8841: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:41 smithi078 ceph-mon[91136]: pgmap v8841: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:41 smithi078 ceph-mon[99959]: pgmap v8841: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:44.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:43 smithi146 ceph-mon[103278]: pgmap v8842: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:44.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:43 smithi078 ceph-mon[99959]: pgmap v8842: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:43 smithi078 ceph-mon[91136]: pgmap v8842: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:14:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:46 smithi146 ceph-mon[103278]: pgmap v8843: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:46 smithi078 ceph-mon[91136]: pgmap v8843: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:46 smithi078 ceph-mon[99959]: pgmap v8843: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:46.758 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:14:47.286 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:14:47.286 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 5h - - 2024-09-21T01:14:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 5h - - 2024-09-21T01:14:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (4h) 7s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:14:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 3s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:14:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 7s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:14:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 3s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:14:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 7s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 7s ago 5h 492M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 3s ago 5h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 7s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 7s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 3s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 7s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:14:47.288 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 7s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 7s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 7s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 3s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 3s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 3s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 3s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 3s ago 5h 67.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:14:47.289 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 7s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:14:47.290 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 3s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:14:47.290 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 7s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:14:47.290 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 3s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:14:47.597 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:14:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:14:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:14:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:14:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:14:47.598 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:14:47.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:14:47.865 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:14:47.866 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:14:48.177 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:14:48.177 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:48.177 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:14:48.177 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:48.177 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:14:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:48 smithi146 ceph-mon[103278]: from='client.64095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:48 smithi146 ceph-mon[103278]: pgmap v8844: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:48 smithi146 ceph-mon[103278]: from='client.64101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:48 smithi146 ceph-mon[103278]: from='client.64107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/960883317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2858931164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[91136]: from='client.64095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[91136]: pgmap v8844: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[91136]: from='client.64101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[91136]: from='client.64107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/960883317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2858931164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[99959]: from='client.64095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[99959]: pgmap v8844: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[99959]: from='client.64101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[99959]: from='client.64107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/960883317' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:14:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2858931164' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:14:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:49 smithi146 ceph-mon[103278]: from='client.64119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:49.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:49 smithi078 ceph-mon[91136]: from='client.64119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:49.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:49 smithi078 ceph-mon[99959]: from='client.64119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:14:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:50 smithi146 ceph-mon[103278]: pgmap v8845: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:50 smithi078 ceph-mon[91136]: pgmap v8845: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:50 smithi078 ceph-mon[99959]: pgmap v8845: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:14:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:14:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:51.377Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:51.378Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:14:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:14:51.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:14:52.385 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:52 smithi078 ceph-mon[91136]: pgmap v8846: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:52.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:52 smithi078 ceph-mon[99959]: pgmap v8846: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:52 smithi146 ceph-mon[103278]: pgmap v8846: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:54 smithi078 ceph-mon[91136]: pgmap v8847: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:54 smithi078 ceph-mon[99959]: pgmap v8847: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:54 smithi146 ceph-mon[103278]: pgmap v8847: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:55.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:14:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:56 smithi146 ceph-mon[103278]: pgmap v8848: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:56 smithi078 ceph-mon[91136]: pgmap v8848: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:56 smithi078 ceph-mon[99959]: pgmap v8848: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:14:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:58 smithi146 ceph-mon[103278]: pgmap v8849: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:58 smithi078 ceph-mon[99959]: pgmap v8849: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:58 smithi078 ceph-mon[91136]: pgmap v8849: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:14:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:14:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:14:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:14:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:14:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:00 smithi146 ceph-mon[103278]: pgmap v8850: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[91136]: pgmap v8850: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[99959]: pgmap v8850: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:15:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:15:01.723 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:01.378Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:01.724 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:01.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:01.724 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:01.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:02 smithi146 ceph-mon[103278]: pgmap v8851: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:02 smithi078 ceph-mon[91136]: pgmap v8851: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:02 smithi078 ceph-mon[99959]: pgmap v8851: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:04 smithi146 ceph-mon[103278]: pgmap v8852: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:04 smithi078 ceph-mon[91136]: pgmap v8852: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:04 smithi078 ceph-mon[99959]: pgmap v8852: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:06 smithi146 ceph-mon[103278]: pgmap v8853: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:06 smithi078 ceph-mon[91136]: pgmap v8853: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:06 smithi078 ceph-mon[99959]: pgmap v8853: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:08 smithi146 ceph-mon[103278]: pgmap v8854: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:08 smithi078 ceph-mon[91136]: pgmap v8854: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:08 smithi078 ceph-mon[99959]: pgmap v8854: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:10.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:10 smithi078 ceph-mon[99959]: pgmap v8855: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:10 smithi078 ceph-mon[91136]: pgmap v8855: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:10 smithi146 ceph-mon[103278]: pgmap v8855: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:15:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:15:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:11.379Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:11.379Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:11.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:12 smithi146 ceph-mon[103278]: pgmap v8856: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:12 smithi078 ceph-mon[91136]: pgmap v8856: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:12 smithi078 ceph-mon[99959]: pgmap v8856: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:14 smithi078 ceph-mon[91136]: pgmap v8857: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:14 smithi078 ceph-mon[99959]: pgmap v8857: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:14 smithi146 ceph-mon[103278]: pgmap v8857: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:16 smithi146 ceph-mon[103278]: pgmap v8858: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:16 smithi078 ceph-mon[99959]: pgmap v8858: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:16 smithi078 ceph-mon[91136]: pgmap v8858: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:18.478 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:15:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:18 smithi146 ceph-mon[103278]: pgmap v8859: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:18 smithi078 ceph-mon[99959]: pgmap v8859: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:18 smithi078 ceph-mon[91136]: pgmap v8859: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:19.004 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 5h - - 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 14s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 19s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 14s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 19s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 19s ago 5h 493M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 14s ago 5h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 19s ago 5h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 14s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 19s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:15:19.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 19s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 19s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 19s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 14s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 14s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 14s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 14s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 14s ago 5h 67.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 19s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 14s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 19s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:15:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 14s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:15:19.312 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:15:19.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:15:19.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:15:19.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:15:19.314 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:15:19.581 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:15:19.582 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:15:19.582 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1210211563' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:19.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1210211563' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:19.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1210211563' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:19.908 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:15:19.908 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:19.908 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:19.908 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='client.64131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='client.64137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: pgmap v8860: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='client.84062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2443646285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:15:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='client.64131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='client.64137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: pgmap v8860: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='client.84062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2443646285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='client.64131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='client.64137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: pgmap v8860: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='client.84062 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2443646285' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:21.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:21 smithi078 ceph-mon[99959]: from='client.64155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:21.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:21 smithi078 ceph-mon[91136]: from='client.64155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:21.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:21.379Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:21.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:21.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:21.688 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:21.380Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:21 smithi146 ceph-mon[103278]: from='client.64155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:22.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:22 smithi078 ceph-mon[99959]: pgmap v8861: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:22.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:22 smithi078 ceph-mon[91136]: pgmap v8861: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:22 smithi146 ceph-mon[103278]: pgmap v8861: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:24 smithi146 ceph-mon[103278]: pgmap v8862: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:24 smithi078 ceph-mon[99959]: pgmap v8862: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:24 smithi078 ceph-mon[91136]: pgmap v8862: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:26 smithi146 ceph-mon[103278]: pgmap v8863: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:26 smithi078 ceph-mon[99959]: pgmap v8863: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:26 smithi078 ceph-mon[91136]: pgmap v8863: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:28 smithi146 ceph-mon[103278]: pgmap v8864: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:28 smithi078 ceph-mon[99959]: pgmap v8864: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:28 smithi078 ceph-mon[91136]: pgmap v8864: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:30 smithi146 ceph-mon[103278]: pgmap v8865: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:15:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:15:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:30 smithi078 ceph-mon[99959]: pgmap v8865: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:30 smithi078 ceph-mon[91136]: pgmap v8865: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:31.380Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:31.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:31.381Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:32.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:32 smithi078 ceph-mon[99959]: pgmap v8866: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:32.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:32 smithi078 ceph-mon[91136]: pgmap v8866: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:32 smithi146 ceph-mon[103278]: pgmap v8866: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:34.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:34 smithi078 ceph-mon[91136]: pgmap v8867: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:34.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:34 smithi078 ceph-mon[99959]: pgmap v8867: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:34.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:34 smithi146 ceph-mon[103278]: pgmap v8867: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:36 smithi146 ceph-mon[103278]: pgmap v8868: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:36 smithi078 ceph-mon[91136]: pgmap v8868: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:36 smithi078 ceph-mon[99959]: pgmap v8868: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:38 smithi146 ceph-mon[103278]: pgmap v8869: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:38 smithi078 ceph-mon[99959]: pgmap v8869: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:38 smithi078 ceph-mon[91136]: pgmap v8869: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:39 smithi146 ceph-mon[103278]: pgmap v8870: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:39.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:39 smithi078 ceph-mon[99959]: pgmap v8870: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:39 smithi078 ceph-mon[91136]: pgmap v8870: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:40.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:15:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:15:41.192 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.192 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:41.381Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:41.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:41.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:42 smithi146 ceph-mon[103278]: pgmap v8871: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:42 smithi078 ceph-mon[91136]: pgmap v8871: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:42 smithi078 ceph-mon[99959]: pgmap v8871: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:44 smithi146 ceph-mon[103278]: pgmap v8872: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:44 smithi078 ceph-mon[91136]: pgmap v8872: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:44 smithi078 ceph-mon[99959]: pgmap v8872: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:45.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:15:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:46 smithi078 ceph-mon[91136]: pgmap v8873: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:46 smithi078 ceph-mon[99959]: pgmap v8873: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:46 smithi146 ceph-mon[103278]: pgmap v8873: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:48 smithi078 ceph-mon[91136]: pgmap v8874: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:48 smithi078 ceph-mon[99959]: pgmap v8874: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:48 smithi146 ceph-mon[103278]: pgmap v8874: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:50.205 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:15:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:50 smithi078 ceph-mon[99959]: pgmap v8875: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:50 smithi078 ceph-mon[91136]: pgmap v8875: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:50 smithi146 ceph-mon[103278]: pgmap v8875: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 5h - - 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 5h - - 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 10s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 6s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 10s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 6s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 10s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:15:50.729 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 10s ago 5h 493M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 6s ago 5h 328M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 10s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 10s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 6s ago 5h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 10s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 10s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 10s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 10s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 6s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 6s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 6s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 6s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 6s ago 5h 67.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:15:50.730 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 10s ago 4h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:15:50.731 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 6s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:15:50.731 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 10s ago 4h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:15:50.731 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 6s ago 4h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:15:51.025 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:15:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:15:51.039 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:15:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:15:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:15:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:15:51.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:15:51.040 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:15:51.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:51 smithi078 ceph-mon[99959]: from='client.64167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:51 smithi078 ceph-mon[91136]: from='client.64167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:51.307 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:15:51.307 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:15:51.307 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:15:51.307 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:15:51.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:15:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:51 smithi146 ceph-mon[103278]: from='client.64167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:15:51.622 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:51.382Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:51.622 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:51.382Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:15:51.622 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:15:51.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:15:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[91136]: from='client.64173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[91136]: from='client.64179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[91136]: pgmap v8876: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/906029354' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[91136]: from='client.84110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3308375142' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[99959]: from='client.64173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[99959]: from='client.64179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[99959]: pgmap v8876: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/906029354' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[99959]: from='client.84110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3308375142' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:52 smithi146 ceph-mon[103278]: from='client.64173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:52 smithi146 ceph-mon[103278]: from='client.64179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:52 smithi146 ceph-mon[103278]: pgmap v8876: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/906029354' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:15:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:52 smithi146 ceph-mon[103278]: from='client.84110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:15:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3308375142' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:15:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:54 smithi078 ceph-mon[91136]: pgmap v8877: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:54 smithi078 ceph-mon[99959]: pgmap v8877: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:54.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:54 smithi146 ceph-mon[103278]: pgmap v8877: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:15:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:56 smithi146 ceph-mon[103278]: pgmap v8878: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:56 smithi078 ceph-mon[91136]: pgmap v8878: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:56 smithi078 ceph-mon[99959]: pgmap v8878: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:15:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:58 smithi146 ceph-mon[103278]: pgmap v8879: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:58 smithi078 ceph-mon[91136]: pgmap v8879: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:58 smithi078 ceph-mon[99959]: pgmap v8879: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:15:59.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:15:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:00 smithi146 ceph-mon[103278]: pgmap v8880: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:00 smithi078 ceph-mon[91136]: pgmap v8880: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:00 smithi078 ceph-mon[99959]: pgmap v8880: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:16:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:16:01.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:01.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:01.383Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:01.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:02 smithi146 ceph-mon[103278]: pgmap v8881: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:02 smithi078 ceph-mon[91136]: pgmap v8881: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:02 smithi078 ceph-mon[99959]: pgmap v8881: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:04 smithi146 ceph-mon[103278]: pgmap v8882: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:04 smithi078 ceph-mon[91136]: pgmap v8882: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:04 smithi078 ceph-mon[99959]: pgmap v8882: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:06 smithi146 ceph-mon[103278]: pgmap v8883: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:06 smithi078 ceph-mon[91136]: pgmap v8883: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:06 smithi078 ceph-mon[99959]: pgmap v8883: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:08 smithi146 ceph-mon[103278]: pgmap v8884: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:08 smithi078 ceph-mon[91136]: pgmap v8884: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:08 smithi078 ceph-mon[99959]: pgmap v8884: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:10 smithi146 ceph-mon[103278]: pgmap v8885: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:10 smithi078 ceph-mon[91136]: pgmap v8885: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:10 smithi078 ceph-mon[99959]: pgmap v8885: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:16:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:16:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:11.383Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:11.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:11.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:12 smithi146 ceph-mon[103278]: pgmap v8886: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:12 smithi078 ceph-mon[99959]: pgmap v8886: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:12 smithi078 ceph-mon[91136]: pgmap v8886: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:14 smithi146 ceph-mon[103278]: pgmap v8887: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:14 smithi078 ceph-mon[99959]: pgmap v8887: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:14 smithi078 ceph-mon[91136]: pgmap v8887: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:16.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:16 smithi078 ceph-mon[99959]: pgmap v8888: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:16.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:16 smithi078 ceph-mon[91136]: pgmap v8888: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:16 smithi146 ceph-mon[103278]: pgmap v8888: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:18 smithi146 ceph-mon[103278]: pgmap v8889: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:18 smithi078 ceph-mon[91136]: pgmap v8889: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:18 smithi078 ceph-mon[99959]: pgmap v8889: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:20 smithi146 ceph-mon[103278]: pgmap v8890: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:20 smithi078 ceph-mon[91136]: pgmap v8890: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:20 smithi078 ceph-mon[99959]: pgmap v8890: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:16:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:16:21.384 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.384 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.384 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.385 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.385 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.513 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:21.384Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:21.384Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:21.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:21.918 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:16:22.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:22 smithi146 ceph-mon[103278]: pgmap v8891: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 5h - - 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 5h - - 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 2s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 17s ago 5h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 2s ago 4h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 17s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 2s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:16:22.469 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 2s ago 5h 493M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 17s ago 5h 329M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 2s ago 5h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 2s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 17s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 2s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 2s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 2s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 2s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 17s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 17s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 17s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 17s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 17s ago 5h 67.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:16:22.470 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 2s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:16:22.471 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:16:22.471 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 2s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:16:22.471 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:16:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:22 smithi078 ceph-mon[99959]: pgmap v8891: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:22 smithi078 ceph-mon[91136]: pgmap v8891: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:22.791 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:22.792 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:16:22.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:16:22.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:16:22.793 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:16:23.069 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:16:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:16:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:16:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:16:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:16:23.070 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:16:23.388 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:16:23.388 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:23.388 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:23.388 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:23 smithi078 ceph-mon[99959]: from='client.64203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:23 smithi078 ceph-mon[99959]: from='client.64209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/39589732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:23 smithi078 ceph-mon[91136]: from='client.64203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:23 smithi078 ceph-mon[91136]: from='client.64209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:23.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/39589732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:23 smithi146 ceph-mon[103278]: from='client.64203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:23 smithi146 ceph-mon[103278]: from='client.64209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/39589732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[91136]: from='client.84134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[91136]: pgmap v8892: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[91136]: from='client.64227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4163862128' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[99959]: from='client.84134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[99959]: pgmap v8892: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[99959]: from='client.64227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4163862128' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:24.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:24 smithi146 ceph-mon[103278]: from='client.84134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:24.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:24 smithi146 ceph-mon[103278]: pgmap v8892: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:24.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:24 smithi146 ceph-mon[103278]: from='client.64227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:24.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4163862128' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:25.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:25.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:26 smithi146 ceph-mon[103278]: pgmap v8893: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:26 smithi078 ceph-mon[99959]: pgmap v8893: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:26 smithi078 ceph-mon[91136]: pgmap v8893: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:28 smithi146 ceph-mon[103278]: pgmap v8894: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:28 smithi078 ceph-mon[99959]: pgmap v8894: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:28 smithi078 ceph-mon[91136]: pgmap v8894: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:30.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:30 smithi078 ceph-mon[99959]: pgmap v8895: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:30 smithi078 ceph-mon[91136]: pgmap v8895: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:30 smithi146 ceph-mon[103278]: pgmap v8895: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:16:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:16:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:31.384Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:31.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:31.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:32.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:32 smithi078 ceph-mon[91136]: pgmap v8896: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:32.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:32 smithi078 ceph-mon[99959]: pgmap v8896: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:32 smithi146 ceph-mon[103278]: pgmap v8896: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:34 smithi146 ceph-mon[103278]: pgmap v8897: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:34 smithi078 ceph-mon[91136]: pgmap v8897: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:34 smithi078 ceph-mon[99959]: pgmap v8897: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:36.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:36 smithi078 ceph-mon[99959]: pgmap v8898: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:36.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:36 smithi078 ceph-mon[91136]: pgmap v8898: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:36 smithi146 ceph-mon[103278]: pgmap v8898: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:38 smithi078 ceph-mon[91136]: pgmap v8899: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:38 smithi078 ceph-mon[99959]: pgmap v8899: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:38 smithi146 ceph-mon[103278]: pgmap v8899: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:40 smithi078 ceph-mon[91136]: pgmap v8900: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:40 smithi078 ceph-mon[99959]: pgmap v8900: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:40.599 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:40 smithi146 ceph-mon[103278]: pgmap v8900: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:16:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:16:41.646 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:41.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:41.385Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:41.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:42 smithi146 ceph-mon[103278]: pgmap v8901: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:42 smithi078 ceph-mon[99959]: pgmap v8901: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:42 smithi078 ceph-mon[91136]: pgmap v8901: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:44 smithi146 ceph-mon[103278]: pgmap v8902: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:44 smithi078 ceph-mon[91136]: pgmap v8902: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:44 smithi078 ceph-mon[99959]: pgmap v8902: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:16:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:46 smithi146 ceph-mon[103278]: pgmap v8903: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:46 smithi078 ceph-mon[91136]: pgmap v8903: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:46 smithi078 ceph-mon[99959]: pgmap v8903: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:48 smithi146 ceph-mon[103278]: pgmap v8904: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:48.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:48 smithi078 ceph-mon[91136]: pgmap v8904: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:48 smithi078 ceph-mon[99959]: pgmap v8904: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:50 smithi078 ceph-mon[91136]: pgmap v8905: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:50 smithi078 ceph-mon[99959]: pgmap v8905: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:50 smithi146 ceph-mon[103278]: pgmap v8905: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:16:50] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T01:16:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:51.385Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:51.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:16:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:16:51.386Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:16:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:52 smithi146 ceph-mon[103278]: pgmap v8906: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:52 smithi078 ceph-mon[91136]: pgmap v8906: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:52 smithi078 ceph-mon[99959]: pgmap v8906: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:53.687 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 5h - - 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 5h - - 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 13s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 9s ago 5h 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 13s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 9s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:16:54.215 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 13s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 13s ago 5h 494M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 9s ago 5h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 13s ago 5h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 13s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 9s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 13s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 13s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 13s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 13s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 9s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 9s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 9s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 9s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 9s ago 5h 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:16:54.216 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 13s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:16:54.217 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:16:54.217 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 13s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:16:54.217 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:16:54.526 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:16:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:16:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:16:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:16:54.528 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:16:54.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:54 smithi078 ceph-mon[99959]: pgmap v8907: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:54.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:54 smithi078 ceph-mon[91136]: pgmap v8907: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:54.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:54 smithi146 ceph-mon[103278]: pgmap v8907: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:16:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:16:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:16:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:16:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:16:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:16:54.799 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:16:55.107 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:16:55.107 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:16:55.107 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:55.107 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:55 smithi146 ceph-mon[103278]: from='client.64239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:55 smithi146 ceph-mon[103278]: from='client.84164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:55 smithi146 ceph-mon[103278]: from='client.84170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/560458512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/409143832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[99959]: from='client.64239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[99959]: from='client.84164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[99959]: from='client.84170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/560458512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/409143832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[91136]: from='client.64239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[91136]: from='client.84164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[91136]: from='client.84170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/560458512' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:16:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/409143832' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:16:56.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:56 smithi078 ceph-mon[91136]: from='client.84182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:56.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:56 smithi078 ceph-mon[91136]: pgmap v8908: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:56.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:56 smithi078 ceph-mon[99959]: from='client.84182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:56.708 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:56 smithi078 ceph-mon[99959]: pgmap v8908: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:56 smithi146 ceph-mon[103278]: from='client.84182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:16:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:56 smithi146 ceph-mon[103278]: pgmap v8908: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:16:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:58 smithi146 ceph-mon[103278]: pgmap v8909: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:58 smithi078 ceph-mon[91136]: pgmap v8909: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:58 smithi078 ceph-mon[99959]: pgmap v8909: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:16:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:16:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:00 smithi146 ceph-mon[103278]: pgmap v8910: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:00 smithi078 ceph-mon[91136]: pgmap v8910: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:17:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T01:17:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:00 smithi078 ceph-mon[99959]: pgmap v8910: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:01 smithi146 ceph-mon[103278]: pgmap v8911: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[91136]: pgmap v8911: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-mon[99959]: pgmap v8911: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:01.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:01.386Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:01.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:01.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:01.788 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:01.387Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:03 smithi146 ceph-mon[103278]: pgmap v8912: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:03 smithi078 ceph-mon[91136]: pgmap v8912: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:03 smithi078 ceph-mon[99959]: pgmap v8912: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:06 smithi146 ceph-mon[103278]: pgmap v8913: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[99959]: pgmap v8913: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[91136]: pgmap v8913: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:08 smithi146 ceph-mon[103278]: pgmap v8914: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:08 smithi078 ceph-mon[91136]: pgmap v8914: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:08 smithi078 ceph-mon[99959]: pgmap v8914: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:10 smithi146 ceph-mon[103278]: pgmap v8915: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:10 smithi078 ceph-mon[99959]: pgmap v8915: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:10 smithi078 ceph-mon[91136]: pgmap v8915: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:17:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T01:17:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:11.387Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:11.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:11.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:12 smithi146 ceph-mon[103278]: pgmap v8916: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:12 smithi078 ceph-mon[91136]: pgmap v8916: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:12 smithi078 ceph-mon[99959]: pgmap v8916: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:14.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:14 smithi078 ceph-mon[91136]: pgmap v8917: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:14.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:14.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:14 smithi078 ceph-mon[99959]: pgmap v8917: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:14.425 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:14 smithi146 ceph-mon[103278]: pgmap v8917: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:15.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:16 smithi146 ceph-mon[103278]: pgmap v8918: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:16 smithi078 ceph-mon[91136]: pgmap v8918: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:16 smithi078 ceph-mon[99959]: pgmap v8918: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:18.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:18 smithi078 ceph-mon[91136]: pgmap v8919: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:18.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:18 smithi078 ceph-mon[99959]: pgmap v8919: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:18 smithi146 ceph-mon[103278]: pgmap v8919: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:20.337 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:20 smithi078 ceph-mon[99959]: pgmap v8920: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:20.337 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:20 smithi078 ceph-mon[91136]: pgmap v8920: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:20 smithi146 ceph-mon[103278]: pgmap v8920: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:17:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:17:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[91136]: pgmap v8921: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-mon[99959]: pgmap v8921: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:21.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:21.388Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:21.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:21.388Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:21.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:21.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:21.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.883 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:21.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:21 smithi146 ceph-mon[103278]: pgmap v8921: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:23 smithi146 ceph-mon[103278]: pgmap v8922: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:23 smithi078 ceph-mon[91136]: pgmap v8922: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:23 smithi078 ceph-mon[99959]: pgmap v8922: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:25.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:25.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:25.477 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 5h - - 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 5h - - 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 5s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 0s ago 5h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 5s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 0s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 5s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 5s ago 5h 495M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 0s ago 5h 331M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:17:25.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 5s ago 5h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 5s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 0s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 5s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 5s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 5s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 5s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 0s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 0s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 0s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 0s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 0s ago 5h 70.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 5s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:17:25.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:17:25.954 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 5s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:17:25.954 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:17:26.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:17:26.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:17:26.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:17:26.263 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:17:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:26 smithi146 ceph-mon[103278]: pgmap v8923: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:26 smithi146 ceph-mon[103278]: from='client.64275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:26.530 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:17:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:17:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:17:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:17:26.531 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:17:26.531 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[99959]: pgmap v8923: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:26.531 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[99959]: from='client.64275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[91136]: pgmap v8923: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:26.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:26 smithi078 ceph-mon[91136]: from='client.64275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:27 smithi146 ceph-mon[103278]: from='client.84200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:27 smithi146 ceph-mon[103278]: from='client.84206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4294899491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/117446704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[91136]: from='client.84200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[91136]: from='client.84206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4294899491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/117446704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[99959]: from='client.84200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[99959]: from='client.84206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4294899491' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/117446704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:28 smithi146 ceph-mon[103278]: from='client.84218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:28 smithi146 ceph-mon[103278]: pgmap v8924: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:28 smithi078 ceph-mon[91136]: from='client.84218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:28 smithi078 ceph-mon[91136]: pgmap v8924: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:28 smithi078 ceph-mon[99959]: from='client.84218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:28 smithi078 ceph-mon[99959]: pgmap v8924: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:30 smithi146 ceph-mon[103278]: pgmap v8925: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:30 smithi078 ceph-mon[91136]: pgmap v8925: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:30 smithi078 ceph-mon[99959]: pgmap v8925: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:17:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T01:17:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:31.388Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:31.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:31.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:32 smithi146 ceph-mon[103278]: pgmap v8926: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:32 smithi078 ceph-mon[99959]: pgmap v8926: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:32 smithi078 ceph-mon[91136]: pgmap v8926: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:17:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:34 smithi146 ceph-mon[103278]: pgmap v8927: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:34 smithi078 ceph-mon[91136]: pgmap v8927: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:34 smithi078 ceph-mon[99959]: pgmap v8927: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:36 smithi078 ceph-mon[99959]: pgmap v8928: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:36 smithi078 ceph-mon[91136]: pgmap v8928: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:36 smithi146 ceph-mon[103278]: pgmap v8928: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:38.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:38 smithi078 ceph-mon[91136]: pgmap v8929: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:38.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:38 smithi078 ceph-mon[99959]: pgmap v8929: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:38 smithi146 ceph-mon[103278]: pgmap v8929: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:40.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:40 smithi078 ceph-mon[91136]: pgmap v8930: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:40.479 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:40 smithi078 ceph-mon[99959]: pgmap v8930: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:40.758 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:17:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T01:17:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:40 smithi146 ceph-mon[103278]: pgmap v8930: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:41.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:41.389Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:41.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:41.389Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:41.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:41.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:41 smithi146 ceph-mon[103278]: pgmap v8931: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[99959]: pgmap v8931: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:41 smithi078 ceph-mon[91136]: pgmap v8931: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:43 smithi146 ceph-mon[103278]: pgmap v8932: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:43 smithi078 ceph-mon[91136]: pgmap v8932: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:43 smithi078 ceph-mon[99959]: pgmap v8932: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:46 smithi146 ceph-mon[103278]: pgmap v8933: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[99959]: pgmap v8933: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[91136]: pgmap v8933: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:17:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:48 smithi146 ceph-mon[103278]: pgmap v8934: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:48 smithi078 ceph-mon[99959]: pgmap v8934: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:48 smithi078 ceph-mon[91136]: pgmap v8934: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:50 smithi146 ceph-mon[103278]: pgmap v8935: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:50.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:50 smithi078 ceph-mon[99959]: pgmap v8935: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:50 smithi078 ceph-mon[91136]: pgmap v8935: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:17:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:17:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:51.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:51.390Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:17:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:17:51.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:17:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:52 smithi146 ceph-mon[103278]: pgmap v8936: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:52 smithi078 ceph-mon[99959]: pgmap v8936: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:52 smithi078 ceph-mon[91136]: pgmap v8936: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:54 smithi146 ceph-mon[103278]: pgmap v8937: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:54 smithi078 ceph-mon[99959]: pgmap v8937: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:54 smithi078 ceph-mon[91136]: pgmap v8937: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:17:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:17:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:56 smithi146 ceph-mon[103278]: pgmap v8938: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:56 smithi078 ceph-mon[91136]: pgmap v8938: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:56 smithi078 ceph-mon[99959]: pgmap v8938: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:57.130 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:17:57.661 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 5h - - 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 16s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 12s ago 5h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 16s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 12s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 16s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 16s ago 5h 496M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 12s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 16s ago 5h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 16s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:17:57.662 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 12s ago 5h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 16s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 16s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 16s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 16s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 12s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 12s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 12s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 12s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 12s ago 5h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 16s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 16s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:17:57.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:17:57.971 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:17:57.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:17:57.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:17:57.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:57.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:17:57.972 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:17:58.238 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:17:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:17:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:17:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:17:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:17:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:17:58.239 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:17:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[91136]: pgmap v8939: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[91136]: from='client.64311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[91136]: from='client.64317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/88343225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[99959]: pgmap v8939: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[99959]: from='client.64311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[99959]: from='client.64317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/88343225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:58.548 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:17:58.548 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:17:58.548 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:17:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:58 smithi146 ceph-mon[103278]: pgmap v8939: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:17:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:58 smithi146 ceph-mon[103278]: from='client.64311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:58 smithi146 ceph-mon[103278]: from='client.64317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/88343225' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:17:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[91136]: from='client.64323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[91136]: from='client.64335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2437100400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[99959]: from='client.64323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[99959]: from='client.64335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:17:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2437100400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:17:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:59 smithi146 ceph-mon[103278]: from='client.64323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:59 smithi146 ceph-mon[103278]: from='client.64335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:17:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:17:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:17:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2437100400' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:00 smithi078 ceph-mon[91136]: pgmap v8940: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:00 smithi078 ceph-mon[99959]: pgmap v8940: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:00 smithi146 ceph-mon[103278]: pgmap v8940: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:00.882 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:18:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:18:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:01.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:01.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:01.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:01.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[99959]: pgmap v8941: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:01 smithi078 ceph-mon[91136]: pgmap v8941: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:01 smithi146 ceph-mon[103278]: pgmap v8941: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:03 smithi146 ceph-mon[103278]: pgmap v8942: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:03 smithi078 ceph-mon[99959]: pgmap v8942: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:03 smithi078 ceph-mon[91136]: pgmap v8942: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:05 smithi146 ceph-mon[103278]: pgmap v8943: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[99959]: pgmap v8943: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[91136]: pgmap v8943: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:07 smithi146 ceph-mon[103278]: pgmap v8944: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:07 smithi078 ceph-mon[99959]: pgmap v8944: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:07 smithi078 ceph-mon[91136]: pgmap v8944: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:09 smithi146 ceph-mon[103278]: pgmap v8945: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:09 smithi078 ceph-mon[99959]: pgmap v8945: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:09 smithi078 ceph-mon[91136]: pgmap v8945: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:18:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:18:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:11.390Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:11.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:11.391Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:11 smithi146 ceph-mon[103278]: pgmap v8946: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:11 smithi078 ceph-mon[99959]: pgmap v8946: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:11 smithi078 ceph-mon[91136]: pgmap v8946: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:13 smithi146 ceph-mon[103278]: pgmap v8947: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:13 smithi078 ceph-mon[99959]: pgmap v8947: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:13 smithi078 ceph-mon[91136]: pgmap v8947: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:15 smithi146 ceph-mon[103278]: pgmap v8948: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:15 smithi078 ceph-mon[99959]: pgmap v8948: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:15 smithi078 ceph-mon[91136]: pgmap v8948: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:17 smithi146 ceph-mon[103278]: pgmap v8949: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:17 smithi078 ceph-mon[99959]: pgmap v8949: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:17 smithi078 ceph-mon[91136]: pgmap v8949: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:19 smithi146 ceph-mon[103278]: pgmap v8950: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:19 smithi078 ceph-mon[91136]: pgmap v8950: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:19 smithi078 ceph-mon[99959]: pgmap v8950: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:20.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:18:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:18:21.495 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:21.391Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:21.495 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:21.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:21.495 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:21.392Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:22.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[91136]: pgmap v8951: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:22.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.207 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[99959]: pgmap v8951: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:22.207 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:21 smithi146 ceph-mon[103278]: pgmap v8951: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:24.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:23 smithi146 ceph-mon[103278]: pgmap v8952: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:23 smithi078 ceph-mon[91136]: pgmap v8952: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:23 smithi078 ceph-mon[99959]: pgmap v8952: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:25 smithi146 ceph-mon[103278]: pgmap v8953: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[91136]: pgmap v8953: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[99959]: pgmap v8953: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:27 smithi146 ceph-mon[103278]: pgmap v8954: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:27 smithi078 ceph-mon[91136]: pgmap v8954: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:27 smithi078 ceph-mon[99959]: pgmap v8954: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:28.847 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:18:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 5h - - 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 5h - - 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 8s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 3s ago 5h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 8s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 3s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 8s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 8s ago 5h 496M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 3s ago 5h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:18:29.371 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 8s ago 5h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 8s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 3s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 8s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 8s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 8s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 8s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 3s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 3s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 3s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 3s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 3s ago 5h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 8s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 8s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:18:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:18:29.685 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:18:29.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:18:29.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:18:29.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:18:29.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:18:29.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:18:29.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:18:29.687 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:18:29.956 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:18:29.957 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:18:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:30 smithi146 ceph-mon[103278]: from='client.84266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:30 smithi146 ceph-mon[103278]: pgmap v8955: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:30 smithi146 ceph-mon[103278]: from='client.84272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:30 smithi146 ceph-mon[103278]: from='client.64359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3932948257' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:30.266 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:18:30.266 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:18:30.266 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:18:30.266 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:18:30.266 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:18:30.266 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[99959]: from='client.84266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[99959]: pgmap v8955: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[99959]: from='client.84272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[99959]: from='client.64359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3932948257' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[91136]: from='client.84266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[91136]: pgmap v8955: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[91136]: from='client.84272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[91136]: from='client.64359 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:30.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3932948257' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:18:31.012 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:18:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:18:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:31 smithi146 ceph-mon[103278]: from='client.84290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1846179894' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-mon[91136]: from='client.84290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1846179894' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-mon[99959]: from='client.84290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:18:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1846179894' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:18:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:31.392Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:31.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:31.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:32 smithi146 ceph-mon[103278]: pgmap v8956: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:32 smithi078 ceph-mon[91136]: pgmap v8956: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:32 smithi078 ceph-mon[99959]: pgmap v8956: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:34 smithi078 ceph-mon[91136]: pgmap v8957: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:34 smithi078 ceph-mon[99959]: pgmap v8957: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:34 smithi146 ceph-mon[103278]: pgmap v8957: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:36 smithi078 ceph-mon[91136]: pgmap v8958: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:36 smithi078 ceph-mon[99959]: pgmap v8958: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:36 smithi146 ceph-mon[103278]: pgmap v8958: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:38 smithi146 ceph-mon[103278]: pgmap v8959: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:38 smithi078 ceph-mon[91136]: pgmap v8959: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:38 smithi078 ceph-mon[99959]: pgmap v8959: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:40 smithi078 ceph-mon[91136]: pgmap v8960: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:40 smithi078 ceph-mon[99959]: pgmap v8960: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:40 smithi146 ceph-mon[103278]: pgmap v8960: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:40.921 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:18:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:18:41.649 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:41.392Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:41.649 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:41.393Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:41.649 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:41.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[91136]: pgmap v8961: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[99959]: pgmap v8961: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:42 smithi146 ceph-mon[103278]: pgmap v8961: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:44.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:44 smithi078 ceph-mon[91136]: pgmap v8962: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:44.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:44.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:44 smithi078 ceph-mon[99959]: pgmap v8962: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:44.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:44 smithi146 ceph-mon[103278]: pgmap v8962: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:46.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:46.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:46 smithi146 ceph-mon[103278]: pgmap v8963: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:46.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[91136]: pgmap v8963: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[99959]: pgmap v8963: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:18:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:48 smithi146 ceph-mon[103278]: pgmap v8964: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:48 smithi078 ceph-mon[91136]: pgmap v8964: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:48 smithi078 ceph-mon[99959]: pgmap v8964: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:50 smithi146 ceph-mon[103278]: pgmap v8965: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:50 smithi078 ceph-mon[99959]: pgmap v8965: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:50 smithi078 ceph-mon[91136]: pgmap v8965: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:18:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:18:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:51.394Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:51.394Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:18:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:18:51.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:18:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:52 smithi146 ceph-mon[103278]: pgmap v8966: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:52 smithi078 ceph-mon[99959]: pgmap v8966: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:52 smithi078 ceph-mon[91136]: pgmap v8966: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:54.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:54 smithi078 ceph-mon[99959]: pgmap v8967: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:54.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:54 smithi078 ceph-mon[91136]: pgmap v8967: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:54 smithi146 ceph-mon[103278]: pgmap v8967: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:18:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:56 smithi146 ceph-mon[103278]: pgmap v8968: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:56 smithi078 ceph-mon[99959]: pgmap v8968: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:18:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:56 smithi078 ceph-mon[91136]: pgmap v8968: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:58 smithi146 ceph-mon[103278]: pgmap v8969: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:58 smithi078 ceph-mon[99959]: pgmap v8969: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:58 smithi078 ceph-mon[91136]: pgmap v8969: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:18:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:18:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:18:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:18:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:18:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:00 smithi146 ceph-mon[103278]: pgmap v8970: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:00 smithi078 ceph-mon[91136]: pgmap v8970: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:00 smithi078 ceph-mon[99959]: pgmap v8970: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:00.563 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:19:00.855 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:19:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:19:01.106 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:19:01.106 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T01:19:01.106 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 5h - - 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 15s ago 5h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 19s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 15s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 19s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 19s ago 5h 496M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 15s ago 5h 330M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 19s ago 5h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 15s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 19s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 19s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:19:01.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 19s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 19s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 15s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 15s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 15s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 15s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 15s ago 5h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 19s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 19s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:19:01.108 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:19:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:19:01.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:19:01.434 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:19:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:19:01.712 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:01.395Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:01.713 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:01.395Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:01.713 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:01.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:19:01.713 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:19:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:19:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:19:01.714 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:19:01.714 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:19:02.028 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:19:02.028 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:19:02.028 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:19:02.028 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:19:02.028 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:19:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='client.84302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='client.84308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: pgmap v8971: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='client.84314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1258844517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2431571586' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='client.84302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='client.84308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: pgmap v8971: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='client.84314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1258844517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2431571586' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='client.84302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='client.84308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: pgmap v8971: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='client.84314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1258844517' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2431571586' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:03.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:03 smithi078 ceph-mon[91136]: from='client.64407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:03.396 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:03 smithi078 ceph-mon[99959]: from='client.64407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:03 smithi146 ceph-mon[103278]: from='client.64407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:04.509 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:04 smithi078 ceph-mon[99959]: pgmap v8972: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:04.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:04 smithi078 ceph-mon[91136]: pgmap v8972: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:04 smithi146 ceph-mon[103278]: pgmap v8972: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:06.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:06.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:06 smithi146 ceph-mon[103278]: pgmap v8973: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:06.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.467 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[99959]: pgmap v8973: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[91136]: pgmap v8973: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:08 smithi146 ceph-mon[103278]: pgmap v8974: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:08 smithi078 ceph-mon[99959]: pgmap v8974: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:08 smithi078 ceph-mon[91136]: pgmap v8974: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:10 smithi146 ceph-mon[103278]: pgmap v8975: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:10 smithi078 ceph-mon[99959]: pgmap v8975: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:10 smithi078 ceph-mon[91136]: pgmap v8975: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:19:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:19:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:11.395Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:11.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:11.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:12 smithi078 ceph-mon[99959]: pgmap v8976: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:12 smithi078 ceph-mon[91136]: pgmap v8976: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:12 smithi146 ceph-mon[103278]: pgmap v8976: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:14.544 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:14 smithi078 ceph-mon[99959]: pgmap v8977: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:14.545 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:14.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:14 smithi078 ceph-mon[91136]: pgmap v8977: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:14.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:14 smithi146 ceph-mon[103278]: pgmap v8977: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:16 smithi078 ceph-mon[91136]: pgmap v8978: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:16 smithi078 ceph-mon[99959]: pgmap v8978: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:16 smithi146 ceph-mon[103278]: pgmap v8978: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:18 smithi078 ceph-mon[91136]: pgmap v8979: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:18 smithi078 ceph-mon[99959]: pgmap v8979: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:18 smithi146 ceph-mon[103278]: pgmap v8979: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:20 smithi078 ceph-mon[91136]: pgmap v8980: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:20 smithi078 ceph-mon[99959]: pgmap v8980: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:20 smithi146 ceph-mon[103278]: pgmap v8980: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:19:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:19:21.733 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:21.396Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:21.733 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:21.396Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:21.733 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:21.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:22.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[99959]: pgmap v8981: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:22.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[91136]: pgmap v8981: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:22.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:22 smithi146 ceph-mon[103278]: pgmap v8981: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:19:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:24 smithi146 ceph-mon[103278]: pgmap v8982: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:24 smithi078 ceph-mon[91136]: pgmap v8982: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:24 smithi078 ceph-mon[99959]: pgmap v8982: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:26.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:26.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[99959]: pgmap v8983: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:26.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:26.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[91136]: pgmap v8983: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:26.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:26 smithi146 ceph-mon[103278]: pgmap v8983: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:28 smithi146 ceph-mon[103278]: pgmap v8984: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:28 smithi078 ceph-mon[99959]: pgmap v8984: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:28 smithi078 ceph-mon[91136]: pgmap v8984: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:30 smithi146 ceph-mon[103278]: pgmap v8985: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:30 smithi078 ceph-mon[91136]: pgmap v8985: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:30 smithi078 ceph-mon[99959]: pgmap v8985: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:19:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:19:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:31.396Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:31.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:31.397Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:32.331 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:19:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:32 smithi146 ceph-mon[103278]: pgmap v8986: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:32 smithi078 ceph-mon[99959]: pgmap v8986: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:32 smithi078 ceph-mon[91136]: pgmap v8986: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 5h - - 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 5h - - 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 11s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 6s ago 5h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 11s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:19:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 6s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 11s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 11s ago 5h 497M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 6s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 11s ago 5h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 11s ago 5h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 6s ago 5h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 11s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 11s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 11s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 11s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 6s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 6s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 6s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:19:32.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 6s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:19:32.869 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 6s ago 5h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:19:32.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 11s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:19:32.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:19:32.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 11s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:19:32.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:19:33.180 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:19:33.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:19:33.182 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:19:33.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:19:33.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:19:33.182 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:19:33.455 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:19:33.456 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:19:33.456 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:19:33.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:33 smithi078 ceph-mon[99959]: from='client.84338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:33.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1761126817' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:33.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:33 smithi078 ceph-mon[91136]: from='client.84338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:33.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1761126817' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:19:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:19:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:19:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:19:33.763 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:19:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:33 smithi146 ceph-mon[103278]: from='client.84338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1761126817' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:19:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:34 smithi146 ceph-mon[103278]: from='client.64425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:34 smithi146 ceph-mon[103278]: from='client.64431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:34 smithi146 ceph-mon[103278]: pgmap v8987: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:34 smithi146 ceph-mon[103278]: from='client.84362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1442565683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[91136]: from='client.64425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[91136]: from='client.64431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[91136]: pgmap v8987: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[91136]: from='client.84362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1442565683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[99959]: from='client.64425 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[99959]: from='client.64431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[99959]: pgmap v8987: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[99959]: from='client.84362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:19:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1442565683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:19:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:36 smithi146 ceph-mon[103278]: pgmap v8988: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:36 smithi078 ceph-mon[91136]: pgmap v8988: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:36 smithi078 ceph-mon[99959]: pgmap v8988: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:38.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:38 smithi146 ceph-mon[103278]: pgmap v8989: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:38 smithi078 ceph-mon[91136]: pgmap v8989: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:38 smithi078 ceph-mon[99959]: pgmap v8989: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:40 smithi146 ceph-mon[103278]: pgmap v8990: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:40 smithi078 ceph-mon[91136]: pgmap v8990: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:40 smithi078 ceph-mon[99959]: pgmap v8990: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:19:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:19:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:41.397Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:41.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:41.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:42 smithi146 ceph-mon[103278]: pgmap v8991: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[91136]: pgmap v8991: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[99959]: pgmap v8991: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:44.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:44 smithi078 ceph-mon[91136]: pgmap v8992: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:44.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:44.552 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:44 smithi078 ceph-mon[99959]: pgmap v8992: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:44.553 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:44 smithi146 ceph-mon[103278]: pgmap v8992: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:46.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:46.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:46 smithi146 ceph-mon[103278]: pgmap v8993: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:46.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[91136]: pgmap v8993: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[99959]: pgmap v8993: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:19:48.576 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:48 smithi146 ceph-mon[103278]: pgmap v8994: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:48 smithi078 ceph-mon[99959]: pgmap v8994: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:48 smithi078 ceph-mon[91136]: pgmap v8994: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:50 smithi146 ceph-mon[103278]: pgmap v8995: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:19:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:19:50.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:50 smithi078 ceph-mon[99959]: pgmap v8995: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:50 smithi078 ceph-mon[91136]: pgmap v8995: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:51.398Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:51.398Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:19:51.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:19:51.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:19:52.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:52 smithi078 ceph-mon[99959]: pgmap v8996: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:52.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:52 smithi078 ceph-mon[91136]: pgmap v8996: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:52 smithi146 ceph-mon[103278]: pgmap v8996: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:54 smithi146 ceph-mon[103278]: pgmap v8997: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:54 smithi078 ceph-mon[91136]: pgmap v8997: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:54 smithi078 ceph-mon[99959]: pgmap v8997: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:19:56.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:56.750 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:56 smithi146 ceph-mon[103278]: pgmap v8998: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:56 smithi078 ceph-mon[91136]: pgmap v8998: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:19:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:56 smithi078 ceph-mon[99959]: pgmap v8998: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:58 smithi146 ceph-mon[103278]: pgmap v8999: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:58 smithi078 ceph-mon[91136]: pgmap v8999: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:58 smithi078 ceph-mon[99959]: pgmap v8999: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:19:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:19:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:19:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:19:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:19:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T01:19:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:00 smithi146 ceph-mon[103278]: pgmap v9000: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:00 smithi078 ceph-mon[99959]: pgmap v9000: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:00 smithi078 ceph-mon[91136]: pgmap v9000: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:20:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:20:01.654 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:01.399Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:01.654 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:01.399Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:01.654 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:01.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:02.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[99959]: pgmap v9001: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:02.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[91136]: pgmap v9001: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:02.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:02 smithi146 ceph-mon[103278]: pgmap v9001: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:04.107 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:20:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:04 smithi078 ceph-mon[91136]: pgmap v9002: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:04 smithi078 ceph-mon[99959]: pgmap v9002: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 5h - - 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 18s ago 5h 92.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 3s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:20:04.660 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 18s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 3s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 3s ago 5h 498M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 18s ago 5h 332M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 5h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 18s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 18s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 18s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 18s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 18s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:20:04.661 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 18s ago 5h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:20:04.662 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:20:04.662 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:20:04.662 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:20:04.662 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:20:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:04 smithi146 ceph-mon[103278]: pgmap v9002: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:04.990 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:20:04.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:20:04.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:20:04.992 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:20:05.272 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:20:05.273 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:20:05.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:05 smithi078 ceph-mon[99959]: from='client.64455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:05.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:05 smithi078 ceph-mon[99959]: from='client.64461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:05.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/920645370' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:05.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:05 smithi078 ceph-mon[91136]: from='client.64455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:05.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:05 smithi078 ceph-mon[91136]: from='client.64461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:05.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/920645370' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:05.599 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:05.599 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:05.599 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:05.599 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:05 smithi146 ceph-mon[103278]: from='client.64455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:05 smithi146 ceph-mon[103278]: from='client.64461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/920645370' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='client.64467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: pgmap v9003: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='client.64479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3822055115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='client.64467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: pgmap v9003: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='client.64479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3822055115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='client.64467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: pgmap v9003: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='client.64479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3822055115' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:07 smithi078 ceph-mon[91136]: pgmap v9004: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:07 smithi078 ceph-mon[99959]: pgmap v9004: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:07.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:07 smithi146 ceph-mon[103278]: pgmap v9004: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:10 smithi078 ceph-mon[91136]: pgmap v9005: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:10 smithi078 ceph-mon[99959]: pgmap v9005: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:10 smithi146 ceph-mon[103278]: pgmap v9005: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:20:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:20:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:11.400Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:11.400Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:11.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:12 smithi078 ceph-mon[99959]: pgmap v9006: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:12 smithi078 ceph-mon[91136]: pgmap v9006: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:12 smithi146 ceph-mon[103278]: pgmap v9006: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:14 smithi078 ceph-mon[99959]: pgmap v9007: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:14 smithi078 ceph-mon[91136]: pgmap v9007: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:14 smithi146 ceph-mon[103278]: pgmap v9007: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:16 smithi078 ceph-mon[91136]: pgmap v9008: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:16 smithi078 ceph-mon[99959]: pgmap v9008: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:16 smithi146 ceph-mon[103278]: pgmap v9008: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:18 smithi078 ceph-mon[91136]: pgmap v9009: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:18 smithi078 ceph-mon[99959]: pgmap v9009: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:18 smithi146 ceph-mon[103278]: pgmap v9009: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:20 smithi078 ceph-mon[91136]: pgmap v9010: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:20.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:20 smithi078 ceph-mon[99959]: pgmap v9010: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:20 smithi146 ceph-mon[103278]: pgmap v9010: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:20:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:20:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:21.401Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:21.401Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:21.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[99959]: pgmap v9011: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[91136]: pgmap v9011: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:22 smithi146 ceph-mon[103278]: pgmap v9011: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:24 smithi078 ceph-mon[91136]: pgmap v9012: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:24 smithi078 ceph-mon[99959]: pgmap v9012: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:24.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:24 smithi146 ceph-mon[103278]: pgmap v9012: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:26 smithi078 ceph-mon[99959]: pgmap v9013: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:26 smithi078 ceph-mon[91136]: pgmap v9013: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:26.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:26.414 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:26 smithi146 ceph-mon[103278]: pgmap v9013: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:27.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:28.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:28 smithi146 ceph-mon[103278]: pgmap v9014: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:28 smithi078 ceph-mon[91136]: pgmap v9014: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:28 smithi078 ceph-mon[99959]: pgmap v9014: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:29.609 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:30 smithi146 ceph-mon[103278]: pgmap v9015: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:30 smithi078 ceph-mon[99959]: pgmap v9015: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:30 smithi078 ceph-mon[91136]: pgmap v9015: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:20:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:20:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:31.401Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:31.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:31.402Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:32 smithi146 ceph-mon[103278]: pgmap v9016: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:32 smithi078 ceph-mon[91136]: pgmap v9016: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:32 smithi078 ceph-mon[99959]: pgmap v9016: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:34 smithi146 ceph-mon[103278]: pgmap v9017: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:34 smithi078 ceph-mon[99959]: pgmap v9017: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:34 smithi078 ceph-mon[91136]: pgmap v9017: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:35.894 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:20:36.418 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 5h - - 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 14s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 14s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (4h) 14s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 14s ago 5h 500M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 10s ago 5h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 14s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 14s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 14s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:20:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 14s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 14s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 14s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 10s ago 5h 71.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 14s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 14s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:20:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:20:36.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:20:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:20:36.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:20:36.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:20:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:20:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:20:36.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:20:36.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:20:36.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:36.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:36 smithi078 ceph-mon[91136]: pgmap v9018: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:36.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:36.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:36 smithi078 ceph-mon[99959]: pgmap v9018: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:36 smithi146 ceph-mon[103278]: pgmap v9018: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:37.000 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:20:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:20:37.000 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:20:37.001 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:20:37.309 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:20:37.309 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:37.309 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:20:37.309 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:37.310 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:20:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:37 smithi146 ceph-mon[103278]: from='client.84410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:37 smithi146 ceph-mon[103278]: from='client.84416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:37 smithi146 ceph-mon[103278]: from='client.64503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/385772189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3224776802' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[99959]: from='client.84410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[99959]: from='client.84416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[99959]: from='client.64503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/385772189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3224776802' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[91136]: from='client.84410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[91136]: from='client.84416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[91136]: from='client.64503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/385772189' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:20:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3224776802' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:20:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:38 smithi146 ceph-mon[103278]: pgmap v9019: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:38 smithi146 ceph-mon[103278]: from='client.64515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:38 smithi078 ceph-mon[91136]: pgmap v9019: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:38 smithi078 ceph-mon[91136]: from='client.64515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:38 smithi078 ceph-mon[99959]: pgmap v9019: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:38 smithi078 ceph-mon[99959]: from='client.64515 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:20:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:40 smithi146 ceph-mon[103278]: pgmap v9020: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:40 smithi078 ceph-mon[91136]: pgmap v9020: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:40 smithi078 ceph-mon[99959]: pgmap v9020: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:20:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:20:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:41.402Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:41.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:41.403Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:42 smithi146 ceph-mon[103278]: pgmap v9021: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[91136]: pgmap v9021: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[99959]: pgmap v9021: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:42.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:42.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:44 smithi146 ceph-mon[103278]: pgmap v9022: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:44 smithi078 ceph-mon[91136]: pgmap v9022: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:44 smithi078 ceph-mon[99959]: pgmap v9022: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:46.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:46.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:46 smithi078 ceph-mon[99959]: pgmap v9023: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:46.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:46.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:46 smithi078 ceph-mon[91136]: pgmap v9023: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:46 smithi146 ceph-mon[103278]: pgmap v9023: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[91136]: pgmap v9024: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:47 smithi078 ceph-mon[99959]: pgmap v9024: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:20:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:47 smithi146 ceph-mon[103278]: pgmap v9024: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:50.282 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:50 smithi146 ceph-mon[103278]: pgmap v9025: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:50 smithi078 ceph-mon[99959]: pgmap v9025: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:50 smithi078 ceph-mon[91136]: pgmap v9025: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:20:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:20:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:20:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:51.403Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:51.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:20:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:20:51.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:20:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:52 smithi078 ceph-mon[99959]: pgmap v9026: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:52 smithi078 ceph-mon[91136]: pgmap v9026: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:52 smithi146 ceph-mon[103278]: pgmap v9026: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:20:54.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:54 smithi146 ceph-mon[103278]: pgmap v9027: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:20:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:54 smithi078 ceph-mon[99959]: pgmap v9027: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:20:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:54 smithi078 ceph-mon[91136]: pgmap v9027: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:20:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:56 smithi146 ceph-mon[103278]: pgmap v9028: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:20:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:56 smithi078 ceph-mon[99959]: pgmap v9028: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:20:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:20:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:56 smithi078 ceph-mon[91136]: pgmap v9028: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:20:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:58 smithi146 ceph-mon[103278]: pgmap v9029: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:20:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:58 smithi078 ceph-mon[91136]: pgmap v9029: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:20:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:58 smithi078 ceph-mon[99959]: pgmap v9029: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:20:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:20:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:00 smithi146 ceph-mon[103278]: pgmap v9030: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:21:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:00 smithi078 ceph-mon[91136]: pgmap v9030: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:21:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:00 smithi078 ceph-mon[99959]: pgmap v9030: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:21:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:21:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:21:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:01.404Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:01.404Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:01.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:02 smithi146 ceph-mon[103278]: pgmap v9031: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:21:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[99959]: pgmap v9031: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:21:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[91136]: pgmap v9031: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:21:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:04 smithi146 ceph-mon[103278]: pgmap v9032: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T01:21:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:04 smithi078 ceph-mon[91136]: pgmap v9032: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T01:21:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:04 smithi078 ceph-mon[99959]: pgmap v9032: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T01:21:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:06 smithi078 ceph-mon[91136]: pgmap v9033: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-21T01:21:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:06 smithi078 ceph-mon[99959]: pgmap v9033: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-21T01:21:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:06 smithi146 ceph-mon[103278]: pgmap v9033: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-21T01:21:07.597 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:21:07.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:07.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:07.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:07.893 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[99959]: pgmap v9034: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:21:07.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:07.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:07.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:07.893 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:07 smithi078 ceph-mon[91136]: pgmap v9034: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:21:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:07 smithi146 ceph-mon[103278]: pgmap v9034: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 5h - - 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 5h - - 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 6s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 1s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 6s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 1s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 6s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:21:08.153 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 6s ago 5h 500M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 1s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 6s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 6s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 1s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 5h 93.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 5h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 1s ago 5h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 1s ago 5h 82.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 1s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 1s ago 5h 85.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (4h) 1s ago 5h 71.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:21:08.154 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:21:08.155 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:21:08.155 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:21:08.466 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:08.467 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:21:08.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:21:08.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:21:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:21:08.739 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:21:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:21:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:21:08.739 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:21:08.740 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:21:08.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[99959]: from='client.64527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[99959]: from='client.64533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[99959]: from='client.64539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.741 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3025747380' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:08.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[91136]: from='client.64527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[91136]: from='client.64533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[91136]: from='client.64539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3025747380' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:08.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:08 smithi146 ceph-mon[103278]: from='client.64527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:08 smithi146 ceph-mon[103278]: from='client.64533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:08 smithi146 ceph-mon[103278]: from='client.64539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:08.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3025747380' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:09.047 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:21:09.047 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:09.048 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:09.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:09 smithi146 ceph-mon[103278]: from='client.64551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:09.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:09 smithi146 ceph-mon[103278]: pgmap v9035: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:21:09.953 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/977212703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:09 smithi078 ceph-mon[99959]: from='client.64551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:09 smithi078 ceph-mon[99959]: pgmap v9035: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:21:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/977212703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:09 smithi078 ceph-mon[91136]: from='client.64551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:09 smithi078 ceph-mon[91136]: pgmap v9035: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:21:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/977212703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:21:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:21:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:11.405Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:11.405Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:11.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:12 smithi146 ceph-mon[103278]: pgmap v9036: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:21:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:12 smithi078 ceph-mon[91136]: pgmap v9036: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:21:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:12 smithi078 ceph-mon[99959]: pgmap v9036: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:21:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:14 smithi146 ceph-mon[103278]: pgmap v9037: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T01:21:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:14 smithi078 ceph-mon[91136]: pgmap v9037: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T01:21:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:14 smithi078 ceph-mon[99959]: pgmap v9037: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T01:21:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:16 smithi146 ceph-mon[103278]: pgmap v9038: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T01:21:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:16 smithi078 ceph-mon[91136]: pgmap v9038: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T01:21:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:16 smithi078 ceph-mon[99959]: pgmap v9038: 161 pgs: 161 active+clean; 585 KiB data, 301 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T01:21:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:18 smithi146 ceph-mon[103278]: pgmap v9039: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:21:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:18 smithi078 ceph-mon[91136]: pgmap v9039: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:21:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:18 smithi078 ceph-mon[99959]: pgmap v9039: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:21:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:20 smithi146 ceph-mon[103278]: pgmap v9040: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T01:21:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:20 smithi078 ceph-mon[99959]: pgmap v9040: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T01:21:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:20 smithi078 ceph-mon[91136]: pgmap v9040: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T01:21:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:21:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:21:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:21.405Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:21.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:21.406Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:22.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:22 smithi078 ceph-mon[91136]: pgmap v9041: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:21:22.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:22 smithi078 ceph-mon[99959]: pgmap v9041: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:21:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:22 smithi146 ceph-mon[103278]: pgmap v9041: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T01:21:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:24.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:24 smithi146 ceph-mon[103278]: pgmap v9042: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T01:21:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:24 smithi078 ceph-mon[99959]: pgmap v9042: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T01:21:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:24 smithi078 ceph-mon[91136]: pgmap v9042: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T01:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:26 smithi078 ceph-mon[99959]: pgmap v9043: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:26 smithi078 ceph-mon[91136]: pgmap v9043: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:21:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:26 smithi146 ceph-mon[103278]: pgmap v9043: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[99959]: pgmap v9044: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:27 smithi078 ceph-mon[91136]: pgmap v9044: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:21:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:27 smithi146 ceph-mon[103278]: pgmap v9044: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:21:28.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:28.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:30.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:29 smithi146 ceph-mon[103278]: pgmap v9045: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:21:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:29 smithi078 ceph-mon[91136]: pgmap v9045: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:21:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:29 smithi078 ceph-mon[99959]: pgmap v9045: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:21:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:21:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:21:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:31.406Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:31.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:31.407Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:32 smithi146 ceph-mon[103278]: pgmap v9046: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-21T01:21:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:32 smithi078 ceph-mon[99959]: pgmap v9046: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-21T01:21:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:32 smithi078 ceph-mon[91136]: pgmap v9046: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-09-21T01:21:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:34 smithi146 ceph-mon[103278]: pgmap v9047: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T01:21:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:34 smithi078 ceph-mon[99959]: pgmap v9047: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T01:21:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:34 smithi078 ceph-mon[91136]: pgmap v9047: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T01:21:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:36 smithi146 ceph-mon[103278]: pgmap v9048: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:21:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:36 smithi078 ceph-mon[99959]: pgmap v9048: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:21:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:36 smithi078 ceph-mon[91136]: pgmap v9048: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:21:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:38 smithi146 ceph-mon[103278]: pgmap v9049: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:38 smithi078 ceph-mon[91136]: pgmap v9049: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:38 smithi078 ceph-mon[99959]: pgmap v9049: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:39.342 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:21:39.869 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 5h - - 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 5h - - 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 17s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 13s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (4h) 17s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 13s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 17s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 17s ago 5h 500M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 13s ago 5h 333M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 17s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:21:39.870 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 17s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 13s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 17s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 17s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 17s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 17s ago 5h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 5h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 5h 82.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 5h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 5h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 13s ago 5h 71.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 17s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 17s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:21:39.871 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:21:40.181 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:21:40.182 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:21:40.183 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:21:40.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:21:40.183 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:21:40.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:40 smithi078 ceph-mon[99959]: pgmap v9050: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:40.183 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:40 smithi078 ceph-mon[99959]: from='client.64563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:40.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:40 smithi078 ceph-mon[91136]: pgmap v9050: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:40.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:40 smithi078 ceph-mon[91136]: from='client.64563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:21:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:21:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:21:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:21:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:21:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:21:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:21:40.453 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:21:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:40 smithi146 ceph-mon[103278]: pgmap v9050: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:40 smithi146 ceph-mon[103278]: from='client.64563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:40.765 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:21:40.765 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:40.765 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:21:40.765 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:40.765 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:21:40.766 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:21:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:21:41.407 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[99959]: from='client.84488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[99959]: from='client.64575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4029856965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[99959]: from='client.84506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3960525655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[91136]: from='client.84488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[91136]: from='client.64575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4029856965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[91136]: from='client.84506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3960525655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:41 smithi146 ceph-mon[103278]: from='client.84488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:41 smithi146 ceph-mon[103278]: from='client.64575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4029856965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:21:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:41 smithi146 ceph-mon[103278]: from='client.84506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3960525655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:21:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:41.407Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:41.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:41.408Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:42 smithi146 ceph-mon[103278]: pgmap v9051: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:42 smithi078 ceph-mon[91136]: pgmap v9051: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:42 smithi078 ceph-mon[99959]: pgmap v9051: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:43.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:44.657 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:44 smithi078 ceph-mon[99959]: pgmap v9052: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:44.657 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:44.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:44 smithi078 ceph-mon[91136]: pgmap v9052: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:44.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:44 smithi146 ceph-mon[103278]: pgmap v9052: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:46 smithi146 ceph-mon[103278]: pgmap v9053: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:46 smithi078 ceph-mon[91136]: pgmap v9053: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:46 smithi078 ceph-mon[99959]: pgmap v9053: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:47 smithi146 ceph-mon[103278]: pgmap v9054: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[91136]: pgmap v9054: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:21:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:47 smithi078 ceph-mon[99959]: pgmap v9054: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:50 smithi146 ceph-mon[103278]: pgmap v9055: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:50 smithi078 ceph-mon[91136]: pgmap v9055: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:50 smithi078 ceph-mon[99959]: pgmap v9055: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:21:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:21:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:51.408Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:51.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:21:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:21:51.409Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:21:52.453 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:52 smithi146 ceph-mon[103278]: pgmap v9056: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:52 smithi078 ceph-mon[91136]: pgmap v9056: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:52 smithi078 ceph-mon[99959]: pgmap v9056: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:54 smithi146 ceph-mon[103278]: pgmap v9057: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:54 smithi078 ceph-mon[91136]: pgmap v9057: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:54 smithi078 ceph-mon[99959]: pgmap v9057: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:21:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:56 smithi146 ceph-mon[103278]: pgmap v9058: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:56 smithi078 ceph-mon[91136]: pgmap v9058: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:21:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:56 smithi078 ceph-mon[99959]: pgmap v9058: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:58 smithi146 ceph-mon[103278]: pgmap v9059: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:58 smithi078 ceph-mon[91136]: pgmap v9059: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:58 smithi078 ceph-mon[99959]: pgmap v9059: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:21:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:21:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:21:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:21:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:21:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:00 smithi146 ceph-mon[103278]: pgmap v9060: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:00 smithi078 ceph-mon[99959]: pgmap v9060: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:00 smithi078 ceph-mon[91136]: pgmap v9060: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:22:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:22:01.718 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:01.409Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:01.718 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:01.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:01.718 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:01.410Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:02.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:02 smithi078 ceph-mon[99959]: pgmap v9061: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:02.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:02 smithi078 ceph-mon[91136]: pgmap v9061: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:02 smithi146 ceph-mon[103278]: pgmap v9061: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:04.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:04 smithi078 ceph-mon[99959]: pgmap v9062: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:04.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:04 smithi078 ceph-mon[91136]: pgmap v9062: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:04 smithi146 ceph-mon[103278]: pgmap v9062: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:05 smithi146 ceph-mon[103278]: pgmap v9063: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:05 smithi078 ceph-mon[91136]: pgmap v9063: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:05 smithi078 ceph-mon[99959]: pgmap v9063: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[99959]: pgmap v9064: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[91136]: pgmap v9064: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.104 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:08 smithi146 ceph-mon[103278]: pgmap v9064: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:10 smithi146 ceph-mon[103278]: pgmap v9065: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:10 smithi078 ceph-mon[91136]: pgmap v9065: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:10 smithi078 ceph-mon[99959]: pgmap v9065: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:22:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:22:11.077 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 5h - - 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 5h - - 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 9s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 4s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 9s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 4s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 9s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 9s ago 5h 501M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 4s ago 5h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 9s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 9s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:22:11.619 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 4s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 9s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 9s ago 5h 94.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 9s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 9s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 4s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 4s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 4s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 4s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 4s ago 5h 71.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 9s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 9s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:22:11.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:22:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:11.410Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:11.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:11.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:22:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:22:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:22:11.936 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:22:12.223 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:22:12.224 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:22:12.224 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:22:12.224 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:22:12.224 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:22:12.224 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:22:12.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[91136]: pgmap v9066: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:12.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[91136]: from='client.84518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:12.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[91136]: from='client.84524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:12.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/431291806' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:12.224 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[99959]: pgmap v9066: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:12.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[99959]: from='client.84518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:12.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[99959]: from='client.84524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:12.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/431291806' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:12 smithi146 ceph-mon[103278]: pgmap v9066: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:12 smithi146 ceph-mon[103278]: from='client.84518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:12 smithi146 ceph-mon[103278]: from='client.84524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/431291806' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:12.532 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:22:12.532 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:12.532 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:12.532 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:13 smithi146 ceph-mon[103278]: from='client.64611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:13 smithi146 ceph-mon[103278]: from='client.84542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2462638847' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:13 smithi078 ceph-mon[91136]: from='client.64611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:13 smithi078 ceph-mon[91136]: from='client.84542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2462638847' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:13 smithi078 ceph-mon[99959]: from='client.64611 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:13 smithi078 ceph-mon[99959]: from='client.84542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2462638847' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:14 smithi146 ceph-mon[103278]: pgmap v9067: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:14 smithi078 ceph-mon[91136]: pgmap v9067: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:14 smithi078 ceph-mon[99959]: pgmap v9067: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:16 smithi146 ceph-mon[103278]: pgmap v9068: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:16 smithi078 ceph-mon[99959]: pgmap v9068: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:16 smithi078 ceph-mon[91136]: pgmap v9068: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:18 smithi146 ceph-mon[103278]: pgmap v9069: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:18 smithi078 ceph-mon[99959]: pgmap v9069: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:18 smithi078 ceph-mon[91136]: pgmap v9069: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:20 smithi146 ceph-mon[103278]: pgmap v9070: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:20 smithi078 ceph-mon[99959]: pgmap v9070: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:20 smithi078 ceph-mon[91136]: pgmap v9070: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:22:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:22:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:21.411Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:21.411Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:21.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:22 smithi146 ceph-mon[103278]: pgmap v9071: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:22 smithi078 ceph-mon[91136]: pgmap v9071: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:22 smithi078 ceph-mon[99959]: pgmap v9071: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:23 smithi146 ceph-mon[103278]: pgmap v9072: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:24.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[91136]: pgmap v9072: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:23 smithi078 ceph-mon[99959]: pgmap v9072: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:26 smithi146 ceph-mon[103278]: pgmap v9073: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:26 smithi078 ceph-mon[91136]: pgmap v9073: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:26 smithi078 ceph-mon[99959]: pgmap v9073: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:28 smithi146 ceph-mon[103278]: pgmap v9074: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[91136]: pgmap v9074: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[99959]: pgmap v9074: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:30 smithi146 ceph-mon[103278]: pgmap v9075: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:30 smithi078 ceph-mon[91136]: pgmap v9075: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:30 smithi078 ceph-mon[99959]: pgmap v9075: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:31.010 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:22:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:22:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:31.411Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:31.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:31.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:31.412Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:32.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:32 smithi146 ceph-mon[103278]: pgmap v9076: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:32 smithi078 ceph-mon[91136]: pgmap v9076: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:32 smithi078 ceph-mon[99959]: pgmap v9076: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:34 smithi146 ceph-mon[103278]: pgmap v9077: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:34 smithi078 ceph-mon[91136]: pgmap v9077: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:34 smithi078 ceph-mon[99959]: pgmap v9077: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:36 smithi146 ceph-mon[103278]: pgmap v9078: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:36 smithi078 ceph-mon[91136]: pgmap v9078: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:36 smithi078 ceph-mon[99959]: pgmap v9078: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:38 smithi146 ceph-mon[103278]: pgmap v9079: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:38 smithi078 ceph-mon[99959]: pgmap v9079: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:38 smithi078 ceph-mon[91136]: pgmap v9079: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:40 smithi146 ceph-mon[103278]: pgmap v9080: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:40 smithi078 ceph-mon[99959]: pgmap v9080: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:40 smithi078 ceph-mon[91136]: pgmap v9080: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:22:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:22:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:41.412Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:41.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:41.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:42 smithi146 ceph-mon[103278]: pgmap v9081: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:42 smithi078 ceph-mon[99959]: pgmap v9081: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:42 smithi078 ceph-mon[91136]: pgmap v9081: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:42.838 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 5h - - 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 5h - - 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 0s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 16s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 0s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (4h) 16s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:22:43.363 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 0s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (4h) 0s ago 5h 501M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 16s ago 5h 334M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 0s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 0s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 16s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 0s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 0s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 0s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 0s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 16s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 16s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 16s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 16s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 5h 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 0s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 0s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:22:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:22:43.677 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:22:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:22:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:22:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:22:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:22:43.947 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='client.84554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: pgmap v9082: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='client.64641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='client.64647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:43.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='client.84554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: pgmap v9082: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='client.64641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='client.64647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:43.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='client.84554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: pgmap v9082: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='client.64641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='client.64647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:44.257 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:22:44.257 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:22:44.257 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:22:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3641225867' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:44 smithi146 ceph-mon[103278]: from='client.84578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3721480170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3641225867' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:44 smithi078 ceph-mon[91136]: from='client.84578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3721480170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3641225867' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:44 smithi078 ceph-mon[99959]: from='client.84578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3721480170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:22:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:45 smithi146 ceph-mon[103278]: pgmap v9083: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:45 smithi078 ceph-mon[91136]: pgmap v9083: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:45 smithi078 ceph-mon[99959]: pgmap v9083: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:48 smithi146 ceph-mon[103278]: pgmap v9084: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[91136]: pgmap v9084: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[99959]: pgmap v9084: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:22:50.439 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:50 smithi078 ceph-mon[99959]: pgmap v9085: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:50.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:50 smithi078 ceph-mon[91136]: pgmap v9085: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:50 smithi146 ceph-mon[103278]: pgmap v9085: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:22:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:22:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:51.412Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:51.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:22:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:22:51.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:22:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:52 smithi146 ceph-mon[103278]: pgmap v9086: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:52 smithi078 ceph-mon[91136]: pgmap v9086: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:52 smithi078 ceph-mon[99959]: pgmap v9086: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:54.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:54 smithi078 ceph-mon[99959]: pgmap v9087: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:54.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:54 smithi078 ceph-mon[91136]: pgmap v9087: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:54 smithi146 ceph-mon[103278]: pgmap v9087: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:22:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:56 smithi146 ceph-mon[103278]: pgmap v9088: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:56 smithi078 ceph-mon[91136]: pgmap v9088: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:22:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:56 smithi078 ceph-mon[99959]: pgmap v9088: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:58 smithi146 ceph-mon[103278]: pgmap v9089: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:58 smithi078 ceph-mon[91136]: pgmap v9089: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:58 smithi078 ceph-mon[99959]: pgmap v9089: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:22:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:22:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:22:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:22:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:00 smithi078 ceph-mon[99959]: pgmap v9090: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:00 smithi078 ceph-mon[91136]: pgmap v9090: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:00 smithi146 ceph-mon[103278]: pgmap v9090: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:23:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:23:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:01.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:01.413Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:01.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:02 smithi146 ceph-mon[103278]: pgmap v9091: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:02 smithi078 ceph-mon[99959]: pgmap v9091: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:02 smithi078 ceph-mon[91136]: pgmap v9091: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:03 smithi146 ceph-mon[103278]: pgmap v9092: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[99959]: pgmap v9092: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:03 smithi078 ceph-mon[91136]: pgmap v9092: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:06 smithi146 ceph-mon[103278]: pgmap v9093: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:06 smithi078 ceph-mon[99959]: pgmap v9093: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:06 smithi078 ceph-mon[91136]: pgmap v9093: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:08 smithi146 ceph-mon[103278]: pgmap v9094: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[99959]: pgmap v9094: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[91136]: pgmap v9094: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:10 smithi146 ceph-mon[103278]: pgmap v9095: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:10 smithi078 ceph-mon[91136]: pgmap v9095: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:23:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:23:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:10 smithi078 ceph-mon[99959]: pgmap v9095: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:11.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:11.413Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:11.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:11.414Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:11.477 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:11.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:12.726 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:12 smithi146 ceph-mon[103278]: pgmap v9096: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:12.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:12 smithi078 ceph-mon[91136]: pgmap v9096: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:12.746 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:12 smithi078 ceph-mon[99959]: pgmap v9096: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:14 smithi146 ceph-mon[103278]: pgmap v9097: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:14.553 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:23:14.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:14 smithi078 ceph-mon[99959]: pgmap v9097: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:14.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:14.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:14 smithi078 ceph-mon[91136]: pgmap v9097: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:14.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 5h - - 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 5h - - 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 12s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 7s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 12s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 7s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 12s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 12s ago 5h 502M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 7s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 12s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 12s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:23:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 7s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 12s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 12s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 12s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 12s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 7s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 7s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 7s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 7s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 7s ago 5h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 12s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 12s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:23:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:23:15.394 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:23:15.395 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:23:15.666 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:23:15.666 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:23:15.666 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:23:15.667 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:23:15.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3424420525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:15.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3424420525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3424420525' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:15.976 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:23:15.976 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:15.976 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:15.976 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:15.976 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:16 smithi146 ceph-mon[103278]: from='client.64671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:16 smithi146 ceph-mon[103278]: from='client.84596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:16 smithi146 ceph-mon[103278]: pgmap v9098: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:16 smithi146 ceph-mon[103278]: from='client.84602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3150668408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[91136]: from='client.64671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[91136]: from='client.84596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[91136]: pgmap v9098: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[91136]: from='client.84602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3150668408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[99959]: from='client.64671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[99959]: from='client.84596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[99959]: pgmap v9098: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[99959]: from='client.84602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3150668408' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:17 smithi146 ceph-mon[103278]: from='client.84614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:17.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:17 smithi078 ceph-mon[91136]: from='client.84614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:17.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:17 smithi078 ceph-mon[99959]: from='client.84614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:18 smithi146 ceph-mon[103278]: pgmap v9099: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:18 smithi078 ceph-mon[99959]: pgmap v9099: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:18 smithi078 ceph-mon[91136]: pgmap v9099: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:20 smithi146 ceph-mon[103278]: pgmap v9100: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:20 smithi078 ceph-mon[99959]: pgmap v9100: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:20 smithi078 ceph-mon[91136]: pgmap v9100: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:23:20] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T01:23:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:21.415Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:21.415Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:21.416Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:22 smithi146 ceph-mon[103278]: pgmap v9101: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:22 smithi078 ceph-mon[99959]: pgmap v9101: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:22 smithi078 ceph-mon[91136]: pgmap v9101: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:23 smithi146 ceph-mon[103278]: pgmap v9102: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[91136]: pgmap v9102: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:23 smithi078 ceph-mon[99959]: pgmap v9102: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:26 smithi146 ceph-mon[103278]: pgmap v9103: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:26 smithi078 ceph-mon[91136]: pgmap v9103: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:26 smithi078 ceph-mon[99959]: pgmap v9103: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:28 smithi146 ceph-mon[103278]: pgmap v9104: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[91136]: pgmap v9104: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[99959]: pgmap v9104: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:30 smithi146 ceph-mon[103278]: pgmap v9105: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:30 smithi078 ceph-mon[91136]: pgmap v9105: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:30 smithi078 ceph-mon[99959]: pgmap v9105: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:30.875 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:23:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:23:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:31.416Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:31.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:31.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:31.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:32.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:32 smithi078 ceph-mon[99959]: pgmap v9106: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:32.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:32 smithi078 ceph-mon[91136]: pgmap v9106: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:32 smithi146 ceph-mon[103278]: pgmap v9106: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:34.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:34 smithi146 ceph-mon[103278]: pgmap v9107: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:34 smithi078 ceph-mon[91136]: pgmap v9107: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:34 smithi078 ceph-mon[99959]: pgmap v9107: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:36 smithi146 ceph-mon[103278]: pgmap v9108: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:36 smithi078 ceph-mon[91136]: pgmap v9108: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:36 smithi078 ceph-mon[99959]: pgmap v9108: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:38 smithi146 ceph-mon[103278]: pgmap v9109: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:38 smithi078 ceph-mon[91136]: pgmap v9109: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:38 smithi078 ceph-mon[99959]: pgmap v9109: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:40 smithi146 ceph-mon[103278]: pgmap v9110: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:40 smithi078 ceph-mon[91136]: pgmap v9110: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:40 smithi078 ceph-mon[99959]: pgmap v9110: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:23:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:23:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:41.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:41.417Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:41.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:42 smithi146 ceph-mon[103278]: pgmap v9111: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:42 smithi078 ceph-mon[91136]: pgmap v9111: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:42 smithi078 ceph-mon[99959]: pgmap v9111: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:44 smithi146 ceph-mon[103278]: pgmap v9112: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[99959]: pgmap v9112: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[91136]: pgmap v9112: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:46.271 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:23:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:46 smithi146 ceph-mon[103278]: pgmap v9113: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:46 smithi078 ceph-mon[99959]: pgmap v9113: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:46 smithi078 ceph-mon[91136]: pgmap v9113: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 5h - - 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 19s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 3s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 19s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:23:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 3s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 3s ago 5h 503M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (4h) 19s ago 5h 335M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (4h) 3s ago 5h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 19s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 3s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 3s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 3s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 3s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:23:46.795 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:23:46.796 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 19s ago 5h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:23:46.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 3s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:23:46.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:23:46.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 3s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:23:46.796 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:23:47.106 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:23:47.106 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:23:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:23:47.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:23:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:23:47.377 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:23:47.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:47 smithi078 ceph-mon[99959]: from='client.64707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:47.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:47 smithi078 ceph-mon[91136]: from='client.64707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:47 smithi146 ceph-mon[103278]: from='client.64707 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:47.685 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:23:47.685 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:47.685 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:23:47.685 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:47.685 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:23:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='client.84632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='client.84638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: pgmap v9114: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3304178845' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='client.84650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2188655493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='client.84632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='client.84638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: pgmap v9114: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3304178845' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='client.84650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2188655493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='client.84632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='client.84638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: pgmap v9114: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3304178845' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='client.84650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2188655493' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:23:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:23:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:50 smithi146 ceph-mon[103278]: pgmap v9115: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:50 smithi078 ceph-mon[91136]: pgmap v9115: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:50 smithi078 ceph-mon[99959]: pgmap v9115: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:23:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:23:51.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:51.417Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:51.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:51.418Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:23:51.746 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:23:51.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:23:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:52 smithi146 ceph-mon[103278]: pgmap v9116: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:52 smithi078 ceph-mon[91136]: pgmap v9116: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:52 smithi078 ceph-mon[99959]: pgmap v9116: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:54.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:54 smithi078 ceph-mon[91136]: pgmap v9117: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:54.208 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:54 smithi078 ceph-mon[99959]: pgmap v9117: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:54 smithi146 ceph-mon[103278]: pgmap v9117: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:23:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:56 smithi146 ceph-mon[103278]: pgmap v9118: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:56 smithi078 ceph-mon[91136]: pgmap v9118: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:23:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:56 smithi078 ceph-mon[99959]: pgmap v9118: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:58.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:58 smithi146 ceph-mon[103278]: pgmap v9119: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:58 smithi078 ceph-mon[99959]: pgmap v9119: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:58 smithi078 ceph-mon[91136]: pgmap v9119: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:23:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:23:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:00 smithi146 ceph-mon[103278]: pgmap v9120: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:00 smithi078 ceph-mon[99959]: pgmap v9120: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:00 smithi078 ceph-mon[91136]: pgmap v9120: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:24:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:24:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:01.419Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:01.419Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:01.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:02.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:02 smithi078 ceph-mon[99959]: pgmap v9121: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:02.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:02 smithi078 ceph-mon[91136]: pgmap v9121: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:02 smithi146 ceph-mon[103278]: pgmap v9121: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:04 smithi146 ceph-mon[103278]: pgmap v9122: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[91136]: pgmap v9122: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[99959]: pgmap v9122: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:06 smithi146 ceph-mon[103278]: pgmap v9123: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:06 smithi078 ceph-mon[91136]: pgmap v9123: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:06 smithi078 ceph-mon[99959]: pgmap v9123: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:08 smithi146 ceph-mon[103278]: pgmap v9124: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[91136]: pgmap v9124: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[99959]: pgmap v9124: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:10 smithi146 ceph-mon[103278]: pgmap v9125: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:10 smithi078 ceph-mon[91136]: pgmap v9125: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:10.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:10 smithi078 ceph-mon[99959]: pgmap v9125: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:24:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:24:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:11.420Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:11.420Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:11.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:12.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:12 smithi078 ceph-mon[91136]: pgmap v9126: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:12.449 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:12 smithi078 ceph-mon[99959]: pgmap v9126: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:12 smithi146 ceph-mon[103278]: pgmap v9126: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:14 smithi146 ceph-mon[103278]: pgmap v9127: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:14.370 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:14 smithi078 ceph-mon[99959]: pgmap v9127: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:14.370 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:14.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:14 smithi078 ceph-mon[91136]: pgmap v9127: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:14.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:16 smithi146 ceph-mon[103278]: pgmap v9128: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:16 smithi078 ceph-mon[91136]: pgmap v9128: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:16 smithi078 ceph-mon[99959]: pgmap v9128: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:17.978 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:24:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:18 smithi078 ceph-mon[91136]: pgmap v9129: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:18 smithi078 ceph-mon[99959]: pgmap v9129: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 5h - - 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 15s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 15s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:24:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 15s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 15s ago 5h 503M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 10s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 15s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 15s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 15s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 15s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 15s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 15s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 10s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 10s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 10s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:24:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 10s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:24:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 10s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:24:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 15s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:24:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:24:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 15s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:24:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:24:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:18 smithi146 ceph-mon[103278]: pgmap v9129: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:24:18.817 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:24:19.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:24:19.086 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:24:19.398 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:24:19.399 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:19.399 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:19.399 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:19.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[99959]: from='client.64743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[99959]: from='client.84668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[99959]: from='client.64755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/968487391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:19.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[91136]: from='client.64743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[91136]: from='client.84668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[91136]: from='client.64755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/968487391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:19 smithi146 ceph-mon[103278]: from='client.64743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:19 smithi146 ceph-mon[103278]: from='client.84668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:19 smithi146 ceph-mon[103278]: from='client.64755 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/968487391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:20 smithi146 ceph-mon[103278]: pgmap v9130: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:20 smithi146 ceph-mon[103278]: from='client.84686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/641600749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-mon[91136]: pgmap v9130: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-mon[91136]: from='client.84686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/641600749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-mon[99959]: pgmap v9130: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-mon[99959]: from='client.84686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/641600749' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:24:20] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:24:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:21.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:21.421Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:21.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:22 smithi146 ceph-mon[103278]: pgmap v9131: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:22 smithi078 ceph-mon[91136]: pgmap v9131: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:22 smithi078 ceph-mon[99959]: pgmap v9131: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:24 smithi146 ceph-mon[103278]: pgmap v9132: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[91136]: pgmap v9132: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[99959]: pgmap v9132: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 01:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T01:24:25.847314945Z level=info msg="Completed cleanup jobs" duration=12.017127ms 2024-09-21T01:24:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:26 smithi146 ceph-mon[103278]: pgmap v9133: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:26 smithi078 ceph-mon[91136]: pgmap v9133: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:26 smithi078 ceph-mon[99959]: pgmap v9133: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:28 smithi146 ceph-mon[103278]: pgmap v9134: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[91136]: pgmap v9134: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[99959]: pgmap v9134: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:30.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:30 smithi078 ceph-mon[99959]: pgmap v9135: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:30 smithi078 ceph-mon[91136]: pgmap v9135: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:30 smithi146 ceph-mon[103278]: pgmap v9135: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:24:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:24:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:31.421Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:31.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:31.422Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:32 smithi078 ceph-mon[91136]: pgmap v9136: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:32 smithi078 ceph-mon[99959]: pgmap v9136: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:32 smithi146 ceph-mon[103278]: pgmap v9136: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:34.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:34 smithi078 ceph-mon[91136]: pgmap v9137: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:34.521 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:34 smithi078 ceph-mon[99959]: pgmap v9137: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:34.653 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:34 smithi146 ceph-mon[103278]: pgmap v9137: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:36.596 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:36.597 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:36 smithi146 ceph-mon[103278]: pgmap v9138: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:36 smithi078 ceph-mon[99959]: pgmap v9138: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:36 smithi078 ceph-mon[91136]: pgmap v9138: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:38 smithi146 ceph-mon[103278]: pgmap v9139: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:38 smithi078 ceph-mon[91136]: pgmap v9139: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:38 smithi078 ceph-mon[99959]: pgmap v9139: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:40.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:40 smithi078 ceph-mon[99959]: pgmap v9140: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:40 smithi078 ceph-mon[91136]: pgmap v9140: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:40 smithi146 ceph-mon[103278]: pgmap v9140: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:24:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:24:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:41.422Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:41.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:41.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:42 smithi146 ceph-mon[103278]: pgmap v9141: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:42 smithi078 ceph-mon[91136]: pgmap v9141: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:42 smithi078 ceph-mon[99959]: pgmap v9141: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:44 smithi146 ceph-mon[103278]: pgmap v9142: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[91136]: pgmap v9142: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[99959]: pgmap v9142: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:46 smithi146 ceph-mon[103278]: pgmap v9143: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:46 smithi078 ceph-mon[99959]: pgmap v9143: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:46 smithi078 ceph-mon[91136]: pgmap v9143: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:48 smithi146 ceph-mon[103278]: pgmap v9144: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[99959]: pgmap v9144: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[91136]: pgmap v9144: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:24:49.698 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:24:50.224 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:24:50.224 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 5h - - 2024-09-21T01:24:50.224 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 5h - - 2024-09-21T01:24:50.224 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 6s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:24:50.224 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 2s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 6s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 6s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 6s ago 5h 504M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 6s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 6s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 2s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (4h) 6s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 6s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 6s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 6s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 2s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:24:50.225 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 2s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 2s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 2s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 6s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 6s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:24:50.226 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:24:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:50 smithi078 ceph-mon[99959]: pgmap v9145: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:50 smithi078 ceph-mon[91136]: pgmap v9145: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:50.535 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:24:50.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:24:50.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:24:50.536 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:24:50.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:24:50.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:24:50.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:24:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:50 smithi146 ceph-mon[103278]: pgmap v9145: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:50.803 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:24:50.804 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:24:50.805 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:24:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:24:51.117 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:24:51.117 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:24:51.117 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:24:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:51 smithi146 ceph-mon[103278]: from='client.84698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:51 smithi146 ceph-mon[103278]: from='client.64785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:51 smithi146 ceph-mon[103278]: from='client.64791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3201524557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3066623914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[91136]: from='client.84698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[91136]: from='client.64785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[91136]: from='client.64791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3201524557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3066623914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[99959]: from='client.84698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[99959]: from='client.64785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[99959]: from='client.64791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3201524557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3066623914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:24:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:51.423Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:51.423Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:24:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:24:51.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:24:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:52 smithi146 ceph-mon[103278]: from='client.84722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:52 smithi146 ceph-mon[103278]: pgmap v9146: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:52 smithi078 ceph-mon[91136]: from='client.84722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:52 smithi078 ceph-mon[91136]: pgmap v9146: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:52 smithi078 ceph-mon[99959]: from='client.84722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:52 smithi078 ceph-mon[99959]: pgmap v9146: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:54.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:54 smithi078 ceph-mon[91136]: pgmap v9147: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:54.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:54 smithi078 ceph-mon[99959]: pgmap v9147: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:54 smithi146 ceph-mon[103278]: pgmap v9147: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:24:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:56 smithi146 ceph-mon[103278]: pgmap v9148: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:56 smithi078 ceph-mon[91136]: pgmap v9148: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:24:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:56 smithi078 ceph-mon[99959]: pgmap v9148: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:58 smithi146 ceph-mon[103278]: pgmap v9149: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:58 smithi078 ceph-mon[99959]: pgmap v9149: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:58 smithi078 ceph-mon[91136]: pgmap v9149: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:24:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:24:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:24:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:24:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:00 smithi146 ceph-mon[103278]: pgmap v9150: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:00 smithi078 ceph-mon[99959]: pgmap v9150: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:00 smithi078 ceph-mon[91136]: pgmap v9150: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:25:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:25:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:01.424Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:01.424Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:01.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:02 smithi146 ceph-mon[103278]: pgmap v9151: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:02 smithi078 ceph-mon[99959]: pgmap v9151: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:02 smithi078 ceph-mon[91136]: pgmap v9151: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:04 smithi146 ceph-mon[103278]: pgmap v9152: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[99959]: pgmap v9152: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[91136]: pgmap v9152: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:06 smithi146 ceph-mon[103278]: pgmap v9153: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:06 smithi078 ceph-mon[91136]: pgmap v9153: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:06 smithi078 ceph-mon[99959]: pgmap v9153: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:08 smithi146 ceph-mon[103278]: pgmap v9154: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[99959]: pgmap v9154: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[91136]: pgmap v9154: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:10 smithi146 ceph-mon[103278]: pgmap v9155: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:10 smithi078 ceph-mon[99959]: pgmap v9155: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:10 smithi078 ceph-mon[91136]: pgmap v9155: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:25:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:25:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:11.424Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:11.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:11.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:12 smithi146 ceph-mon[103278]: pgmap v9156: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:12 smithi078 ceph-mon[99959]: pgmap v9156: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:12 smithi078 ceph-mon[91136]: pgmap v9156: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:13.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:14.739 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:14 smithi078 ceph-mon[99959]: pgmap v9157: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:14.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:14 smithi078 ceph-mon[91136]: pgmap v9157: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:14 smithi146 ceph-mon[103278]: pgmap v9157: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:16.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:16.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:16 smithi146 ceph-mon[103278]: pgmap v9158: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:16.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:16.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:16 smithi078 ceph-mon[99959]: pgmap v9158: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:16.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:16.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:16 smithi078 ceph-mon[91136]: pgmap v9158: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:18 smithi146 ceph-mon[103278]: pgmap v9159: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:18 smithi078 ceph-mon[99959]: pgmap v9159: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:18 smithi078 ceph-mon[91136]: pgmap v9159: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:20 smithi146 ceph-mon[103278]: pgmap v9160: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:20 smithi078 ceph-mon[99959]: pgmap v9160: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:20 smithi078 ceph-mon[91136]: pgmap v9160: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:25:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:25:21.413 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:25:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:21.424Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:21.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:21.686 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:21.425Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 5h - - 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 5h - - 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 18s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 13s ago 5h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 18s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 13s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 18s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:25:21.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 18s ago 5h 504M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 13s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 18s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 18s ago 5h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 13s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 18s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 18s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 18s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 18s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 13s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 13s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 13s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 13s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 13s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 18s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:25:21.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:25:21.941 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 18s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:25:21.941 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:25:22.245 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:25:22.245 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:25:22.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:25:22.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:22.245 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:25:22.246 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:25:22.513 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:25:22.513 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:25:22.513 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:25:22.514 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:25:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:22 smithi146 ceph-mon[103278]: pgmap v9161: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:25:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:22 smithi146 ceph-mon[103278]: from='client.84734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/286024095' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:22 smithi078 ceph-mon[91136]: pgmap v9161: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:25:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:22 smithi078 ceph-mon[91136]: from='client.84734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/286024095' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:22 smithi078 ceph-mon[99959]: pgmap v9161: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:25:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:22 smithi078 ceph-mon[99959]: from='client.84734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/286024095' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:25:22.827 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:22.827 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:22.828 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:22.828 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:23 smithi146 ceph-mon[103278]: from='client.64821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:23 smithi146 ceph-mon[103278]: from='client.64827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:23 smithi146 ceph-mon[103278]: from='client.64839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/396493655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[99959]: from='client.64821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[99959]: from='client.64827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[99959]: from='client.64839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/396493655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[91136]: from='client.64821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[91136]: from='client.64827 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[91136]: from='client.64839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/396493655' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:24 smithi146 ceph-mon[103278]: pgmap v9162: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[91136]: pgmap v9162: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[99959]: pgmap v9162: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:26 smithi146 ceph-mon[103278]: pgmap v9163: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:26 smithi078 ceph-mon[91136]: pgmap v9163: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:26 smithi078 ceph-mon[99959]: pgmap v9163: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:28 smithi146 ceph-mon[103278]: pgmap v9164: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[91136]: pgmap v9164: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[99959]: pgmap v9164: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:30 smithi078 ceph-mon[91136]: pgmap v9165: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:25:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:25:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:30 smithi078 ceph-mon[99959]: pgmap v9165: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:30 smithi146 ceph-mon[103278]: pgmap v9165: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:31.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:31 smithi078 ceph-mon[99959]: pgmap v9166: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:31 smithi078 ceph-mon[91136]: pgmap v9166: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:31.425Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:31.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:31.426Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:31 smithi146 ceph-mon[103278]: pgmap v9166: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:34 smithi078 ceph-mon[99959]: pgmap v9167: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:34 smithi078 ceph-mon[91136]: pgmap v9167: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:34 smithi146 ceph-mon[103278]: pgmap v9167: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:36.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:36.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:36 smithi078 ceph-mon[91136]: pgmap v9168: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:36.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:36.417 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:36 smithi078 ceph-mon[99959]: pgmap v9168: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:36.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:36.449 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:36 smithi146 ceph-mon[103278]: pgmap v9168: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:38 smithi146 ceph-mon[103278]: pgmap v9169: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:38 smithi078 ceph-mon[91136]: pgmap v9169: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:38 smithi078 ceph-mon[99959]: pgmap v9169: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:40 smithi146 ceph-mon[103278]: pgmap v9170: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:40 smithi078 ceph-mon[91136]: pgmap v9170: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:40 smithi078 ceph-mon[99959]: pgmap v9170: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:25:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:25:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:41.426Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:41.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:41.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:42 smithi146 ceph-mon[103278]: pgmap v9171: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:42 smithi078 ceph-mon[99959]: pgmap v9171: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:42 smithi078 ceph-mon[91136]: pgmap v9171: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:44 smithi146 ceph-mon[103278]: pgmap v9172: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[99959]: pgmap v9172: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[91136]: pgmap v9172: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:45 smithi146 ceph-mon[103278]: pgmap v9173: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:45 smithi078 ceph-mon[99959]: pgmap v9173: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:45 smithi078 ceph-mon[91136]: pgmap v9173: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:48 smithi146 ceph-mon[103278]: pgmap v9174: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:48 smithi078 ceph-mon[91136]: pgmap v9174: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:48 smithi078 ceph-mon[99959]: pgmap v9174: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:25:50.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:50 smithi078 ceph-mon[99959]: pgmap v9175: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:50 smithi078 ceph-mon[91136]: pgmap v9175: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:25:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:25:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:50 smithi146 ceph-mon[103278]: pgmap v9175: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:51.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:51.427Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:51.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:51.427Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:25:51.685 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:25:51.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:25:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:52 smithi078 ceph-mon[91136]: pgmap v9176: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:52 smithi078 ceph-mon[99959]: pgmap v9176: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:52 smithi146 ceph-mon[103278]: pgmap v9176: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:53.129 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:25:53.734 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:25:53.734 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 5h - - 2024-09-21T01:25:53.734 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 5h - - 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 9s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 5s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 9s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 5s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 9s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 9s ago 5h 506M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 5s ago 5h 339M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 9s ago 5h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 9s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 5s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 9s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (4h) 9s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 9s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:25:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 9s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 5s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 5s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 5s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 9s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 9s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:25:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:25:54.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:25:54.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:25:54.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:25:54.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:25:54.062 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:25:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:25:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:25:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:25:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:25:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:25:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:25:54.329 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:25:54.639 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:25:54.639 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:25:54.639 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[99959]: pgmap v9177: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[99959]: from='client.64851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[99959]: from='client.84776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2867500576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[91136]: pgmap v9177: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[91136]: from='client.64851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[91136]: from='client.84776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:54.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2867500576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:54.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:54 smithi146 ceph-mon[103278]: pgmap v9177: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:25:54.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:54 smithi146 ceph-mon[103278]: from='client.64851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:54.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:54 smithi146 ceph-mon[103278]: from='client.84776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:54.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2867500576' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:25:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:55 smithi146 ceph-mon[103278]: from='client.64863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:55 smithi146 ceph-mon[103278]: from='client.64875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3782157439' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:55 smithi078 ceph-mon[91136]: from='client.64863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:55 smithi078 ceph-mon[91136]: from='client.64875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3782157439' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:55 smithi078 ceph-mon[99959]: from='client.64863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:55 smithi078 ceph-mon[99959]: from='client.64875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3782157439' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:25:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:56 smithi078 ceph-mon[91136]: pgmap v9178: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:56 smithi078 ceph-mon[99959]: pgmap v9178: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:56.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:25:56.816 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:56 smithi146 ceph-mon[103278]: pgmap v9178: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:57.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:57 smithi078 ceph-mon[91136]: pgmap v9179: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:57.628 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:57 smithi078 ceph-mon[99959]: pgmap v9179: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:57 smithi146 ceph-mon[103278]: pgmap v9179: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:25:58.634 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:58.858 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:58.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:25:59 smithi146 ceph-mon[103278]: pgmap v9180: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:25:59 smithi078 ceph-mon[91136]: pgmap v9180: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:25:59 smithi078 ceph-mon[99959]: pgmap v9180: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:26:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:26:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:01.428Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:01.428Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:01.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:02 smithi146 ceph-mon[103278]: pgmap v9181: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:02 smithi078 ceph-mon[99959]: pgmap v9181: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:02 smithi078 ceph-mon[91136]: pgmap v9181: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[99959]: pgmap v9182: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[91136]: pgmap v9182: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:04 smithi146 ceph-mon[103278]: pgmap v9182: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:06 smithi146 ceph-mon[103278]: pgmap v9183: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:06 smithi078 ceph-mon[99959]: pgmap v9183: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:06.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:06 smithi078 ceph-mon[91136]: pgmap v9183: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:08 smithi146 ceph-mon[103278]: pgmap v9184: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:08 smithi078 ceph-mon[91136]: pgmap v9184: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:08 smithi078 ceph-mon[99959]: pgmap v9184: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:09 smithi146 ceph-mon[103278]: pgmap v9185: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[99959]: pgmap v9185: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:09 smithi078 ceph-mon[91136]: pgmap v9185: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:26:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:26:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:11.428Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:11.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:11.429Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:12 smithi146 ceph-mon[103278]: pgmap v9186: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:12 smithi078 ceph-mon[91136]: pgmap v9186: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:12 smithi078 ceph-mon[99959]: pgmap v9186: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:14 smithi146 ceph-mon[103278]: pgmap v9187: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:14 smithi078 ceph-mon[91136]: pgmap v9187: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:14 smithi078 ceph-mon[99959]: pgmap v9187: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:16 smithi146 ceph-mon[103278]: pgmap v9188: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:16 smithi078 ceph-mon[91136]: pgmap v9188: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:16 smithi078 ceph-mon[99959]: pgmap v9188: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:18.429 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:18 smithi078 ceph-mon[99959]: pgmap v9189: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:18.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:18 smithi078 ceph-mon[91136]: pgmap v9189: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:18.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:18 smithi146 ceph-mon[103278]: pgmap v9189: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:20 smithi078 ceph-mon[91136]: pgmap v9190: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:20 smithi078 ceph-mon[99959]: pgmap v9190: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:20.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:20 smithi146 ceph-mon[103278]: pgmap v9190: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:26:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:26:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:21.429Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:21.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:21.430Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:22 smithi146 ceph-mon[103278]: pgmap v9191: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:22 smithi078 ceph-mon[99959]: pgmap v9191: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:22 smithi078 ceph-mon[91136]: pgmap v9191: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:24 smithi146 ceph-mon[103278]: pgmap v9192: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[99959]: pgmap v9192: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[91136]: pgmap v9192: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:24.928 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:26:25.457 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:26:25.457 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 5h - - 2024-09-21T01:26:25.457 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 5h - - 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 1s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 16s ago 5h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 1s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 16s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 1s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 1s ago 5h 505M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 16s ago 5h 336M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 1s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 1s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 16s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 1s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 1s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (4h) 1s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:26:25.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 1s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 16s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 16s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 16s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 16s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 1s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 1s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:26:25.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:26:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:26:25.770 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:26:26.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:26:26.035 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:26:26.035 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:26:26.035 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:26:26.036 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:26:26.345 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:26:26.345 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:26.345 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:26.345 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:26.345 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:26.345 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:26.345 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[99959]: from='client.84806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[99959]: pgmap v9193: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[99959]: from='client.64893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[99959]: from='client.84818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/801209748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[91136]: from='client.84806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[91136]: pgmap v9193: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[91136]: from='client.64893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[91136]: from='client.84818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.346 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/801209748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:26 smithi146 ceph-mon[103278]: from='client.84806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:26 smithi146 ceph-mon[103278]: pgmap v9193: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:26 smithi146 ceph-mon[103278]: from='client.64893 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:26 smithi146 ceph-mon[103278]: from='client.84818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/801209748' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:27 smithi146 ceph-mon[103278]: from='client.64911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4189549045' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:27.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:27 smithi078 ceph-mon[99959]: from='client.64911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:27.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4189549045' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:27 smithi078 ceph-mon[91136]: from='client.64911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4189549045' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:28 smithi146 ceph-mon[103278]: pgmap v9194: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:28 smithi078 ceph-mon[91136]: pgmap v9194: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:28 smithi078 ceph-mon[99959]: pgmap v9194: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:30 smithi146 ceph-mon[103278]: pgmap v9195: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:30 smithi078 ceph-mon[91136]: pgmap v9195: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:30 smithi078 ceph-mon[99959]: pgmap v9195: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:26:30] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:26:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:31.430Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:31.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:31.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:32 smithi146 ceph-mon[103278]: pgmap v9196: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:32 smithi078 ceph-mon[99959]: pgmap v9196: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:32 smithi078 ceph-mon[91136]: pgmap v9196: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:34 smithi146 ceph-mon[103278]: pgmap v9197: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:34 smithi078 ceph-mon[99959]: pgmap v9197: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:34 smithi078 ceph-mon[91136]: pgmap v9197: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:36 smithi146 ceph-mon[103278]: pgmap v9198: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:36 smithi078 ceph-mon[91136]: pgmap v9198: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:36 smithi078 ceph-mon[99959]: pgmap v9198: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:38.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:38 smithi146 ceph-mon[103278]: pgmap v9199: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:38 smithi078 ceph-mon[91136]: pgmap v9199: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:38 smithi078 ceph-mon[99959]: pgmap v9199: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:40.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:40 smithi146 ceph-mon[103278]: pgmap v9200: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:40 smithi078 ceph-mon[91136]: pgmap v9200: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:40 smithi078 ceph-mon[99959]: pgmap v9200: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:26:40] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:26:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:41.431Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:41.431Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:41.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:42 smithi146 ceph-mon[103278]: pgmap v9201: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:42 smithi078 ceph-mon[91136]: pgmap v9201: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:42 smithi078 ceph-mon[99959]: pgmap v9201: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:44 smithi146 ceph-mon[103278]: pgmap v9202: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:44 smithi078 ceph-mon[91136]: pgmap v9202: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:44 smithi078 ceph-mon[99959]: pgmap v9202: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:46 smithi146 ceph-mon[103278]: pgmap v9203: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:46 smithi078 ceph-mon[91136]: pgmap v9203: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:46 smithi078 ceph-mon[99959]: pgmap v9203: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:48 smithi146 ceph-mon[103278]: pgmap v9204: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:48 smithi078 ceph-mon[99959]: pgmap v9204: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:48 smithi078 ceph-mon[91136]: pgmap v9204: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:49 smithi146 ceph-mon[103278]: pgmap v9205: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[91136]: pgmap v9205: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:26:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:49 smithi078 ceph-mon[99959]: pgmap v9205: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:26:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:26:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:51.431Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:51.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:26:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:26:51.432Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:26:52.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:52 smithi078 ceph-mon[99959]: pgmap v9206: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:52.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:52 smithi078 ceph-mon[91136]: pgmap v9206: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:52 smithi146 ceph-mon[103278]: pgmap v9206: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:54 smithi146 ceph-mon[103278]: pgmap v9207: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:54 smithi078 ceph-mon[91136]: pgmap v9207: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:54 smithi078 ceph-mon[99959]: pgmap v9207: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:26:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:55 smithi146 ceph-mon[103278]: pgmap v9208: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:55 smithi078 ceph-mon[99959]: pgmap v9208: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:26:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:55 smithi078 ceph-mon[91136]: pgmap v9208: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:56.642 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:26:57.166 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 5h - - 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 5h - - 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 12s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 8s ago 5h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 12s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 8s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 12s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 12s ago 5h 505M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 8s ago 5h 338M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 12s ago 5h 347M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 12s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 8s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:26:57.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 12s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 12s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 12s ago 5h 78.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4h) 12s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 8s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 8s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 8s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 8s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 8s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 12s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 12s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:26:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:26:57.473 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:26:57.473 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:26:57.473 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:26:57.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:26:57.475 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:26:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:26:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:26:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:26:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:26:57.744 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:26:58.053 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:26:58.053 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:58.053 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:26:58.053 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:58.053 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:58 smithi146 ceph-mon[103278]: from='client.84842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:58 smithi146 ceph-mon[103278]: from='client.64929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:58 smithi146 ceph-mon[103278]: pgmap v9209: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:58 smithi146 ceph-mon[103278]: from='client.84854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/304105837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2629810924' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[99959]: from='client.84842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[99959]: from='client.64929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[99959]: pgmap v9209: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[99959]: from='client.84854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/304105837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2629810924' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[91136]: from='client.84842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[91136]: from='client.64929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.517 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[91136]: pgmap v9209: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:26:58.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[91136]: from='client.84854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:58.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/304105837' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:26:58.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2629810924' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:26:59.487 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:59 smithi146 ceph-mon[103278]: from='client.84866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:59.487 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:26:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:59 smithi078 ceph-mon[91136]: from='client.84866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:26:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:59 smithi078 ceph-mon[99959]: from='client.84866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:26:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:00.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:00 smithi146 ceph-mon[103278]: pgmap v9210: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:00.424 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:00 smithi078 ceph-mon[91136]: pgmap v9210: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:00.424 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:00 smithi078 ceph-mon[99959]: pgmap v9210: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:27:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:27:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:01.432Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:01.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:01.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:02 smithi146 ceph-mon[103278]: pgmap v9211: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:02 smithi078 ceph-mon[99959]: pgmap v9211: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:02 smithi078 ceph-mon[91136]: pgmap v9211: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:04 smithi146 ceph-mon[103278]: pgmap v9212: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:04 smithi078 ceph-mon[99959]: pgmap v9212: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:04 smithi078 ceph-mon[91136]: pgmap v9212: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:06 smithi146 ceph-mon[103278]: pgmap v9213: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:06 smithi078 ceph-mon[91136]: pgmap v9213: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:06 smithi078 ceph-mon[99959]: pgmap v9213: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:08 smithi146 ceph-mon[103278]: pgmap v9214: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:08 smithi078 ceph-mon[91136]: pgmap v9214: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:08 smithi078 ceph-mon[99959]: pgmap v9214: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[91136]: pgmap v9215: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-mon[99959]: pgmap v9215: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:10.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:10.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:10 smithi146 ceph-mon[103278]: pgmap v9215: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:27:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:27:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:11.433Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:11.935 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:11.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:11.935 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:11.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:12 smithi146 ceph-mon[103278]: pgmap v9216: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:12 smithi078 ceph-mon[91136]: pgmap v9216: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:12 smithi078 ceph-mon[99959]: pgmap v9216: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:14 smithi146 ceph-mon[103278]: pgmap v9217: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:14 smithi078 ceph-mon[91136]: pgmap v9217: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:14 smithi078 ceph-mon[99959]: pgmap v9217: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:16 smithi146 ceph-mon[103278]: pgmap v9218: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:16 smithi078 ceph-mon[91136]: pgmap v9218: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:16 smithi078 ceph-mon[99959]: pgmap v9218: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:18 smithi146 ceph-mon[103278]: pgmap v9219: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:18 smithi078 ceph-mon[99959]: pgmap v9219: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:18 smithi078 ceph-mon[91136]: pgmap v9219: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:20.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:20 smithi146 ceph-mon[103278]: pgmap v9220: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:20.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:20 smithi078 ceph-mon[91136]: pgmap v9220: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:20.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:20 smithi078 ceph-mon[99959]: pgmap v9220: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:27:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:27:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:21.433Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:21.433Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:21.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:22 smithi146 ceph-mon[103278]: pgmap v9221: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:22 smithi078 ceph-mon[91136]: pgmap v9221: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:22 smithi078 ceph-mon[99959]: pgmap v9221: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:24.503 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:24 smithi078 ceph-mon[99959]: pgmap v9222: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:24.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:24 smithi078 ceph-mon[91136]: pgmap v9222: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:24 smithi146 ceph-mon[103278]: pgmap v9222: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:26 smithi146 ceph-mon[103278]: pgmap v9223: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:26 smithi078 ceph-mon[91136]: pgmap v9223: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:26 smithi078 ceph-mon[99959]: pgmap v9223: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:28.347 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:27:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:28 smithi146 ceph-mon[103278]: pgmap v9224: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:28 smithi078 ceph-mon[99959]: pgmap v9224: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:28 smithi078 ceph-mon[91136]: pgmap v9224: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:28.878 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 5h - - 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 5h - - 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 4s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 19s ago 5h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 4s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 19s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 4s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 4s ago 5h 507M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 19s ago 5h 337M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 4s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 4s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 19s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:27:28.879 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 4s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 4s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 4s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 4s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (4h) 19s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 19s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 19s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 19s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 19s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 4s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 4s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:27:28.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:27:29.188 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:27:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:27:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:27:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:27:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:27:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:27:29.189 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:27:29.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:27:29.456 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:27:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:29 smithi146 ceph-mon[103278]: from='client.64959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/94163826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:29.772 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:27:29.772 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:27:29.773 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:27:29.773 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:27:29.773 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[99959]: from='client.64959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/94163826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[91136]: from='client.64959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:29.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.774 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.774 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:29.774 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/94163826' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:27:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:30 smithi146 ceph-mon[103278]: from='client.64965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:30 smithi146 ceph-mon[103278]: from='client.84890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:30 smithi146 ceph-mon[103278]: pgmap v9225: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:30 smithi146 ceph-mon[103278]: from='client.64983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/107658058' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:27:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[99959]: from='client.64965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[99959]: from='client.84890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[99959]: pgmap v9225: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[99959]: from='client.64983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/107658058' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[91136]: from='client.64965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[91136]: from='client.84890 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[91136]: pgmap v9225: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[91136]: from='client.64983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/107658058' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:27:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:31.434Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:31.434Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:31.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:32 smithi146 ceph-mon[103278]: pgmap v9226: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:32 smithi078 ceph-mon[99959]: pgmap v9226: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:32 smithi078 ceph-mon[91136]: pgmap v9226: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:34 smithi146 ceph-mon[103278]: pgmap v9227: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:34 smithi078 ceph-mon[99959]: pgmap v9227: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:34 smithi078 ceph-mon[91136]: pgmap v9227: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:36 smithi078 ceph-mon[91136]: pgmap v9228: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:36 smithi078 ceph-mon[99959]: pgmap v9228: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:37.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:36 smithi146 ceph-mon[103278]: pgmap v9228: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:38 smithi078 ceph-mon[99959]: pgmap v9229: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:38 smithi078 ceph-mon[91136]: pgmap v9229: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:38 smithi146 ceph-mon[103278]: pgmap v9229: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:40 smithi078 ceph-mon[99959]: pgmap v9230: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:40 smithi078 ceph-mon[91136]: pgmap v9230: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:40.698 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:40 smithi146 ceph-mon[103278]: pgmap v9230: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:27:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:27:41.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:41.435Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:41.719 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:41.435Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:41.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:41.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:42 smithi078 ceph-mon[91136]: pgmap v9231: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:42 smithi078 ceph-mon[99959]: pgmap v9231: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:42 smithi146 ceph-mon[103278]: pgmap v9231: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:43.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:44 smithi078 ceph-mon[91136]: pgmap v9232: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:44 smithi078 ceph-mon[99959]: pgmap v9232: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:44 smithi146 ceph-mon[103278]: pgmap v9232: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:45 smithi146 ceph-mon[103278]: pgmap v9233: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[91136]: pgmap v9233: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:45 smithi078 ceph-mon[99959]: pgmap v9233: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:48 smithi146 ceph-mon[103278]: pgmap v9234: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:48 smithi078 ceph-mon[91136]: pgmap v9234: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:48 smithi078 ceph-mon[99959]: pgmap v9234: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[91136]: pgmap v9235: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[99959]: pgmap v9235: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:50 smithi146 ceph-mon[103278]: pgmap v9235: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:27:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:27:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:27:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:51.435Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:51.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:27:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:27:51.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:27:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:52 smithi078 ceph-mon[91136]: pgmap v9236: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:52 smithi078 ceph-mon[99959]: pgmap v9236: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:52.686 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:52 smithi146 ceph-mon[103278]: pgmap v9236: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:27:54.662 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:54 smithi146 ceph-mon[103278]: pgmap v9237: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:54 smithi078 ceph-mon[91136]: pgmap v9237: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:54 smithi078 ceph-mon[99959]: pgmap v9237: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:27:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:56 smithi146 ceph-mon[103278]: pgmap v9238: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:56 smithi078 ceph-mon[91136]: pgmap v9238: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:27:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:56 smithi078 ceph-mon[99959]: pgmap v9238: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:58 smithi146 ceph-mon[103278]: pgmap v9239: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:58 smithi078 ceph-mon[99959]: pgmap v9239: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:58 smithi078 ceph-mon[91136]: pgmap v9239: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:27:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:27:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:27:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:27:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:00.107 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:28:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:00 smithi078 ceph-mon[99959]: pgmap v9240: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:00 smithi078 ceph-mon[91136]: pgmap v9240: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:00.635 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:28:00.635 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T01:28:00.635 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 5h - - 2024-09-21T01:28:00.635 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 16s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:28:00.635 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 11s ago 5h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 11s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 5h 507M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 11s ago 5h 340M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 16s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 11s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 11s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 11s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:28:00.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 11s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:28:00.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 11s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:28:00.637 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 11s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:28:00.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 16s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:28:00.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:28:00.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 16s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:28:00.637 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:28:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:00 smithi146 ceph-mon[103278]: pgmap v9240: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:00.874 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:28:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:28:00.960 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:28:00.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:28:00.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:28:00.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:00.960 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:28:00.960 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:28:00.961 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:28:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:28:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:28:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:28:01.237 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:28:01.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-mon[91136]: from='client.84914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:01.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-mon[91136]: from='client.84920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:01.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3510209657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:01.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-mon[99959]: from='client.84914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:01.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-mon[99959]: from='client.84920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:01.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3510209657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:01.551 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:28:01.551 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:01.551 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:01 smithi146 ceph-mon[103278]: from='client.84914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:01 smithi146 ceph-mon[103278]: from='client.84920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3510209657' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:01.435Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:01.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:01.436Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:02 smithi146 ceph-mon[103278]: from='client.84926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:02 smithi146 ceph-mon[103278]: pgmap v9241: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:02 smithi146 ceph-mon[103278]: from='client.65019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3856186097' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[99959]: from='client.84926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[99959]: pgmap v9241: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[99959]: from='client.65019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3856186097' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[91136]: from='client.84926 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[91136]: pgmap v9241: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[91136]: from='client.65019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:02.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3856186097' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:04 smithi146 ceph-mon[103278]: pgmap v9242: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:04.766 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:04 smithi078 ceph-mon[99959]: pgmap v9242: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:04.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:04 smithi078 ceph-mon[91136]: pgmap v9242: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:05 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:05 smithi146 ceph-mon[103278]: pgmap v9243: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[99959]: pgmap v9243: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:05 smithi078 ceph-mon[91136]: pgmap v9243: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:08 smithi146 ceph-mon[103278]: pgmap v9244: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:08 smithi078 ceph-mon[99959]: pgmap v9244: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:08 smithi078 ceph-mon[91136]: pgmap v9244: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:10 smithi146 ceph-mon[103278]: pgmap v9245: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:28:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:28:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[91136]: pgmap v9245: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[99959]: pgmap v9245: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:11.436Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:11.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:11.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:12 smithi146 ceph-mon[103278]: pgmap v9246: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:12 smithi078 ceph-mon[91136]: pgmap v9246: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:12 smithi078 ceph-mon[99959]: pgmap v9246: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:14 smithi146 ceph-mon[103278]: pgmap v9247: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:14 smithi078 ceph-mon[91136]: pgmap v9247: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:14 smithi078 ceph-mon[99959]: pgmap v9247: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:16 smithi146 ceph-mon[103278]: pgmap v9248: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:16 smithi078 ceph-mon[91136]: pgmap v9248: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:16 smithi078 ceph-mon[99959]: pgmap v9248: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:18 smithi146 ceph-mon[103278]: pgmap v9249: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:18 smithi078 ceph-mon[91136]: pgmap v9249: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:18 smithi078 ceph-mon[99959]: pgmap v9249: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:20 smithi078 ceph-mon[91136]: pgmap v9250: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:20 smithi078 ceph-mon[99959]: pgmap v9250: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:20 smithi146 ceph-mon[103278]: pgmap v9250: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:28:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:28:21.778 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:21.437Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:21.778 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:21.437Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:21.779 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:21.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:22 smithi078 ceph-mon[91136]: pgmap v9251: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:22 smithi078 ceph-mon[99959]: pgmap v9251: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:22.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:22 smithi146 ceph-mon[103278]: pgmap v9251: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:24 smithi078 ceph-mon[91136]: pgmap v9252: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:24 smithi078 ceph-mon[99959]: pgmap v9252: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:24 smithi146 ceph-mon[103278]: pgmap v9252: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:25 smithi146 ceph-mon[103278]: pgmap v9253: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[99959]: pgmap v9253: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:25 smithi078 ceph-mon[91136]: pgmap v9253: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:28 smithi146 ceph-mon[103278]: pgmap v9254: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:28 smithi078 ceph-mon[91136]: pgmap v9254: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:28 smithi078 ceph-mon[99959]: pgmap v9254: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:30 smithi146 ceph-mon[103278]: pgmap v9255: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:28:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:28:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[99959]: pgmap v9255: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[91136]: pgmap v9255: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:31.437Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:31.438Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:31.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:31.849 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:28:32.373 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:28:32.373 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 5h - - 2024-09-21T01:28:32.373 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 5h - - 2024-09-21T01:28:32.373 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 7s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:28:32.373 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 2s ago 5h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:28:32.373 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 7s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 7s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 7s ago 5h 509M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 5h 341M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 7s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 7s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 2s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 7s ago 5h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 7s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 7s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 7s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 2s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (4h) 2s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 2s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:28:32.374 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 2s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:28:32.375 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:28:32.375 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 7s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:28:32.375 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:28:32.375 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 7s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:28:32.375 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:32.682 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:28:32.683 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:28:32.683 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:32 smithi078 ceph-mon[99959]: pgmap v9256: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:32.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:32 smithi078 ceph-mon[91136]: pgmap v9256: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:32.949 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:28:32.950 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:28:32.951 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:28:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:32 smithi146 ceph-mon[103278]: pgmap v9256: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:28:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:33.257 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:28:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:33.257 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:28:33.864 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:33 smithi146 ceph-mon[103278]: from='client.84950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:33.864 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:33 smithi146 ceph-mon[103278]: from='client.84956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:33.864 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:33 smithi146 ceph-mon[103278]: from='client.84962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:33.864 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3628003145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:33.864 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1190969914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[91136]: from='client.84950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[91136]: from='client.84956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[91136]: from='client.84962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3628003145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1190969914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[99959]: from='client.84950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[99959]: from='client.84956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[99959]: from='client.84962 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3628003145' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:28:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1190969914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:28:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:34 smithi146 ceph-mon[103278]: from='client.84974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:34 smithi146 ceph-mon[103278]: pgmap v9257: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:35.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:34 smithi078 ceph-mon[99959]: from='client.84974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:35.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:34 smithi078 ceph-mon[99959]: pgmap v9257: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:34 smithi078 ceph-mon[91136]: from='client.84974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:34 smithi078 ceph-mon[91136]: pgmap v9257: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:35 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:35 smithi146 ceph-mon[103278]: pgmap v9258: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:35 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:35 smithi078 ceph-mon[91136]: pgmap v9258: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:35 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:35 smithi078 ceph-mon[99959]: pgmap v9258: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:38 smithi146 ceph-mon[103278]: pgmap v9259: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:38 smithi078 ceph-mon[91136]: pgmap v9259: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:38 smithi078 ceph-mon[99959]: pgmap v9259: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:40 smithi146 ceph-mon[103278]: pgmap v9260: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:40 smithi078 ceph-mon[91136]: pgmap v9260: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:40 smithi078 ceph-mon[99959]: pgmap v9260: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:28:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:28:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:41.439Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:41.439Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:41.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:42 smithi146 ceph-mon[103278]: pgmap v9261: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:42 smithi078 ceph-mon[91136]: pgmap v9261: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:42 smithi078 ceph-mon[99959]: pgmap v9261: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:44.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:44 smithi078 ceph-mon[99959]: pgmap v9262: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:44.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:44.388 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:44 smithi078 ceph-mon[91136]: pgmap v9262: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:44.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:44.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:44 smithi146 ceph-mon[103278]: pgmap v9262: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:44.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:45 smithi146 ceph-mon[103278]: pgmap v9263: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[99959]: pgmap v9263: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:45 smithi078 ceph-mon[91136]: pgmap v9263: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:48 smithi146 ceph-mon[103278]: pgmap v9264: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:48 smithi078 ceph-mon[91136]: pgmap v9264: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:48 smithi078 ceph-mon[99959]: pgmap v9264: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:50 smithi146 ceph-mon[103278]: pgmap v9265: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[99959]: pgmap v9265: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[91136]: pgmap v9265: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:28:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:28:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:28:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:51.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:51.440Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:28:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:28:51.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:28:52.488 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:52 smithi078 ceph-mon[99959]: pgmap v9266: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:52.488 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:52 smithi078 ceph-mon[91136]: pgmap v9266: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:52 smithi146 ceph-mon[103278]: pgmap v9266: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:54 smithi146 ceph-mon[103278]: pgmap v9267: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:54 smithi078 ceph-mon[91136]: pgmap v9267: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:54 smithi078 ceph-mon[99959]: pgmap v9267: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:28:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:56 smithi146 ceph-mon[103278]: pgmap v9268: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:56 smithi078 ceph-mon[91136]: pgmap v9268: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:28:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:56 smithi078 ceph-mon[99959]: pgmap v9268: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:58 smithi146 ceph-mon[103278]: pgmap v9269: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:58 smithi078 ceph-mon[91136]: pgmap v9269: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:58 smithi078 ceph-mon[99959]: pgmap v9269: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:28:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:28:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:28:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:28:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:00 smithi146 ceph-mon[103278]: pgmap v9270: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:00 smithi078 ceph-mon[91136]: pgmap v9270: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:00 smithi078 ceph-mon[99959]: pgmap v9270: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:29:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:29:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:01.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:01.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:01.441Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:02 smithi146 ceph-mon[103278]: pgmap v9271: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:02 smithi078 ceph-mon[91136]: pgmap v9271: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:02 smithi078 ceph-mon[99959]: pgmap v9271: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:03.554 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:29:04.109 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:29:04.109 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T01:29:04.109 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 5h - - 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 14s ago 5h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 14s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 19s ago 5h 582M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 5h 509M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 14s ago 5h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:29:04.110 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 14s ago 5h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 5h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 14s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 14s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 14s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 14s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 14s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 19s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 19s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:29:04.111 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:29:04.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:04 smithi146 ceph-mon[103278]: pgmap v9272: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:04.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:04 smithi146 ceph-mon[103278]: from='client.84986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:29:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:29:04.432 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:29:04.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:04 smithi078 ceph-mon[99959]: pgmap v9272: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:04.432 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:04 smithi078 ceph-mon[99959]: from='client.84986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:04.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:04 smithi078 ceph-mon[91136]: pgmap v9272: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:04.433 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:04 smithi078 ceph-mon[91136]: from='client.84986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:29:04.759 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:29:04.760 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:29:04.760 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:29:04.760 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:29:04.760 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:29:04.760 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:29:04.760 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:29:05.075 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:29:05.076 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:29:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:29:05.076 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:29:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:29:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='client.84992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='client.84998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1091696768' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4279576551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='client.84992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='client.84998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1091696768' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4279576551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='client.84992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='client.84998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1091696768' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4279576551' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:05.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:06 smithi146 ceph-mon[103278]: from='client.65091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:06 smithi146 ceph-mon[103278]: pgmap v9273: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:06 smithi078 ceph-mon[99959]: from='client.65091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:06 smithi078 ceph-mon[99959]: pgmap v9273: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:06 smithi078 ceph-mon[91136]: from='client.65091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:06 smithi078 ceph-mon[91136]: pgmap v9273: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:08 smithi146 ceph-mon[103278]: pgmap v9274: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:08 smithi078 ceph-mon[91136]: pgmap v9274: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:08 smithi078 ceph-mon[99959]: pgmap v9274: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:10 smithi146 ceph-mon[103278]: pgmap v9275: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[91136]: pgmap v9275: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[99959]: pgmap v9275: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:29:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:29:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:11.441Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:11.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:11.442Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:12 smithi146 ceph-mon[103278]: pgmap v9276: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:12 smithi078 ceph-mon[91136]: pgmap v9276: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:12 smithi078 ceph-mon[99959]: pgmap v9276: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:14 smithi146 ceph-mon[103278]: pgmap v9277: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:14 smithi078 ceph-mon[91136]: pgmap v9277: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:14 smithi078 ceph-mon[99959]: pgmap v9277: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:16 smithi146 ceph-mon[103278]: pgmap v9278: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:16 smithi078 ceph-mon[91136]: pgmap v9278: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:16.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:16 smithi078 ceph-mon[99959]: pgmap v9278: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:18 smithi146 ceph-mon[103278]: pgmap v9279: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:18 smithi078 ceph-mon[91136]: pgmap v9279: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:18 smithi078 ceph-mon[99959]: pgmap v9279: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:20 smithi146 ceph-mon[103278]: pgmap v9280: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:20 smithi078 ceph-mon[99959]: pgmap v9280: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:20 smithi078 ceph-mon[91136]: pgmap v9280: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:20.950 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:29:20] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:29:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:21.442Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:21.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:21.443Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:22 smithi146 ceph-mon[103278]: pgmap v9281: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:22 smithi078 ceph-mon[91136]: pgmap v9281: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:22 smithi078 ceph-mon[99959]: pgmap v9281: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:24.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:24 smithi146 ceph-mon[103278]: pgmap v9282: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:24 smithi078 ceph-mon[99959]: pgmap v9282: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:24 smithi078 ceph-mon[91136]: pgmap v9282: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:26 smithi146 ceph-mon[103278]: pgmap v9283: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[91136]: pgmap v9283: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[99959]: pgmap v9283: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:28 smithi146 ceph-mon[103278]: pgmap v9284: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:28 smithi078 ceph-mon[91136]: pgmap v9284: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:28 smithi078 ceph-mon[99959]: pgmap v9284: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[91136]: pgmap v9285: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[99959]: pgmap v9285: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:30 smithi146 ceph-mon[103278]: pgmap v9285: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:29:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:29:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:31.443Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:31.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:31.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:32 smithi078 ceph-mon[91136]: pgmap v9286: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:32.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:32 smithi078 ceph-mon[99959]: pgmap v9286: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:32 smithi146 ceph-mon[103278]: pgmap v9286: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:34 smithi078 ceph-mon[91136]: pgmap v9287: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:34 smithi078 ceph-mon[99959]: pgmap v9287: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:34 smithi146 ceph-mon[103278]: pgmap v9287: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:35.373 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 5h - - 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 5h - - 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 10s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 5s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 10s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 5s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 10s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 10s ago 5h 510M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 5s ago 5h 343M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 10s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 10s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:29:35.901 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 5s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 10s ago 5h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 10s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 10s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 10s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 5s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 5s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (4h) 5s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 5s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 5s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 10s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 10s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:29:35.902 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:29:36.213 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:29:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:29:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:29:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:29:36.213 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:29:36.214 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:29:36.482 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:29:36.483 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:29:36.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:36.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[99959]: pgmap v9288: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:36.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[99959]: from='client.65103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:36.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3471389700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:36.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:36.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[91136]: pgmap v9288: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:36.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[91136]: from='client.65103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:36.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3471389700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:36.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:36.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:36 smithi146 ceph-mon[103278]: pgmap v9288: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:36.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:36 smithi146 ceph-mon[103278]: from='client.65103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:36.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3471389700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:29:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:29:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:29:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:29:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:29:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:29:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:37 smithi146 ceph-mon[103278]: from='client.65109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:37 smithi146 ceph-mon[103278]: from='client.85034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:37 smithi146 ceph-mon[103278]: from='client.65127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3296656222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[91136]: from='client.65109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[91136]: from='client.85034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[91136]: from='client.65127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3296656222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[99959]: from='client.65109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[99959]: from='client.85034 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[99959]: from='client.65127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:29:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3296656222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:29:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:38 smithi078 ceph-mon[91136]: pgmap v9289: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:38 smithi078 ceph-mon[99959]: pgmap v9289: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:38 smithi146 ceph-mon[103278]: pgmap v9289: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:40 smithi078 ceph-mon[91136]: pgmap v9290: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:40 smithi078 ceph-mon[99959]: pgmap v9290: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:40 smithi146 ceph-mon[103278]: pgmap v9290: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:29:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:29:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:41.444Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:41.444Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:41.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:41.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:42 smithi146 ceph-mon[103278]: pgmap v9291: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:42 smithi078 ceph-mon[91136]: pgmap v9291: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:42 smithi078 ceph-mon[99959]: pgmap v9291: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:44.629 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:44 smithi078 ceph-mon[99959]: pgmap v9292: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:44.629 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:44 smithi078 ceph-mon[91136]: pgmap v9292: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:44 smithi146 ceph-mon[103278]: pgmap v9292: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:46 smithi146 ceph-mon[103278]: pgmap v9293: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[91136]: pgmap v9293: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[99959]: pgmap v9293: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:48 smithi146 ceph-mon[103278]: pgmap v9294: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:48 smithi078 ceph-mon[91136]: pgmap v9294: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:48 smithi078 ceph-mon[99959]: pgmap v9294: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:50 smithi146 ceph-mon[103278]: pgmap v9295: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:29:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:29:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[91136]: pgmap v9295: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[99959]: pgmap v9295: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:29:51.696 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:51.444Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:51.696 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:51.445Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:29:51.696 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:29:51.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:29:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:52 smithi146 ceph-mon[103278]: pgmap v9296: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:52 smithi078 ceph-mon[99959]: pgmap v9296: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:52 smithi078 ceph-mon[91136]: pgmap v9296: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:54 smithi146 ceph-mon[103278]: pgmap v9297: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:54 smithi078 ceph-mon[99959]: pgmap v9297: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:54 smithi078 ceph-mon[91136]: pgmap v9297: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:29:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:56 smithi146 ceph-mon[103278]: pgmap v9298: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:56 smithi078 ceph-mon[91136]: pgmap v9298: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:29:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:56 smithi078 ceph-mon[99959]: pgmap v9298: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:58 smithi146 ceph-mon[103278]: pgmap v9299: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:58 smithi078 ceph-mon[91136]: pgmap v9299: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:58 smithi078 ceph-mon[99959]: pgmap v9299: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:29:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:29:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:29:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:29:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T01:29:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:00 smithi146 ceph-mon[103278]: pgmap v9300: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:00 smithi078 ceph-mon[91136]: pgmap v9300: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:00 smithi078 ceph-mon[99959]: pgmap v9300: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:30:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:30:01.728 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:01.445Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:01.728 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:01.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:01.728 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:01.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:02 smithi146 ceph-mon[103278]: pgmap v9301: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:02 smithi078 ceph-mon[99959]: pgmap v9301: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:02 smithi078 ceph-mon[91136]: pgmap v9301: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:04.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:04 smithi146 ceph-mon[103278]: pgmap v9302: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:04 smithi078 ceph-mon[99959]: pgmap v9302: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:04 smithi078 ceph-mon[91136]: pgmap v9302: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[91136]: pgmap v9303: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[99959]: pgmap v9303: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:06.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:06 smithi146 ceph-mon[103278]: pgmap v9303: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:06.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:06.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:30:07.621 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:30:07.621 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 5h - - 2024-09-21T01:30:07.621 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 5h - - 2024-09-21T01:30:07.621 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 2s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:30:07.621 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 17s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 2s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 17s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 2s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 5h 511M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 17s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 5h 354M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 2s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 17s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 5h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 2s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 17s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 17s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 17s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (4h) 17s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 17s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 2s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 2s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:30:07.623 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:30:07.932 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:30:08.200 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:30:08.200 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:30:08.200 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:30:08.201 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:30:08.508 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:08.508 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:08.508 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:08.508 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:08.508 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[91136]: from='client.85058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[91136]: pgmap v9304: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[91136]: from='client.85064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1533200465' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2390060248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[99959]: from='client.85058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[99959]: pgmap v9304: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[99959]: from='client.85064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1533200465' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2390060248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:08 smithi146 ceph-mon[103278]: from='client.85058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:08 smithi146 ceph-mon[103278]: pgmap v9304: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:08 smithi146 ceph-mon[103278]: from='client.85064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1533200465' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2390060248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:09 smithi146 ceph-mon[103278]: from='client.85070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:09 smithi146 ceph-mon[103278]: from='client.85082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:09 smithi078 ceph-mon[99959]: from='client.85070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:09 smithi078 ceph-mon[99959]: from='client.85082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:09 smithi078 ceph-mon[91136]: from='client.85070 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:09 smithi078 ceph-mon[91136]: from='client.85082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:10 smithi146 ceph-mon[103278]: pgmap v9305: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:30:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:30:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[91136]: pgmap v9305: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[99959]: pgmap v9305: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:11.446Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:11.446Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:11.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:12 smithi146 ceph-mon[103278]: pgmap v9306: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:12 smithi078 ceph-mon[91136]: pgmap v9306: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:12 smithi078 ceph-mon[99959]: pgmap v9306: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:14 smithi146 ceph-mon[103278]: pgmap v9307: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:14 smithi078 ceph-mon[91136]: pgmap v9307: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:14 smithi078 ceph-mon[99959]: pgmap v9307: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:15 smithi146 ceph-mon[103278]: pgmap v9308: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:15 smithi078 ceph-mon[99959]: pgmap v9308: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:15 smithi078 ceph-mon[91136]: pgmap v9308: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:18 smithi146 ceph-mon[103278]: pgmap v9309: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:18 smithi078 ceph-mon[99959]: pgmap v9309: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:18 smithi078 ceph-mon[91136]: pgmap v9309: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:20 smithi146 ceph-mon[103278]: pgmap v9310: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:20 smithi078 ceph-mon[91136]: pgmap v9310: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:20 smithi078 ceph-mon[99959]: pgmap v9310: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:30:20] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:30:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:21.446Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:21.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:21.725 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:21.447Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:22 smithi146 ceph-mon[103278]: pgmap v9311: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:22 smithi078 ceph-mon[99959]: pgmap v9311: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:22 smithi078 ceph-mon[91136]: pgmap v9311: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:24 smithi146 ceph-mon[103278]: pgmap v9312: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:24 smithi078 ceph-mon[99959]: pgmap v9312: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:24 smithi078 ceph-mon[91136]: pgmap v9312: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:26 smithi146 ceph-mon[103278]: pgmap v9313: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[91136]: pgmap v9313: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[99959]: pgmap v9313: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:28 smithi146 ceph-mon[103278]: pgmap v9314: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:28 smithi078 ceph-mon[91136]: pgmap v9314: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:28 smithi078 ceph-mon[99959]: pgmap v9314: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:30 smithi146 ceph-mon[103278]: pgmap v9315: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:30 smithi078 ceph-mon[99959]: pgmap v9315: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:30 smithi078 ceph-mon[91136]: pgmap v9315: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:30:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:30:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:31.447Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:31.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:31.448Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:32 smithi146 ceph-mon[103278]: pgmap v9316: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:32 smithi078 ceph-mon[99959]: pgmap v9316: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:32 smithi078 ceph-mon[91136]: pgmap v9316: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:34 smithi146 ceph-mon[103278]: pgmap v9317: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:34 smithi078 ceph-mon[99959]: pgmap v9317: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:34 smithi078 ceph-mon[91136]: pgmap v9317: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:36 smithi146 ceph-mon[103278]: pgmap v9318: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:36 smithi078 ceph-mon[91136]: pgmap v9318: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:36 smithi078 ceph-mon[99959]: pgmap v9318: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:38 smithi146 ceph-mon[103278]: pgmap v9319: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:38 smithi078 ceph-mon[91136]: pgmap v9319: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:38 smithi078 ceph-mon[99959]: pgmap v9319: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:38.804 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:30:39.326 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:30:39.326 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 5h - - 2024-09-21T01:30:39.326 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 5h - - 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 13s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 8s ago 5h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 13s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 8s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 13s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 13s ago 5h 510M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 8s ago 5h 342M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 13s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 13s ago 5h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 8s ago 5h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 13s ago 5h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 13s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:30:39.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 13s ago 5h 78.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 13s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 8s ago 5h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 8s ago 5h 83.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 8s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 8s ago 5h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 8s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (4h) 13s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (4h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (4h) 13s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:30:39.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (4h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:30:39.637 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:30:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:30:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:30:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:30:39.638 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:30:39.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:30:39.907 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:30:40.215 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:30:40.215 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:40.215 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:30:40.215 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:40.215 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:30:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[99959]: from='client.85094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[99959]: from='client.85100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[99959]: pgmap v9320: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[99959]: from='client.85106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3006224779' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2751107264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:40.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[91136]: from='client.85094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[91136]: from='client.85100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[91136]: pgmap v9320: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:40.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[91136]: from='client.85106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3006224779' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:40.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2751107264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:40 smithi146 ceph-mon[103278]: from='client.85094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:40 smithi146 ceph-mon[103278]: from='client.85100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:40 smithi146 ceph-mon[103278]: pgmap v9320: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:40 smithi146 ceph-mon[103278]: from='client.85106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3006224779' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:30:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2751107264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:30:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:30:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:30:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:41 smithi146 ceph-mon[103278]: from='client.65199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:41.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:41 smithi078 ceph-mon[91136]: from='client.65199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:41 smithi078 ceph-mon[99959]: from='client.65199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:30:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:41.448Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:41.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:41.449Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:42 smithi146 ceph-mon[103278]: pgmap v9321: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:42 smithi078 ceph-mon[91136]: pgmap v9321: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:42 smithi078 ceph-mon[99959]: pgmap v9321: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:44 smithi146 ceph-mon[103278]: pgmap v9322: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:44 smithi078 ceph-mon[91136]: pgmap v9322: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:44 smithi078 ceph-mon[99959]: pgmap v9322: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:46 smithi146 ceph-mon[103278]: pgmap v9323: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[99959]: pgmap v9323: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[91136]: pgmap v9323: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:46.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:48.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:48 smithi078 ceph-mon[99959]: pgmap v9324: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:48.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:48 smithi078 ceph-mon[91136]: pgmap v9324: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:48 smithi146 ceph-mon[103278]: pgmap v9324: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:50 smithi146 ceph-mon[103278]: pgmap v9325: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:50 smithi078 ceph-mon[91136]: pgmap v9325: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:50 smithi078 ceph-mon[99959]: pgmap v9325: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:30:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:30:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:30:51.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.700 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.700 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:51.449Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:51.700 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:51.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:30:51.701 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:30:51.450Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:30:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:51.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:30:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:52 smithi146 ceph-mon[103278]: pgmap v9326: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:52 smithi078 ceph-mon[91136]: pgmap v9326: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:52 smithi078 ceph-mon[99959]: pgmap v9326: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:30:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:54 smithi146 ceph-mon[103278]: pgmap v9327: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:30:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:54 smithi078 ceph-mon[91136]: pgmap v9327: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:30:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:54 smithi078 ceph-mon[99959]: pgmap v9327: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:30:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:56 smithi146 ceph-mon[103278]: pgmap v9328: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T01:30:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:56 smithi078 ceph-mon[91136]: pgmap v9328: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T01:30:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:30:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:56 smithi078 ceph-mon[99959]: pgmap v9328: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T01:30:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:58 smithi146 ceph-mon[103278]: pgmap v9329: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T01:30:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:58 smithi078 ceph-mon[91136]: pgmap v9329: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T01:30:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:58 smithi078 ceph-mon[99959]: pgmap v9329: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T01:30:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:00 smithi146 ceph-mon[103278]: pgmap v9330: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:31:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:00 smithi078 ceph-mon[91136]: pgmap v9330: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:31:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:00 smithi078 ceph-mon[99959]: pgmap v9330: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T01:31:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:31:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:31:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:01.450Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:01.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:01.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:02 smithi146 ceph-mon[103278]: pgmap v9331: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T01:31:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:02 smithi078 ceph-mon[91136]: pgmap v9331: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T01:31:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:02 smithi078 ceph-mon[99959]: pgmap v9331: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T01:31:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:04 smithi146 ceph-mon[103278]: pgmap v9332: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T01:31:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:04 smithi078 ceph-mon[91136]: pgmap v9332: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T01:31:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:04 smithi078 ceph-mon[99959]: pgmap v9332: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-21T01:31:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:06 smithi146 ceph-mon[103278]: pgmap v9333: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-21T01:31:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[99959]: pgmap v9333: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-21T01:31:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[91136]: pgmap v9333: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-09-21T01:31:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:08.617 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:08 smithi146 ceph-mon[103278]: pgmap v9334: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T01:31:08.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:08 smithi078 ceph-mon[99959]: pgmap v9334: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T01:31:08.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:08 smithi078 ceph-mon[91136]: pgmap v9334: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T01:31:10.510 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:31:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:10 smithi146 ceph-mon[103278]: pgmap v9335: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T01:31:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:31:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:31:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:10 smithi078 ceph-mon[99959]: pgmap v9335: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T01:31:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:10 smithi078 ceph-mon[91136]: pgmap v9335: 161 pgs: 161 active+clean; 585 KiB data, 309 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T01:31:11.049 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:31:11.049 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 5h - - 2024-09-21T01:31:11.049 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 5h - - 2024-09-21T01:31:11.049 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 5s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:31:11.049 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 0s ago 5h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:31:11.049 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 5s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 0s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 5s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 5s ago 5h 511M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 0s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 5s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 5s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 0s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 5s ago 5h 89.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 5s ago 5h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 5s ago 5h 79.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 5s ago 5h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 0s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 0s ago 5h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:31:11.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 0s ago 5h 90.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:31:11.051 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 0s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:31:11.051 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 0s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:31:11.051 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 5s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:31:11.051 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:31:11.051 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 5s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:31:11.051 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:31:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:31:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:31:11.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:31:11.629 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:31:11.629 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:31:11.629 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:31:11.629 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:31:11.630 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:31:11.630 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[99959]: from='client.65211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:11.630 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4142884153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[91136]: from='client.65211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4142884153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:11.631 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:11.450Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:11.632 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:11.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:11.633 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:11.451Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:11 smithi146 ceph-mon[103278]: from='client.65211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4142884153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:12 smithi146 ceph-mon[103278]: from='client.85136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:12 smithi146 ceph-mon[103278]: from='client.65223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:12 smithi146 ceph-mon[103278]: pgmap v9336: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 156 op/s 2024-09-21T01:31:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2550320004' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[99959]: from='client.85136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[99959]: from='client.65223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[99959]: pgmap v9336: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 156 op/s 2024-09-21T01:31:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2550320004' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[91136]: from='client.85136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[91136]: from='client.65223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[91136]: pgmap v9336: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 156 op/s 2024-09-21T01:31:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2550320004' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:13.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:13 smithi078 ceph-mon[99959]: from='client.65235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:13.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:13 smithi078 ceph-mon[91136]: from='client.65235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:13 smithi146 ceph-mon[103278]: from='client.65235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:14 smithi078 ceph-mon[99959]: pgmap v9337: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-09-21T01:31:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:14 smithi078 ceph-mon[91136]: pgmap v9337: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-09-21T01:31:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:14 smithi146 ceph-mon[103278]: pgmap v9337: 161 pgs: 161 active+clean; 585 KiB data, 313 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-09-21T01:31:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:16 smithi078 ceph-mon[91136]: pgmap v9338: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 158 op/s 2024-09-21T01:31:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:16 smithi078 ceph-mon[99959]: pgmap v9338: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 158 op/s 2024-09-21T01:31:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:16 smithi146 ceph-mon[103278]: pgmap v9338: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 158 op/s 2024-09-21T01:31:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:18 smithi078 ceph-mon[91136]: pgmap v9339: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 0 B/s wr, 154 op/s 2024-09-21T01:31:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:18 smithi078 ceph-mon[99959]: pgmap v9339: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 0 B/s wr, 154 op/s 2024-09-21T01:31:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:18 smithi146 ceph-mon[103278]: pgmap v9339: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 0 B/s wr, 154 op/s 2024-09-21T01:31:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:20 smithi078 ceph-mon[99959]: pgmap v9340: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T01:31:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:20 smithi078 ceph-mon[91136]: pgmap v9340: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T01:31:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:31:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:31:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:20 smithi146 ceph-mon[103278]: pgmap v9340: 161 pgs: 161 active+clean; 585 KiB data, 317 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-21T01:31:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:21.451Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:21.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:21.452Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:22 smithi078 ceph-mon[91136]: pgmap v9341: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T01:31:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:22 smithi078 ceph-mon[99959]: pgmap v9341: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T01:31:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:22 smithi146 ceph-mon[103278]: pgmap v9341: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T01:31:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:24 smithi078 ceph-mon[91136]: pgmap v9342: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:31:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:24 smithi078 ceph-mon[99959]: pgmap v9342: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:31:24.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:24 smithi146 ceph-mon[103278]: pgmap v9342: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T01:31:27.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:26 smithi146 ceph-mon[103278]: pgmap v9343: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:31:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[99959]: pgmap v9343: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:31:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[91136]: pgmap v9343: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:31:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:28 smithi146 ceph-mon[103278]: pgmap v9344: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T01:31:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:28 smithi078 ceph-mon[99959]: pgmap v9344: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T01:31:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:28 smithi078 ceph-mon[91136]: pgmap v9344: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T01:31:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:30 smithi146 ceph-mon[103278]: pgmap v9345: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:31:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:30 smithi078 ceph-mon[91136]: pgmap v9345: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:31:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:30 smithi078 ceph-mon[99959]: pgmap v9345: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:31:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:31:30] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T01:31:31.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:31.452Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:31.730 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:31.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:31.730 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:31.453Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:31 smithi146 ceph-mon[103278]: pgmap v9346: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:31:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[99959]: pgmap v9346: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:31:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:31 smithi078 ceph-mon[91136]: pgmap v9346: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:31:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:34 smithi146 ceph-mon[103278]: pgmap v9347: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:31:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:34 smithi078 ceph-mon[91136]: pgmap v9347: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:31:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:34 smithi078 ceph-mon[99959]: pgmap v9347: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:31:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:36 smithi146 ceph-mon[103278]: pgmap v9348: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:36 smithi078 ceph-mon[91136]: pgmap v9348: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:36.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:36 smithi078 ceph-mon[99959]: pgmap v9348: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:38 smithi146 ceph-mon[103278]: pgmap v9349: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:38 smithi078 ceph-mon[91136]: pgmap v9349: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:38 smithi078 ceph-mon[99959]: pgmap v9349: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:40 smithi146 ceph-mon[103278]: pgmap v9350: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:40 smithi078 ceph-mon[91136]: pgmap v9350: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:40 smithi078 ceph-mon[99959]: pgmap v9350: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:31:40] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T01:31:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:41.453Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:41.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:41.454Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:42.236 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:31:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:42 smithi146 ceph-mon[103278]: pgmap v9351: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:42 smithi078 ceph-mon[99959]: pgmap v9351: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:42 smithi078 ceph-mon[91136]: pgmap v9351: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:42.778 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:31:42.778 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T01:31:42.778 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 5h - - 2024-09-21T01:31:42.778 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 16s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:31:42.778 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 12s ago 5h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 12s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 5h 512M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 5h 344M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 16s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 12s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 5h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 5h 79.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:31:42.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:31:42.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 5h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:31:42.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 5h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:31:42.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:31:42.780 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 12s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:31:42.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 16s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:31:42.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:31:42.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 16s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:31:42.781 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:31:43.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:31:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:31:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:31:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:31:43.378 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:31:43.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:43 smithi078 ceph-mon[99959]: from='client.65247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:43.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:43 smithi078 ceph-mon[99959]: from='client.65253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:43.378 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/32957958' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:43.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:43 smithi078 ceph-mon[91136]: from='client.65247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:43.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:43 smithi078 ceph-mon[91136]: from='client.65253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:43.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/32957958' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:43.690 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:31:43.690 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:31:43.690 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:31:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:43 smithi146 ceph-mon[103278]: from='client.65247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:43 smithi146 ceph-mon[103278]: from='client.65253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/32957958' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:31:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[99959]: from='client.65259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[99959]: pgmap v9352: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[99959]: from='client.65271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/444666785' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[91136]: from='client.65259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[91136]: pgmap v9352: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[91136]: from='client.65271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/444666785' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:44 smithi146 ceph-mon[103278]: from='client.65259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:44 smithi146 ceph-mon[103278]: pgmap v9352: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:44 smithi146 ceph-mon[103278]: from='client.65271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:31:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/444666785' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:31:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[91136]: pgmap v9353: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[99959]: pgmap v9353: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:46 smithi146 ceph-mon[103278]: pgmap v9353: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:48.377 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:48 smithi146 ceph-mon[103278]: pgmap v9354: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:48.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:48 smithi078 ceph-mon[91136]: pgmap v9354: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:48.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:48 smithi078 ceph-mon[99959]: pgmap v9354: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:50 smithi078 ceph-mon[91136]: pgmap v9355: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:50 smithi078 ceph-mon[99959]: pgmap v9355: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:50 smithi146 ceph-mon[103278]: pgmap v9355: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:50.965 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:31:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:31:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:51.454Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:51.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:31:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:31:51.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:31:52.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[91136]: pgmap v9356: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:52.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:51 smithi078 ceph-mon[99959]: pgmap v9356: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:31:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:51 smithi146 ceph-mon[103278]: pgmap v9356: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:54 smithi146 ceph-mon[103278]: pgmap v9357: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:54 smithi078 ceph-mon[91136]: pgmap v9357: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:54 smithi078 ceph-mon[99959]: pgmap v9357: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:31:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:56 smithi146 ceph-mon[103278]: pgmap v9358: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:56.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:56 smithi078 ceph-mon[99959]: pgmap v9358: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:56 smithi078 ceph-mon[91136]: pgmap v9358: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:31:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:58 smithi146 ceph-mon[103278]: pgmap v9359: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:58 smithi078 ceph-mon[91136]: pgmap v9359: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:58 smithi078 ceph-mon[99959]: pgmap v9359: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:31:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:00 smithi078 ceph-mon[99959]: pgmap v9360: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:00 smithi078 ceph-mon[91136]: pgmap v9360: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:00 smithi146 ceph-mon[103278]: pgmap v9360: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:32:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:32:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:01.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:01.455Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:01.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:02 smithi078 ceph-mon[99959]: pgmap v9361: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:02 smithi078 ceph-mon[91136]: pgmap v9361: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:02 smithi146 ceph-mon[103278]: pgmap v9361: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:04 smithi078 ceph-mon[99959]: pgmap v9362: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:04 smithi078 ceph-mon[91136]: pgmap v9362: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:04.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:04 smithi146 ceph-mon[103278]: pgmap v9362: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:06 smithi078 ceph-mon[99959]: pgmap v9363: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:06 smithi078 ceph-mon[91136]: pgmap v9363: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:06 smithi146 ceph-mon[103278]: pgmap v9363: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:08.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:08 smithi146 ceph-mon[103278]: pgmap v9364: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:08 smithi078 ceph-mon[99959]: pgmap v9364: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:08 smithi078 ceph-mon[91136]: pgmap v9364: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:10 smithi078 ceph-mon[91136]: pgmap v9365: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:10 smithi078 ceph-mon[99959]: pgmap v9365: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:10.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:10 smithi146 ceph-mon[103278]: pgmap v9365: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:32:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:32:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:11.455Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:11.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:11.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:11 smithi146 ceph-mon[103278]: pgmap v9366: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[91136]: pgmap v9366: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:11 smithi078 ceph-mon[99959]: pgmap v9366: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:13.987 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 5h - - 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 5h - - 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 8s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 3s ago 5h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 8s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 3s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 8s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 8s ago 5h 513M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 3s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 8s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 8s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 3s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 8s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 8s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 8s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 8s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 3s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 3s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 3s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 3s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 3s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 8s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 8s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:32:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:14 smithi146 ceph-mon[103278]: pgmap v9367: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:14 smithi078 ceph-mon[91136]: pgmap v9367: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:14 smithi078 ceph-mon[99959]: pgmap v9367: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:32:14.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:32:14.823 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:32:15.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:32:15.091 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:32:15.404 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:32:15.404 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:15.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:15.404 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:15.404 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[91136]: from='client.65283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[91136]: from='client.85208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[91136]: from='client.85214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1773288446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[99959]: from='client.65283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[99959]: from='client.85208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[99959]: from='client.85214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1773288446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:15 smithi146 ceph-mon[103278]: from='client.65283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:15 smithi146 ceph-mon[103278]: from='client.85208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:15 smithi146 ceph-mon[103278]: from='client.85214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1773288446' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[91136]: from='client.85226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[91136]: pgmap v9368: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2994404328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[99959]: from='client.85226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[99959]: pgmap v9368: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2994404328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:16 smithi146 ceph-mon[103278]: from='client.85226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:16 smithi146 ceph-mon[103278]: pgmap v9368: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2994404328' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:18 smithi078 ceph-mon[91136]: pgmap v9369: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:18 smithi078 ceph-mon[99959]: pgmap v9369: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:18 smithi146 ceph-mon[103278]: pgmap v9369: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:20 smithi078 ceph-mon[91136]: pgmap v9370: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:20 smithi078 ceph-mon[99959]: pgmap v9370: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:20 smithi146 ceph-mon[103278]: pgmap v9370: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:32:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:32:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:21.456Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:21.456Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:21.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:22 smithi078 ceph-mon[91136]: pgmap v9371: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:22.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:22 smithi078 ceph-mon[99959]: pgmap v9371: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:22 smithi146 ceph-mon[103278]: pgmap v9371: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:24 smithi078 ceph-mon[99959]: pgmap v9372: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:24 smithi078 ceph-mon[91136]: pgmap v9372: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:24.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:24 smithi146 ceph-mon[103278]: pgmap v9372: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:26 smithi078 ceph-mon[91136]: pgmap v9373: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:26 smithi078 ceph-mon[99959]: pgmap v9373: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:26 smithi146 ceph-mon[103278]: pgmap v9373: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:28 smithi146 ceph-mon[103278]: pgmap v9374: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:28 smithi078 ceph-mon[99959]: pgmap v9374: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:28 smithi078 ceph-mon[91136]: pgmap v9374: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:29.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:29.732 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:29.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:30.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:30 smithi078 ceph-mon[99959]: pgmap v9375: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:30.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:30 smithi078 ceph-mon[91136]: pgmap v9375: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:30.728 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:32:30] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:32:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:30 smithi146 ceph-mon[103278]: pgmap v9375: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:31.661 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:31.456Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:31.662 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:31.457Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:31.662 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:31.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:32 smithi146 ceph-mon[103278]: pgmap v9376: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[99959]: pgmap v9376: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:32 smithi078 ceph-mon[91136]: pgmap v9376: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:34.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:34 smithi078 ceph-mon[99959]: pgmap v9377: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:34 smithi078 ceph-mon[91136]: pgmap v9377: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:34 smithi146 ceph-mon[103278]: pgmap v9377: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:36 smithi078 ceph-mon[91136]: pgmap v9378: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:36 smithi078 ceph-mon[99959]: pgmap v9378: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:36 smithi146 ceph-mon[103278]: pgmap v9378: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:38 smithi078 ceph-mon[91136]: pgmap v9379: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:38 smithi078 ceph-mon[99959]: pgmap v9379: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:38 smithi146 ceph-mon[103278]: pgmap v9379: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:40 smithi078 ceph-mon[91136]: pgmap v9380: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:40 smithi078 ceph-mon[99959]: pgmap v9380: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:40 smithi146 ceph-mon[103278]: pgmap v9380: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:32:40] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:32:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:41.457Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:41.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:41.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:42 smithi078 ceph-mon[91136]: pgmap v9381: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:42 smithi078 ceph-mon[99959]: pgmap v9381: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:42.590 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:42 smithi146 ceph-mon[103278]: pgmap v9381: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:44 smithi146 ceph-mon[103278]: pgmap v9382: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:44 smithi078 ceph-mon[91136]: pgmap v9382: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:44 smithi078 ceph-mon[99959]: pgmap v9382: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:45.703 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:32:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:32:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T01:32:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 5h - - 2024-09-21T01:32:46.229 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 15s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 15s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 19s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 5h 513M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 15s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 15s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:32:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 15s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 15s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 15s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 15s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 15s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 19s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 19s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:32:46.231 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:32:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:46 smithi078 ceph-mon[91136]: pgmap v9383: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:46.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:46 smithi078 ceph-mon[99959]: pgmap v9383: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:32:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:32:46.545 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:32:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:46 smithi146 ceph-mon[103278]: pgmap v9383: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:46.818 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:32:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:32:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:32:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:32:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:32:46.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:32:47.126 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:32:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:47.127 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:32:47.127 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:47.127 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:32:47.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='client.85238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='client.65325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='client.65331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1273366244' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/169680076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='client.85238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='client.65325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='client.65331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1273366244' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/169680076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='client.85238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='client.65325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='client.65331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1273366244' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/169680076' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:32:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:48 smithi146 ceph-mon[103278]: from='client.65343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:48 smithi146 ceph-mon[103278]: pgmap v9384: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:48 smithi078 ceph-mon[91136]: from='client.65343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:48 smithi078 ceph-mon[91136]: pgmap v9384: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:48 smithi078 ceph-mon[99959]: from='client.65343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:32:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:48 smithi078 ceph-mon[99959]: pgmap v9384: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:50 smithi078 ceph-mon[91136]: pgmap v9385: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:32:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:32:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:50 smithi078 ceph-mon[99959]: pgmap v9385: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:50.863 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:50 smithi146 ceph-mon[103278]: pgmap v9385: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:51.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:51.458Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:51.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:51.458Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:32:51.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:32:51.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:32:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[99959]: pgmap v9386: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[91136]: pgmap v9386: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:52 smithi146 ceph-mon[103278]: pgmap v9386: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:32:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:54 smithi146 ceph-mon[103278]: pgmap v9387: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:54 smithi078 ceph-mon[91136]: pgmap v9387: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:54 smithi078 ceph-mon[99959]: pgmap v9387: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:32:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:55 smithi146 ceph-mon[103278]: pgmap v9388: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:55 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:55 smithi078 ceph-mon[99959]: pgmap v9388: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:55 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:55 smithi078 ceph-mon[91136]: pgmap v9388: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:55 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:32:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:58 smithi078 ceph-mon[91136]: pgmap v9389: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:58 smithi078 ceph-mon[99959]: pgmap v9389: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:58 smithi146 ceph-mon[103278]: pgmap v9389: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:32:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:00 smithi078 ceph-mon[99959]: pgmap v9390: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:00 smithi078 ceph-mon[91136]: pgmap v9390: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:00 smithi146 ceph-mon[103278]: pgmap v9390: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:33:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:33:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:01.459Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:01.459Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:01.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:02.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:02 smithi078 ceph-mon[91136]: pgmap v9391: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:02.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:02 smithi078 ceph-mon[99959]: pgmap v9391: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:02 smithi146 ceph-mon[103278]: pgmap v9391: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:04 smithi146 ceph-mon[103278]: pgmap v9392: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:04 smithi078 ceph-mon[91136]: pgmap v9392: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:04 smithi078 ceph-mon[99959]: pgmap v9392: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:06.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:06 smithi146 ceph-mon[103278]: pgmap v9393: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:06.567 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:06.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:06 smithi078 ceph-mon[99959]: pgmap v9393: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:06.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:06.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:06 smithi078 ceph-mon[91136]: pgmap v9393: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:06.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:07 smithi146 ceph-mon[103278]: pgmap v9394: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[91136]: pgmap v9394: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:07 smithi078 ceph-mon[99959]: pgmap v9394: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:10 smithi078 ceph-mon[91136]: pgmap v9395: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:10 smithi078 ceph-mon[99959]: pgmap v9395: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:10.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:10 smithi146 ceph-mon[103278]: pgmap v9395: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:33:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:33:11.644 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:11.459Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:11.645 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:11.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:11.645 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:11.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:12.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:12 smithi146 ceph-mon[103278]: pgmap v9396: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:12.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.616 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[91136]: pgmap v9396: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[99959]: pgmap v9396: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:14 smithi146 ceph-mon[103278]: pgmap v9397: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:14 smithi078 ceph-mon[91136]: pgmap v9397: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:14 smithi078 ceph-mon[99959]: pgmap v9397: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:16 smithi146 ceph-mon[103278]: pgmap v9398: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:16 smithi078 ceph-mon[91136]: pgmap v9398: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:16 smithi078 ceph-mon[99959]: pgmap v9398: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:17.420 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 5h - - 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 5h - - 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 11s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 6s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 11s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:33:17.947 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 6s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 11s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 11s ago 5h 511M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 6s ago 5h 347M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 11s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 11s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 6s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 11s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 11s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 11s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 11s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 6s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 6s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 6s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 6s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:33:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 6s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:33:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 11s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:33:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:33:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 11s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:33:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:33:18.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:33:18.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:18.261 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:33:18.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:33:18.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:33:18.261 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:33:18.528 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:33:18.528 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:33:18.529 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:33:18.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:18 smithi078 ceph-mon[91136]: pgmap v9399: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:18.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:18 smithi078 ceph-mon[91136]: from='client.85274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:18.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1466043803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:18.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:18 smithi078 ceph-mon[99959]: pgmap v9399: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:18.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:18 smithi078 ceph-mon[99959]: from='client.85274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:18.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1466043803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:18 smithi146 ceph-mon[103278]: pgmap v9399: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:18 smithi146 ceph-mon[103278]: from='client.85274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1466043803' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:18.838 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:33:18.839 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:18.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:18.839 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:18.839 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:19 smithi146 ceph-mon[103278]: from='client.65361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:19 smithi146 ceph-mon[103278]: from='client.65367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:19 smithi146 ceph-mon[103278]: from='client.85298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/231502635' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[99959]: from='client.65361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[99959]: from='client.65367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[99959]: from='client.85298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/231502635' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[91136]: from='client.65361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[91136]: from='client.65367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[91136]: from='client.85298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/231502635' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:20 smithi146 ceph-mon[103278]: pgmap v9400: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:20 smithi078 ceph-mon[99959]: pgmap v9400: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:20 smithi078 ceph-mon[91136]: pgmap v9400: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:33:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:33:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:21.460Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:21.460Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:21.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:22 smithi146 ceph-mon[103278]: pgmap v9401: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:22 smithi078 ceph-mon[91136]: pgmap v9401: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:22 smithi078 ceph-mon[99959]: pgmap v9401: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:24 smithi146 ceph-mon[103278]: pgmap v9402: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:24 smithi078 ceph-mon[91136]: pgmap v9402: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:24 smithi078 ceph-mon[99959]: pgmap v9402: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:26 smithi146 ceph-mon[103278]: pgmap v9403: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:26 smithi078 ceph-mon[99959]: pgmap v9403: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:26 smithi078 ceph-mon[91136]: pgmap v9403: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:27 smithi146 ceph-mon[103278]: pgmap v9404: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[99959]: pgmap v9404: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:27 smithi078 ceph-mon[91136]: pgmap v9404: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:29 smithi146 ceph-mon[103278]: pgmap v9405: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:29 smithi078 ceph-mon[99959]: pgmap v9405: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:29 smithi078 ceph-mon[91136]: pgmap v9405: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:33:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:33:31.732 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:31.461Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:31.732 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:31.461Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:31.732 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:31.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:32.653 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[99959]: pgmap v9406: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:32.653 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[91136]: pgmap v9406: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:32.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:32 smithi146 ceph-mon[103278]: pgmap v9406: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:34 smithi146 ceph-mon[103278]: pgmap v9407: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:34 smithi078 ceph-mon[91136]: pgmap v9407: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:34 smithi078 ceph-mon[99959]: pgmap v9407: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:36 smithi146 ceph-mon[103278]: pgmap v9408: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:36 smithi078 ceph-mon[99959]: pgmap v9408: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:36 smithi078 ceph-mon[91136]: pgmap v9408: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:38 smithi146 ceph-mon[103278]: pgmap v9409: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:38 smithi078 ceph-mon[99959]: pgmap v9409: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:38 smithi078 ceph-mon[91136]: pgmap v9409: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:40 smithi146 ceph-mon[103278]: pgmap v9410: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:40 smithi078 ceph-mon[99959]: pgmap v9410: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:40 smithi078 ceph-mon[91136]: pgmap v9410: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:33:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:33:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:41.462Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:41.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:41.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:42 smithi146 ceph-mon[103278]: pgmap v9411: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:42 smithi078 ceph-mon[91136]: pgmap v9411: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:42 smithi078 ceph-mon[99959]: pgmap v9411: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:43.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:44 smithi146 ceph-mon[103278]: pgmap v9412: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:44 smithi078 ceph-mon[99959]: pgmap v9412: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:44 smithi078 ceph-mon[91136]: pgmap v9412: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:46 smithi146 ceph-mon[103278]: pgmap v9413: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:46 smithi078 ceph-mon[99959]: pgmap v9413: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:46 smithi078 ceph-mon[91136]: pgmap v9413: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:47 smithi146 ceph-mon[103278]: pgmap v9414: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[99959]: pgmap v9414: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:47 smithi078 ceph-mon[91136]: pgmap v9414: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:49.132 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:33:49.658 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:33:49.658 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 5h - - 2024-09-21T01:33:49.658 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 5h - - 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 2s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 18s ago 5h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 2s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 18s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 2s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 5h 514M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 18s ago 5h 345M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 2s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 18s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 2s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:33:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 18s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 2s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 2s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:33:49.660 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:49.974 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:33:49.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:33:49.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:33:49.975 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:33:49.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:33:49.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:33:49.975 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:33:50.242 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:33:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:33:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:33:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:33:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:33:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:33:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:33:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[99959]: from='client.65391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[99959]: pgmap v9415: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[99959]: from='client.65397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:50.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/360620596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[91136]: from='client.65391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[91136]: pgmap v9415: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[91136]: from='client.65397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/360620596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:50.552 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:33:50.553 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:50.553 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:33:50.553 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:50.553 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:33:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:50 smithi146 ceph-mon[103278]: from='client.65391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:50 smithi146 ceph-mon[103278]: pgmap v9415: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:50 smithi146 ceph-mon[103278]: from='client.65397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/360620596' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:33:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:33:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:33:51.634 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:51 smithi146 ceph-mon[103278]: from='client.65403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:51.634 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:51 smithi146 ceph-mon[103278]: from='client.85334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:51.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/692730762' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:51.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-mon[99959]: from='client.65403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:51.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-mon[99959]: from='client.85334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:51.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/692730762' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-mon[91136]: from='client.65403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-mon[91136]: from='client.85334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:33:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/692730762' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:33:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:51.462Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:51.462Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:33:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:33:51.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:33:52.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[99959]: pgmap v9416: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:52.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.473 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.474 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[91136]: pgmap v9416: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:52.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:52 smithi146 ceph-mon[103278]: pgmap v9416: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:33:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:54 smithi146 ceph-mon[103278]: pgmap v9417: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:54.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:54 smithi078 ceph-mon[91136]: pgmap v9417: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:54.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:54 smithi078 ceph-mon[99959]: pgmap v9417: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:33:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:56 smithi146 ceph-mon[103278]: pgmap v9418: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:56 smithi078 ceph-mon[91136]: pgmap v9418: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:56 smithi078 ceph-mon[99959]: pgmap v9418: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:33:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:58 smithi146 ceph-mon[103278]: pgmap v9419: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:58 smithi078 ceph-mon[99959]: pgmap v9419: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:58 smithi078 ceph-mon[91136]: pgmap v9419: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:33:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:00 smithi078 ceph-mon[99959]: pgmap v9420: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:00 smithi078 ceph-mon[91136]: pgmap v9420: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:00 smithi146 ceph-mon[103278]: pgmap v9420: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:34:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:34:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:01.462Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:01.463Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:01.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:02 smithi146 ceph-mon[103278]: pgmap v9421: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:02 smithi078 ceph-mon[91136]: pgmap v9421: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:02 smithi078 ceph-mon[99959]: pgmap v9421: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:04 smithi078 ceph-mon[99959]: pgmap v9422: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:04 smithi078 ceph-mon[91136]: pgmap v9422: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:04 smithi146 ceph-mon[103278]: pgmap v9422: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:06 smithi146 ceph-mon[103278]: pgmap v9423: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:06 smithi078 ceph-mon[91136]: pgmap v9423: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:06 smithi078 ceph-mon[99959]: pgmap v9423: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:08 smithi146 ceph-mon[103278]: pgmap v9424: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[91136]: pgmap v9424: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:08 smithi078 ceph-mon[99959]: pgmap v9424: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:10 smithi078 ceph-mon[91136]: pgmap v9425: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:10.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:10 smithi078 ceph-mon[99959]: pgmap v9425: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:10 smithi146 ceph-mon[103278]: pgmap v9425: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:34:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:34:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:11.463Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:11.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:11.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:12.571 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:12 smithi146 ceph-mon[103278]: pgmap v9426: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:12.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[91136]: pgmap v9426: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[99959]: pgmap v9426: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:14 smithi146 ceph-mon[103278]: pgmap v9427: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:14 smithi078 ceph-mon[91136]: pgmap v9427: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:14 smithi078 ceph-mon[99959]: pgmap v9427: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:16 smithi146 ceph-mon[103278]: pgmap v9428: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:16 smithi078 ceph-mon[99959]: pgmap v9428: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:16 smithi078 ceph-mon[91136]: pgmap v9428: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:18 smithi146 ceph-mon[103278]: pgmap v9429: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:18 smithi078 ceph-mon[99959]: pgmap v9429: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:18 smithi078 ceph-mon[91136]: pgmap v9429: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:20.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:20 smithi078 ceph-mon[99959]: pgmap v9430: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:20.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:20 smithi078 ceph-mon[91136]: pgmap v9430: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:20 smithi146 ceph-mon[103278]: pgmap v9430: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:20.849 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:34:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:34:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 5h - - 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 5h - - 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 14s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 9s ago 5h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 14s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 9s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 14s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 14s ago 5h 514M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:34:21.376 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 9s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 14s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 14s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 9s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 14s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 14s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 14s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 14s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 9s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 9s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 9s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 9s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 9s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 14s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:34:21.377 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 14s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:34:21.378 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:34:21.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:34:21.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:34:21.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:34:21.689 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:21.464Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:21.689 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:21.464Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:21.689 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:21.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:21.958 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:34:21.958 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:34:21.959 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:34:22.275 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:34:22.275 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:22.275 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[91136]: from='client.85346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[91136]: from='client.85352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[91136]: pgmap v9431: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[91136]: from='client.65439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/559548386' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1682905077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[99959]: from='client.85346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[99959]: from='client.85352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[99959]: pgmap v9431: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[99959]: from='client.65439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/559548386' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1682905077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:22 smithi146 ceph-mon[103278]: from='client.85346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:22 smithi146 ceph-mon[103278]: from='client.85352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:22 smithi146 ceph-mon[103278]: pgmap v9431: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:22 smithi146 ceph-mon[103278]: from='client.65439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/559548386' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1682905077' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:23 smithi078 ceph-mon[99959]: from='client.65451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:23.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:23 smithi078 ceph-mon[91136]: from='client.65451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:23 smithi146 ceph-mon[103278]: from='client.65451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:24.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:24 smithi146 ceph-mon[103278]: pgmap v9432: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:24 smithi078 ceph-mon[99959]: pgmap v9432: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:24 smithi078 ceph-mon[91136]: pgmap v9432: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:26.015 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 01:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T01:34:25.847543718Z level=info msg="Completed cleanup jobs" duration=11.625925ms 2024-09-21T01:34:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:26 smithi146 ceph-mon[103278]: pgmap v9433: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:26 smithi078 ceph-mon[91136]: pgmap v9433: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:26 smithi078 ceph-mon[99959]: pgmap v9433: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:28 smithi146 ceph-mon[103278]: pgmap v9434: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[91136]: pgmap v9434: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:28 smithi078 ceph-mon[99959]: pgmap v9434: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:30 smithi146 ceph-mon[103278]: pgmap v9435: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:30.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:30 smithi078 ceph-mon[99959]: pgmap v9435: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:30 smithi078 ceph-mon[91136]: pgmap v9435: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:34:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:34:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:31.465Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:31.465Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:31.466Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[99959]: pgmap v9436: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[91136]: pgmap v9436: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.712 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:32 smithi146 ceph-mon[103278]: pgmap v9436: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:34 smithi146 ceph-mon[103278]: pgmap v9437: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:34.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:34 smithi078 ceph-mon[99959]: pgmap v9437: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:34.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:34 smithi078 ceph-mon[91136]: pgmap v9437: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:36 smithi078 ceph-mon[91136]: pgmap v9438: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:36 smithi078 ceph-mon[99959]: pgmap v9438: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:36 smithi146 ceph-mon[103278]: pgmap v9438: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:38 smithi146 ceph-mon[103278]: pgmap v9439: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:38 smithi078 ceph-mon[99959]: pgmap v9439: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:38 smithi078 ceph-mon[91136]: pgmap v9439: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:40 smithi078 ceph-mon[91136]: pgmap v9440: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:40 smithi078 ceph-mon[99959]: pgmap v9440: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:40 smithi146 ceph-mon[103278]: pgmap v9440: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:34:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:34:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:41.466Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:41.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:41.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:42 smithi146 ceph-mon[103278]: pgmap v9441: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:42 smithi078 ceph-mon[91136]: pgmap v9441: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:42 smithi078 ceph-mon[99959]: pgmap v9441: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:44 smithi146 ceph-mon[103278]: pgmap v9442: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:44 smithi078 ceph-mon[99959]: pgmap v9442: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:44 smithi078 ceph-mon[91136]: pgmap v9442: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:46 smithi146 ceph-mon[103278]: pgmap v9443: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:46 smithi078 ceph-mon[99959]: pgmap v9443: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:46 smithi078 ceph-mon[91136]: pgmap v9443: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:48 smithi146 ceph-mon[103278]: pgmap v9444: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[91136]: pgmap v9444: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[99959]: pgmap v9444: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:50 smithi078 ceph-mon[91136]: pgmap v9445: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:50 smithi078 ceph-mon[99959]: pgmap v9445: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:50 smithi146 ceph-mon[103278]: pgmap v9445: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:34:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:34:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:51.467Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:51.467Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:34:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:34:51.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:34:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[91136]: pgmap v9446: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[99959]: pgmap v9446: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.563 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:34:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:52 smithi146 ceph-mon[103278]: pgmap v9446: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:34:53.092 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:34:53.092 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 5h - - 2024-09-21T01:34:53.092 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 5h - - 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 5s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 1s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 5s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 1s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 5s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 5s ago 5h 515M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 1s ago 5h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 5s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 5s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 1s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 5s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 5s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 5s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:34:53.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 5s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 1s ago 5h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 5s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 5s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:34:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:34:53.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:53 smithi078 ceph-mon[91136]: from='client.65463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:34:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:34:53.402 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:34:53.647 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:53 smithi146 ceph-mon[103278]: from='client.65463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:53.671 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:53 smithi078 ceph-mon[99959]: from='client.65463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:34:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:34:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:34:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:34:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:34:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:34:53.672 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:34:53.983 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:34:53.983 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:53.984 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:34:53.984 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:53.984 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:34:54.593 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:54 smithi146 ceph-mon[103278]: from='client.65469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:54.594 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:54 smithi146 ceph-mon[103278]: from='client.85394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:54.594 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:54 smithi146 ceph-mon[103278]: pgmap v9447: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:54.594 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2017927700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:54.594 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/805605862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[91136]: from='client.65469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[91136]: from='client.85394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[91136]: pgmap v9447: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2017927700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/805605862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[99959]: from='client.65469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:54.632 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[99959]: from='client.85394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:54.633 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[99959]: pgmap v9447: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:34:54.633 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2017927700' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:34:54.633 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/805605862' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:34:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:55 smithi146 ceph-mon[103278]: from='client.65487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:55.653 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:55 smithi078 ceph-mon[91136]: from='client.65487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:55.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:55 smithi078 ceph-mon[99959]: from='client.65487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:34:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:56 smithi078 ceph-mon[91136]: pgmap v9448: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:56 smithi078 ceph-mon[99959]: pgmap v9448: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:56 smithi146 ceph-mon[103278]: pgmap v9448: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:34:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:58 smithi146 ceph-mon[103278]: pgmap v9449: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:58 smithi078 ceph-mon[99959]: pgmap v9449: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:58 smithi078 ceph-mon[91136]: pgmap v9449: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:34:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:00 smithi146 ceph-mon[103278]: pgmap v9450: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:00 smithi078 ceph-mon[99959]: pgmap v9450: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:00 smithi078 ceph-mon[91136]: pgmap v9450: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:35:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:35:01.722 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:01.468Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:01.722 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:01.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:01.722 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:01.468Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:02 smithi146 ceph-mon[103278]: pgmap v9451: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:02 smithi078 ceph-mon[91136]: pgmap v9451: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:02 smithi078 ceph-mon[99959]: pgmap v9451: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:04 smithi146 ceph-mon[103278]: pgmap v9452: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:04 smithi078 ceph-mon[91136]: pgmap v9452: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:04 smithi078 ceph-mon[99959]: pgmap v9452: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:06 smithi146 ceph-mon[103278]: pgmap v9453: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:06 smithi078 ceph-mon[91136]: pgmap v9453: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:06 smithi078 ceph-mon[99959]: pgmap v9453: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:08 smithi146 ceph-mon[103278]: pgmap v9454: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[91136]: pgmap v9454: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:08.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[99959]: pgmap v9454: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:08.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:08.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:10 smithi146 ceph-mon[103278]: pgmap v9455: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:10 smithi078 ceph-mon[91136]: pgmap v9455: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:10 smithi078 ceph-mon[99959]: pgmap v9455: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:35:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:35:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:11.468Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:11.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:11.469Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:12 smithi146 ceph-mon[103278]: pgmap v9456: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[91136]: pgmap v9456: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[99959]: pgmap v9456: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:14 smithi146 ceph-mon[103278]: pgmap v9457: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:14 smithi078 ceph-mon[91136]: pgmap v9457: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:14 smithi078 ceph-mon[99959]: pgmap v9457: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:15 smithi078 ceph-mon[91136]: pgmap v9458: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:15 smithi078 ceph-mon[99959]: pgmap v9458: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:16.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:15 smithi146 ceph-mon[103278]: pgmap v9458: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:16.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:18 smithi146 ceph-mon[103278]: pgmap v9459: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:18 smithi078 ceph-mon[91136]: pgmap v9459: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:18 smithi078 ceph-mon[99959]: pgmap v9459: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:20.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:20 smithi078 ceph-mon[91136]: pgmap v9460: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:20 smithi078 ceph-mon[99959]: pgmap v9460: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:20 smithi146 ceph-mon[103278]: pgmap v9460: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:35:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:35:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:21.469Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:21.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:21.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:22 smithi146 ceph-mon[103278]: pgmap v9461: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:22 smithi078 ceph-mon[91136]: pgmap v9461: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:22 smithi078 ceph-mon[99959]: pgmap v9461: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:24.277 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:35:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:24 smithi078 ceph-mon[91136]: pgmap v9462: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:24.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:24 smithi078 ceph-mon[99959]: pgmap v9462: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:24.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:24 smithi146 ceph-mon[103278]: pgmap v9462: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 5h - - 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 5h - - 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 17s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 12s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 17s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 12s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 17s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 17s ago 5h 516M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 5h 348M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:35:24.806 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 17s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 17s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 12s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 17s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 17s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 17s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 17s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 12s ago 5h 73.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 17s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 17s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:35:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:35:25.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:35:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:35:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:35:25.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:35:25.387 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:35:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:35:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:35:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:35:25.388 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:35:25.388 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:25 smithi078 ceph-mon[99959]: from='client.85418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:25.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:25 smithi078 ceph-mon[99959]: from='client.65505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:25.389 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3192428893' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:25.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:25 smithi078 ceph-mon[91136]: from='client.85418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:25.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:25 smithi078 ceph-mon[91136]: from='client.65505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:25.389 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3192428893' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:25.759 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:35:25.759 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:25.759 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:25.759 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:25.759 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:25 smithi146 ceph-mon[103278]: from='client.85418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:25 smithi146 ceph-mon[103278]: from='client.65505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3192428893' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:26 smithi146 ceph-mon[103278]: from='client.65511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:26 smithi146 ceph-mon[103278]: pgmap v9463: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:26 smithi146 ceph-mon[103278]: from='client.85442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3310889888' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[91136]: from='client.65511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[91136]: pgmap v9463: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[91136]: from='client.85442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3310889888' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[99959]: from='client.65511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[99959]: pgmap v9463: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[99959]: from='client.85442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3310889888' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:28 smithi146 ceph-mon[103278]: pgmap v9464: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[91136]: pgmap v9464: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[99959]: pgmap v9464: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:30 smithi146 ceph-mon[103278]: pgmap v9465: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:30 smithi078 ceph-mon[91136]: pgmap v9465: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:30 smithi078 ceph-mon[99959]: pgmap v9465: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:35:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:35:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:31.470Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:31.470Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:31.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:32 smithi146 ceph-mon[103278]: pgmap v9466: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[91136]: pgmap v9466: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[99959]: pgmap v9466: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:34.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:34 smithi146 ceph-mon[103278]: pgmap v9467: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:34 smithi078 ceph-mon[91136]: pgmap v9467: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:34 smithi078 ceph-mon[99959]: pgmap v9467: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:36 smithi146 ceph-mon[103278]: pgmap v9468: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:36.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:36 smithi078 ceph-mon[91136]: pgmap v9468: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:36.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:36.779 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:36 smithi078 ceph-mon[99959]: pgmap v9468: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:36.779 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:38 smithi146 ceph-mon[103278]: pgmap v9469: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:38 smithi078 ceph-mon[91136]: pgmap v9469: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:38 smithi078 ceph-mon[99959]: pgmap v9469: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:40 smithi146 ceph-mon[103278]: pgmap v9470: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:40 smithi078 ceph-mon[91136]: pgmap v9470: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:40 smithi078 ceph-mon[99959]: pgmap v9470: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:35:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:35:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:41.470Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:41.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:41.471Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:42 smithi146 ceph-mon[103278]: pgmap v9471: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:42 smithi078 ceph-mon[99959]: pgmap v9471: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:42 smithi078 ceph-mon[91136]: pgmap v9471: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:44 smithi146 ceph-mon[103278]: pgmap v9472: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:44 smithi078 ceph-mon[99959]: pgmap v9472: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:44 smithi078 ceph-mon[91136]: pgmap v9472: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:45 smithi078 ceph-mon[99959]: pgmap v9473: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:45 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:45 smithi078 ceph-mon[91136]: pgmap v9473: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:45 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:45 smithi146 ceph-mon[103278]: pgmap v9473: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:45 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:48 smithi146 ceph-mon[103278]: pgmap v9474: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[91136]: pgmap v9474: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[99959]: pgmap v9474: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:50 smithi078 ceph-mon[91136]: pgmap v9475: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:50 smithi078 ceph-mon[99959]: pgmap v9475: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:50 smithi146 ceph-mon[103278]: pgmap v9475: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:35:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:35:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:51.471Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:51.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:35:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:35:51.472Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:35:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:52 smithi078 ceph-mon[91136]: pgmap v9476: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:52 smithi078 ceph-mon[99959]: pgmap v9476: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:52 smithi146 ceph-mon[103278]: pgmap v9476: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:35:54.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:54 smithi146 ceph-mon[103278]: pgmap v9477: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:54 smithi078 ceph-mon[91136]: pgmap v9477: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:54 smithi078 ceph-mon[99959]: pgmap v9477: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:35:56.058 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 5h - - 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 5h - - 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 8s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 4s ago 5h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 8s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 4s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 8s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:35:56.592 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 8s ago 5h 515M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 4s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 8s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 8s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 4s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 8s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 8s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 8s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 8s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 4s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 4s ago 5h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:35:56.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 8s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:35:56.594 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:35:56.594 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 8s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:35:56.594 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:35:56.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:56 smithi078 ceph-mon[99959]: pgmap v9478: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:56.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:56.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:56 smithi078 ceph-mon[91136]: pgmap v9478: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:56.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:56.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:56 smithi146 ceph-mon[103278]: pgmap v9478: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:56.759 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:35:56.914 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:35:56.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:35:56.914 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:35:56.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:56.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:35:56.914 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:35:56.915 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:35:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:35:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:35:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:35:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:35:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:35:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:35:57.190 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[99959]: from='client.65535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[99959]: from='client.65541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[99959]: from='client.65547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1078436391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[91136]: from='client.65535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[91136]: from='client.65541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[91136]: from='client.65547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1078436391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:57.502 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:35:57.502 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:57.502 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:35:57.503 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:57.503 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:35:57.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:57 smithi146 ceph-mon[103278]: from='client.65535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:57 smithi146 ceph-mon[103278]: from='client.65541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:57 smithi146 ceph-mon[103278]: from='client.65547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:57.632 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1078436391' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:35:58.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:58 smithi078 ceph-mon[99959]: from='client.65559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:58.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:58 smithi078 ceph-mon[99959]: pgmap v9479: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:58.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4220804340' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:58.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:58 smithi078 ceph-mon[91136]: from='client.65559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:58.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:58 smithi078 ceph-mon[91136]: pgmap v9479: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:58.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4220804340' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:58 smithi146 ceph-mon[103278]: from='client.65559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:35:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:58 smithi146 ceph-mon[103278]: pgmap v9479: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:35:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4220804340' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:35:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:35:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:35:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:35:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:00 smithi146 ceph-mon[103278]: pgmap v9480: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:00 smithi078 ceph-mon[91136]: pgmap v9480: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:00 smithi078 ceph-mon[99959]: pgmap v9480: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:36:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:36:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:01.472Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:01.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:01.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:02 smithi146 ceph-mon[103278]: pgmap v9481: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:02 smithi078 ceph-mon[91136]: pgmap v9481: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:02 smithi078 ceph-mon[99959]: pgmap v9481: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:04 smithi146 ceph-mon[103278]: pgmap v9482: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:04 smithi078 ceph-mon[91136]: pgmap v9482: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:04 smithi078 ceph-mon[99959]: pgmap v9482: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:06 smithi146 ceph-mon[103278]: pgmap v9483: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:06 smithi078 ceph-mon[91136]: pgmap v9483: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:06 smithi078 ceph-mon[99959]: pgmap v9483: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:08 smithi146 ceph-mon[103278]: pgmap v9484: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[91136]: pgmap v9484: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[99959]: pgmap v9484: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:10 smithi146 ceph-mon[103278]: pgmap v9485: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:10 smithi078 ceph-mon[91136]: pgmap v9485: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:10 smithi078 ceph-mon[99959]: pgmap v9485: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:36:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:36:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:11.473Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:11.473Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:11.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:12 smithi146 ceph-mon[103278]: pgmap v9486: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:12 smithi078 ceph-mon[91136]: pgmap v9486: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:12 smithi078 ceph-mon[99959]: pgmap v9486: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:14 smithi146 ceph-mon[103278]: pgmap v9487: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:14 smithi078 ceph-mon[91136]: pgmap v9487: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:14 smithi078 ceph-mon[99959]: pgmap v9487: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:16 smithi146 ceph-mon[103278]: pgmap v9488: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:16 smithi078 ceph-mon[91136]: pgmap v9488: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:16 smithi078 ceph-mon[99959]: pgmap v9488: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:18 smithi146 ceph-mon[103278]: pgmap v9489: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:18 smithi078 ceph-mon[99959]: pgmap v9489: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:18 smithi078 ceph-mon[91136]: pgmap v9489: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:20 smithi146 ceph-mon[103278]: pgmap v9490: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:20 smithi078 ceph-mon[91136]: pgmap v9490: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:36:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:36:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:20 smithi078 ceph-mon[99959]: pgmap v9490: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:21.473Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:21.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:21.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:22.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:22 smithi078 ceph-mon[91136]: pgmap v9491: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:23.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:22 smithi078 ceph-mon[99959]: pgmap v9491: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:23.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:22 smithi146 ceph-mon[103278]: pgmap v9491: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:24.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:24 smithi146 ceph-mon[103278]: pgmap v9492: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:24 smithi078 ceph-mon[99959]: pgmap v9492: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:24 smithi078 ceph-mon[91136]: pgmap v9492: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:26 smithi078 ceph-mon[91136]: pgmap v9493: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:26 smithi078 ceph-mon[99959]: pgmap v9493: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:26 smithi146 ceph-mon[103278]: pgmap v9493: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:27.800 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 5h - - 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 5h - - 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 0s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 15s ago 5h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 0s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 15s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 0s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 0s ago 5h 517M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 15s ago 5h 346M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:36:28.339 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 0s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 0s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 15s ago 5h 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 0s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 0s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 0s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 0s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 15s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 15s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 15s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 15s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 15s ago 5h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 0s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 0s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:36:28.340 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:36:28.649 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:36:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:36:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:36:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:36:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:36:28.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:36:28.650 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[99959]: pgmap v9494: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[91136]: pgmap v9494: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:28.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:28.923 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:36:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:36:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:36:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:36:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:36:28.923 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:36:28.924 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:36:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:28 smithi146 ceph-mon[103278]: pgmap v9494: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:29.228 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:36:29.228 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:36:29.228 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:36:29.228 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:36:29.229 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:36:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:29 smithi146 ceph-mon[103278]: from='client.85490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:29 smithi146 ceph-mon[103278]: from='client.65577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:29 smithi146 ceph-mon[103278]: from='client.85502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3220501150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1616816915' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[99959]: from='client.85490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[99959]: from='client.65577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[99959]: from='client.85502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3220501150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1616816915' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[91136]: from='client.85490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[91136]: from='client.65577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[91136]: from='client.85502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3220501150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:36:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1616816915' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:36:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:30 smithi146 ceph-mon[103278]: from='client.65595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:30 smithi146 ceph-mon[103278]: pgmap v9495: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:36:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:36:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:30 smithi078 ceph-mon[91136]: from='client.65595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:30 smithi078 ceph-mon[91136]: pgmap v9495: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:30 smithi078 ceph-mon[99959]: from='client.65595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:36:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:30 smithi078 ceph-mon[99959]: pgmap v9495: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:31.473Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:31.474Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:31.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:32 smithi078 ceph-mon[91136]: pgmap v9496: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:32 smithi078 ceph-mon[99959]: pgmap v9496: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:32 smithi146 ceph-mon[103278]: pgmap v9496: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:33 smithi146 ceph-mon[103278]: pgmap v9497: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[91136]: pgmap v9497: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:33 smithi078 ceph-mon[99959]: pgmap v9497: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:36.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:36 smithi146 ceph-mon[103278]: pgmap v9498: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:36.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:36 smithi078 ceph-mon[99959]: pgmap v9498: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:36 smithi078 ceph-mon[91136]: pgmap v9498: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:38.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:38 smithi146 ceph-mon[103278]: pgmap v9499: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:38 smithi078 ceph-mon[99959]: pgmap v9499: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:38 smithi078 ceph-mon[91136]: pgmap v9499: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:40 smithi078 ceph-mon[91136]: pgmap v9500: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:40 smithi078 ceph-mon[99959]: pgmap v9500: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:40 smithi146 ceph-mon[103278]: pgmap v9500: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:36:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:36:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:41.474Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:41.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:41.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:42 smithi146 ceph-mon[103278]: pgmap v9501: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:42 smithi078 ceph-mon[99959]: pgmap v9501: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:42 smithi078 ceph-mon[91136]: pgmap v9501: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:44 smithi146 ceph-mon[103278]: pgmap v9502: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:44 smithi078 ceph-mon[99959]: pgmap v9502: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:44 smithi078 ceph-mon[91136]: pgmap v9502: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:46.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:46 smithi078 ceph-mon[91136]: pgmap v9503: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:46.619 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:46.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:46 smithi078 ceph-mon[99959]: pgmap v9503: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:46.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:46 smithi146 ceph-mon[103278]: pgmap v9503: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:48.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:48 smithi146 ceph-mon[103278]: pgmap v9504: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:48.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.748 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[99959]: pgmap v9504: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[91136]: pgmap v9504: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:50 smithi078 ceph-mon[91136]: pgmap v9505: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:50 smithi078 ceph-mon[99959]: pgmap v9505: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:50 smithi146 ceph-mon[103278]: pgmap v9505: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:36:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:36:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:51.475Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:51.475Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:36:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:36:51.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:36:52.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:52 smithi078 ceph-mon[99959]: pgmap v9506: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:52 smithi078 ceph-mon[91136]: pgmap v9506: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:52 smithi146 ceph-mon[103278]: pgmap v9506: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:53 smithi146 ceph-mon[103278]: pgmap v9507: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[91136]: pgmap v9507: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:36:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:53 smithi078 ceph-mon[99959]: pgmap v9507: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:36:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:56 smithi146 ceph-mon[103278]: pgmap v9508: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:56 smithi078 ceph-mon[91136]: pgmap v9508: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:56 smithi078 ceph-mon[99959]: pgmap v9508: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:36:58.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:58 smithi146 ceph-mon[103278]: pgmap v9509: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:58.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:58 smithi078 ceph-mon[91136]: pgmap v9509: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:58.629 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:58 smithi078 ceph-mon[99959]: pgmap v9509: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:36:59.532 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:36:59.632 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:59.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:59.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:00.078 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 5h - - 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 5h - - 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 11s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 7s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 11s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 7s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 11s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 11s ago 5h 517M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 7s ago 5h 349M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 11s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 11s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 7s ago 5h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 11s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:37:00.079 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 11s ago 5h 94.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 11s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 11s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 7s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 7s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 7s ago 5h 67.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 11s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 11s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:37:00.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:37:00.444 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:37:00.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:37:00.445 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:37:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:00 smithi078 ceph-mon[99959]: pgmap v9510: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:00.446 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:00 smithi078 ceph-mon[99959]: from='client.65607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:00 smithi078 ceph-mon[91136]: pgmap v9510: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:00.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:00 smithi078 ceph-mon[91136]: from='client.65607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:00.732 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:37:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:37:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:37:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:37:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:37:00.733 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:37:00.733 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:37:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:37:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:00 smithi146 ceph-mon[103278]: pgmap v9510: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:00 smithi146 ceph-mon[103278]: from='client.65607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.051 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:37:01.051 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:01.051 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:01.051 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:01.051 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:01.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[99959]: from='client.65613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[99959]: from='client.85538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2267911732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:01.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2653985730' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:01.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[91136]: from='client.65613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[91136]: from='client.85538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2267911732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:01.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2653985730' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:01.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:01.475Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:01.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:01.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:01.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:01.476Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:01 smithi146 ceph-mon[103278]: from='client.65613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:01 smithi146 ceph-mon[103278]: from='client.85538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2267911732' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2653985730' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:02 smithi078 ceph-mon[91136]: from='client.65631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:02 smithi078 ceph-mon[91136]: pgmap v9511: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:02 smithi078 ceph-mon[99959]: from='client.65631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:02.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:02 smithi078 ceph-mon[99959]: pgmap v9511: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:02 smithi146 ceph-mon[103278]: from='client.65631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:02 smithi146 ceph-mon[103278]: pgmap v9511: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:04 smithi146 ceph-mon[103278]: pgmap v9512: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:04 smithi078 ceph-mon[91136]: pgmap v9512: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:04 smithi078 ceph-mon[99959]: pgmap v9512: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:06 smithi146 ceph-mon[103278]: pgmap v9513: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:06 smithi078 ceph-mon[99959]: pgmap v9513: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:06 smithi078 ceph-mon[91136]: pgmap v9513: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[99959]: pgmap v9514: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[91136]: pgmap v9514: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:08 smithi146 ceph-mon[103278]: pgmap v9514: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:10 smithi146 ceph-mon[103278]: pgmap v9515: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:10 smithi078 ceph-mon[91136]: pgmap v9515: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:10 smithi078 ceph-mon[99959]: pgmap v9515: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:37:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:37:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:11.476Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:11.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:11.477Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:12 smithi146 ceph-mon[103278]: pgmap v9516: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:12 smithi078 ceph-mon[91136]: pgmap v9516: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:12 smithi078 ceph-mon[99959]: pgmap v9516: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:13 smithi146 ceph-mon[103278]: pgmap v9517: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[91136]: pgmap v9517: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[99959]: pgmap v9517: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:16 smithi146 ceph-mon[103278]: pgmap v9518: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:16 smithi078 ceph-mon[91136]: pgmap v9518: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:16 smithi078 ceph-mon[99959]: pgmap v9518: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:18 smithi146 ceph-mon[103278]: pgmap v9519: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:18 smithi078 ceph-mon[91136]: pgmap v9519: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:18 smithi078 ceph-mon[99959]: pgmap v9519: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:20 smithi146 ceph-mon[103278]: pgmap v9520: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:20.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:20 smithi078 ceph-mon[91136]: pgmap v9520: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:20.602 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:20 smithi078 ceph-mon[99959]: pgmap v9520: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:20.887 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:37:20] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:37:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:21.477Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:21.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:21.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:22 smithi146 ceph-mon[103278]: pgmap v9521: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:22 smithi078 ceph-mon[91136]: pgmap v9521: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:22 smithi078 ceph-mon[99959]: pgmap v9521: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:24 smithi146 ceph-mon[103278]: pgmap v9522: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:24 smithi078 ceph-mon[91136]: pgmap v9522: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:24 smithi078 ceph-mon[99959]: pgmap v9522: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:26 smithi146 ceph-mon[103278]: pgmap v9523: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:26 smithi078 ceph-mon[91136]: pgmap v9523: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:26 smithi078 ceph-mon[99959]: pgmap v9523: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:28 smithi146 ceph-mon[103278]: pgmap v9524: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:28 smithi078 ceph-mon[91136]: pgmap v9524: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:28 smithi078 ceph-mon[99959]: pgmap v9524: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:30 smithi146 ceph-mon[103278]: pgmap v9525: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:30 smithi078 ceph-mon[91136]: pgmap v9525: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:30 smithi078 ceph-mon[99959]: pgmap v9525: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:37:30] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:37:31.351 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:37:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:31.478Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:31.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:31.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:31.891 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 5h - - 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 18s ago 5h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 3s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 18s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 3s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 3s ago 5h 518M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 18s ago 5h 350M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 3s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:37:31.892 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 18s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 3s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 3s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 3s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 3s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 18s ago 5h 67.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 3s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:37:31.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:37:31.894 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 3s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:37:31.894 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:37:32.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:37:32.203 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:37:32.476 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:37:32.477 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:37:32.477 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:37:32.477 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:37:32.477 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:37:32.477 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:37:32.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[99959]: pgmap v9526: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:32.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[99959]: from='client.85562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:32.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[99959]: from='client.65649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:32.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1811022158' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:32.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[91136]: pgmap v9526: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:32.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[91136]: from='client.85562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:32.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[91136]: from='client.65649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:32.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1811022158' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:32 smithi146 ceph-mon[103278]: pgmap v9526: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:32 smithi146 ceph-mon[103278]: from='client.85562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:32 smithi146 ceph-mon[103278]: from='client.65649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1811022158' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:37:32.786 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:37:32.786 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:32.786 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:37:32.786 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:32.786 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:37:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:33 smithi146 ceph-mon[103278]: from='client.65655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:33 smithi146 ceph-mon[103278]: from='client.65667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2914615175' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[91136]: from='client.65655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[91136]: from='client.65667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2914615175' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[99959]: from='client.65655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[99959]: from='client.65667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2914615175' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:34 smithi146 ceph-mon[103278]: pgmap v9527: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:34 smithi078 ceph-mon[91136]: pgmap v9527: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:34 smithi078 ceph-mon[99959]: pgmap v9527: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:36 smithi146 ceph-mon[103278]: pgmap v9528: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:36 smithi078 ceph-mon[91136]: pgmap v9528: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:36 smithi078 ceph-mon[99959]: pgmap v9528: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:38.619 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:38 smithi146 ceph-mon[103278]: pgmap v9529: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:38 smithi078 ceph-mon[99959]: pgmap v9529: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:38 smithi078 ceph-mon[91136]: pgmap v9529: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:40.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:40 smithi146 ceph-mon[103278]: pgmap v9530: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:40 smithi078 ceph-mon[99959]: pgmap v9530: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:40 smithi078 ceph-mon[91136]: pgmap v9530: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:37:40] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:37:41.762 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:41.478Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:41.762 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:41.478Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:41.763 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:41.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:42 smithi146 ceph-mon[103278]: pgmap v9531: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:42 smithi078 ceph-mon[91136]: pgmap v9531: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:42 smithi078 ceph-mon[99959]: pgmap v9531: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:44 smithi146 ceph-mon[103278]: pgmap v9532: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:44 smithi078 ceph-mon[99959]: pgmap v9532: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:44 smithi078 ceph-mon[91136]: pgmap v9532: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:46 smithi146 ceph-mon[103278]: pgmap v9533: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:46 smithi078 ceph-mon[91136]: pgmap v9533: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:46 smithi078 ceph-mon[99959]: pgmap v9533: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:48 smithi146 ceph-mon[103278]: pgmap v9534: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:48 smithi078 ceph-mon[99959]: pgmap v9534: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:48 smithi078 ceph-mon[91136]: pgmap v9534: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:50 smithi146 ceph-mon[103278]: pgmap v9535: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:50.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:50 smithi078 ceph-mon[99959]: pgmap v9535: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:50 smithi078 ceph-mon[91136]: pgmap v9535: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:37:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:37:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:51.479Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:51.479Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:37:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:37:51.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:37:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:52 smithi146 ceph-mon[103278]: pgmap v9536: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:52 smithi078 ceph-mon[91136]: pgmap v9536: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:52 smithi078 ceph-mon[99959]: pgmap v9536: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:54.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.432 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:54 smithi146 ceph-mon[103278]: pgmap v9537: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[99959]: pgmap v9537: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:37:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:54 smithi078 ceph-mon[91136]: pgmap v9537: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:37:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:56 smithi146 ceph-mon[103278]: pgmap v9538: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:56 smithi078 ceph-mon[99959]: pgmap v9538: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:56 smithi078 ceph-mon[91136]: pgmap v9538: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:37:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:58 smithi146 ceph-mon[103278]: pgmap v9539: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:58 smithi078 ceph-mon[91136]: pgmap v9539: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:58 smithi078 ceph-mon[99959]: pgmap v9539: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:37:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:37:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:37:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:37:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:00.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:00 smithi146 ceph-mon[103278]: pgmap v9540: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:00.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:00 smithi078 ceph-mon[91136]: pgmap v9540: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:00.766 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:00 smithi078 ceph-mon[99959]: pgmap v9540: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:00.766 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:38:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:38:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:01.480Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:01.480Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:01.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:02.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:02 smithi078 ceph-mon[91136]: pgmap v9541: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:02 smithi146 ceph-mon[103278]: pgmap v9541: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:02 smithi078 ceph-mon[99959]: pgmap v9541: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:03.081 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:38:03.610 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:38:03.610 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 5h - - 2024-09-21T01:38:03.610 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 15s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 15s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 15s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 15s ago 5h 519M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 10s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 15s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 15s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 15s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 15s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 15s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 15s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:38:03.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 10s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 10s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 10s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 10s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 10s ago 5h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 15s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 15s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:38:03.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:03.924 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:38:03.925 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:38:04.193 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:38:04.193 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:38:04.193 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:38:04.193 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:38:04.193 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:38:04.193 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:38:04.194 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:38:04.504 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:38:04.504 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:38:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:38:04.504 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:38:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:38:04.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[99959]: from='client.65679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[99959]: pgmap v9542: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:04.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[99959]: from='client.85604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[99959]: from='client.65691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3423566131' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:04.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[91136]: from='client.65679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[91136]: pgmap v9542: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:04.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[91136]: from='client.85604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[91136]: from='client.65691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3423566131' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:04 smithi146 ceph-mon[103278]: from='client.65679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:04 smithi146 ceph-mon[103278]: pgmap v9542: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:04 smithi146 ceph-mon[103278]: from='client.85604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:04 smithi146 ceph-mon[103278]: from='client.65691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3423566131' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:05 smithi146 ceph-mon[103278]: from='client.65703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1693120856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:05 smithi078 ceph-mon[99959]: from='client.65703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1693120856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:05 smithi078 ceph-mon[91136]: from='client.65703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1693120856' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:06 smithi146 ceph-mon[103278]: pgmap v9543: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:06 smithi078 ceph-mon[99959]: pgmap v9543: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:06 smithi078 ceph-mon[91136]: pgmap v9543: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:08 smithi146 ceph-mon[103278]: pgmap v9544: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:08 smithi078 ceph-mon[91136]: pgmap v9544: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:08 smithi078 ceph-mon[99959]: pgmap v9544: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:10 smithi146 ceph-mon[103278]: pgmap v9545: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:10 smithi078 ceph-mon[99959]: pgmap v9545: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:10 smithi078 ceph-mon[91136]: pgmap v9545: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:38:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:38:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:11.480Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:11.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:11.481Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:12 smithi146 ceph-mon[103278]: pgmap v9546: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:12 smithi078 ceph-mon[91136]: pgmap v9546: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:12 smithi078 ceph-mon[99959]: pgmap v9546: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:14 smithi146 ceph-mon[103278]: pgmap v9547: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[99959]: pgmap v9547: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[91136]: pgmap v9547: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:16 smithi146 ceph-mon[103278]: pgmap v9548: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:16 smithi078 ceph-mon[91136]: pgmap v9548: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:16 smithi078 ceph-mon[99959]: pgmap v9548: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:16.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:18 smithi146 ceph-mon[103278]: pgmap v9549: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:18 smithi078 ceph-mon[91136]: pgmap v9549: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:18 smithi078 ceph-mon[99959]: pgmap v9549: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:20.675 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:20 smithi146 ceph-mon[103278]: pgmap v9550: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:20 smithi078 ceph-mon[91136]: pgmap v9550: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:20 smithi078 ceph-mon[99959]: pgmap v9550: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:38:20] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:38:21.768 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:21.482Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:21.768 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:21.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:21.768 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:21.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:22 smithi078 ceph-mon[91136]: pgmap v9551: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:38:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:22 smithi078 ceph-mon[99959]: pgmap v9551: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:38:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:22 smithi146 ceph-mon[103278]: pgmap v9551: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:38:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:24 smithi146 ceph-mon[103278]: pgmap v9552: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:24 smithi078 ceph-mon[91136]: pgmap v9552: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:24 smithi078 ceph-mon[99959]: pgmap v9552: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:25 smithi146 ceph-mon[103278]: pgmap v9553: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:25 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:25 smithi078 ceph-mon[91136]: pgmap v9553: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:25 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:25 smithi078 ceph-mon[99959]: pgmap v9553: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:25 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:28.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:28 smithi078 ceph-mon[99959]: pgmap v9554: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:28.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:28 smithi078 ceph-mon[91136]: pgmap v9554: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:28 smithi146 ceph-mon[103278]: pgmap v9554: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:30 smithi146 ceph-mon[103278]: pgmap v9555: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:30 smithi078 ceph-mon[99959]: pgmap v9555: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:30 smithi078 ceph-mon[91136]: pgmap v9555: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:38:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:38:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:31.482Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:31.482Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:31.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:32 smithi146 ceph-mon[103278]: pgmap v9556: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:32 smithi078 ceph-mon[91136]: pgmap v9556: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:32 smithi078 ceph-mon[99959]: pgmap v9556: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:34 smithi146 ceph-mon[103278]: pgmap v9557: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[99959]: pgmap v9557: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[91136]: pgmap v9557: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:34.798 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 5h - - 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 5h - - 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 6s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 1s ago 5h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 6s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 1s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 6s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 6s ago 5h 520M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:38:35.327 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 1s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 6s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 6s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 1s ago 5h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 6s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 6s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 6s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 6s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 1s ago 5h 70.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 6s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:38:35.328 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:38:35.329 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 6s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:38:35.329 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:38:35.640 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:38:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:38:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:38:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:38:35.640 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:38:35.641 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:38:35.908 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:38:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:38:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:38:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:38:35.909 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:38:35.909 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:38:36.219 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:38:36.220 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:38:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:38:36.220 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:38:36.220 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: from='client.65715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: from='client.85640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: from='client.65727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: pgmap v9558: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3307515961' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3562164659' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: from='client.65715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: from='client.85640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: from='client.65727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: pgmap v9558: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3307515961' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3562164659' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: from='client.65715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: from='client.85640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: from='client.65727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: pgmap v9558: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3307515961' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:38:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3562164659' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:38:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:37 smithi146 ceph-mon[103278]: from='client.65739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:37 smithi078 ceph-mon[91136]: from='client.65739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:37 smithi078 ceph-mon[99959]: from='client.65739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:38:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:38 smithi146 ceph-mon[103278]: pgmap v9559: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:38 smithi078 ceph-mon[91136]: pgmap v9559: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:38 smithi078 ceph-mon[99959]: pgmap v9559: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:40 smithi146 ceph-mon[103278]: pgmap v9560: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:40 smithi078 ceph-mon[91136]: pgmap v9560: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:40 smithi078 ceph-mon[99959]: pgmap v9560: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:38:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:38:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:41.483Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:41.483Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:41.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:42.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:42 smithi146 ceph-mon[103278]: pgmap v9561: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:42 smithi078 ceph-mon[99959]: pgmap v9561: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:42 smithi078 ceph-mon[91136]: pgmap v9561: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:44.628 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:44 smithi078 ceph-mon[99959]: pgmap v9562: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:44.628 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:44 smithi078 ceph-mon[91136]: pgmap v9562: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:44 smithi146 ceph-mon[103278]: pgmap v9562: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:46 smithi146 ceph-mon[103278]: pgmap v9563: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:46 smithi078 ceph-mon[91136]: pgmap v9563: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:46 smithi078 ceph-mon[99959]: pgmap v9563: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:48 smithi146 ceph-mon[103278]: pgmap v9564: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:48 smithi078 ceph-mon[91136]: pgmap v9564: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:48 smithi078 ceph-mon[99959]: pgmap v9564: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:49 smithi146 ceph-mon[103278]: pgmap v9565: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[99959]: pgmap v9565: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:49 smithi078 ceph-mon[91136]: pgmap v9565: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:38:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:38:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:51.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:51.484Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:38:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:38:51.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:38:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:52 smithi146 ceph-mon[103278]: pgmap v9566: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:52 smithi078 ceph-mon[91136]: pgmap v9566: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:52 smithi078 ceph-mon[99959]: pgmap v9566: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:54 smithi146 ceph-mon[103278]: pgmap v9567: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[99959]: pgmap v9567: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[91136]: pgmap v9567: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:38:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:38:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:56 smithi146 ceph-mon[103278]: pgmap v9568: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:56 smithi078 ceph-mon[99959]: pgmap v9568: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:56 smithi078 ceph-mon[91136]: pgmap v9568: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:38:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:58 smithi146 ceph-mon[103278]: pgmap v9569: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:58 smithi078 ceph-mon[99959]: pgmap v9569: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:58 smithi078 ceph-mon[91136]: pgmap v9569: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:38:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:00.700 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:00 smithi146 ceph-mon[103278]: pgmap v9570: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:01.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:00 smithi078 ceph-mon[99959]: pgmap v9570: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:00 smithi078 ceph-mon[91136]: pgmap v9570: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:39:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:39:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:01.484Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:01.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:01.485Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:02.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:02 smithi146 ceph-mon[103278]: pgmap v9571: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:02.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:02 smithi078 ceph-mon[99959]: pgmap v9571: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:02.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:02 smithi078 ceph-mon[91136]: pgmap v9571: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:04.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:04 smithi078 ceph-mon[99959]: pgmap v9572: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:04.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:04 smithi078 ceph-mon[91136]: pgmap v9572: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:04.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:04 smithi146 ceph-mon[103278]: pgmap v9572: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:06.511 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:39:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:06 smithi078 ceph-mon[99959]: pgmap v9573: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:06 smithi078 ceph-mon[91136]: pgmap v9573: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:06 smithi146 ceph-mon[103278]: pgmap v9573: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 5h - - 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 5h - - 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 18s ago 5h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 13s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 18s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:39:07.042 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 13s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 18s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 18s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 13s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 18s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 18s ago 5h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 13s ago 5h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 18s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 18s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 18s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 18s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 13s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 13s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 13s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:39:07.043 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 13s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:39:07.044 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 13s ago 5h 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:39:07.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 18s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:39:07.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:39:07.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 18s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:39:07.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 13s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:39:07.352 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:39:07.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:07.353 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:39:07.353 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:39:07.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:39:07.353 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:39:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:39:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:39:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:39:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:39:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:39:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:39:07.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:39:07.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:07 smithi078 ceph-mon[99959]: from='client.85670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:07.625 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1135055704' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:07.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:07 smithi078 ceph-mon[91136]: from='client.85670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:07.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1135055704' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:07.931 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:39:07.931 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:07.931 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:07.931 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:07 smithi146 ceph-mon[103278]: from='client.85670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1135055704' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[99959]: from='client.65757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[99959]: from='client.85682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[99959]: pgmap v9574: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[99959]: from='client.85694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2853648663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[91136]: from='client.65757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[91136]: from='client.85682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[91136]: pgmap v9574: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[91136]: from='client.85694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2853648663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:08 smithi146 ceph-mon[103278]: from='client.65757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:08 smithi146 ceph-mon[103278]: from='client.85682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:08 smithi146 ceph-mon[103278]: pgmap v9574: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:08 smithi146 ceph-mon[103278]: from='client.85694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2853648663' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:09 smithi146 ceph-mon[103278]: pgmap v9575: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[91136]: pgmap v9575: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:09 smithi078 ceph-mon[99959]: pgmap v9575: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:39:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T01:39:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:11.485Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:11.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:11.486Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:12 smithi146 ceph-mon[103278]: pgmap v9576: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:12 smithi078 ceph-mon[91136]: pgmap v9576: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:12 smithi078 ceph-mon[99959]: pgmap v9576: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:14 smithi146 ceph-mon[103278]: pgmap v9577: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[91136]: pgmap v9577: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[99959]: pgmap v9577: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:16 smithi146 ceph-mon[103278]: pgmap v9578: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:16 smithi078 ceph-mon[91136]: pgmap v9578: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:16 smithi078 ceph-mon[99959]: pgmap v9578: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:18 smithi146 ceph-mon[103278]: pgmap v9579: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:18 smithi078 ceph-mon[91136]: pgmap v9579: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:18 smithi078 ceph-mon[99959]: pgmap v9579: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:20 smithi146 ceph-mon[103278]: pgmap v9580: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:20 smithi078 ceph-mon[99959]: pgmap v9580: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:39:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:39:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:20 smithi078 ceph-mon[91136]: pgmap v9580: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:21.486Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:21.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:21.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:22 smithi146 ceph-mon[103278]: pgmap v9581: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:22 smithi078 ceph-mon[91136]: pgmap v9581: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:22 smithi078 ceph-mon[99959]: pgmap v9581: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:24 smithi146 ceph-mon[103278]: pgmap v9582: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:24 smithi078 ceph-mon[99959]: pgmap v9582: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:24 smithi078 ceph-mon[91136]: pgmap v9582: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:26 smithi146 ceph-mon[103278]: pgmap v9583: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:26 smithi078 ceph-mon[91136]: pgmap v9583: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:26 smithi078 ceph-mon[99959]: pgmap v9583: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:28 smithi146 ceph-mon[103278]: pgmap v9584: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:28 smithi078 ceph-mon[91136]: pgmap v9584: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:28 smithi078 ceph-mon[99959]: pgmap v9584: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:30 smithi146 ceph-mon[103278]: pgmap v9585: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:30 smithi078 ceph-mon[99959]: pgmap v9585: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:30 smithi078 ceph-mon[91136]: pgmap v9585: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:39:30] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T01:39:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:31.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:31.487Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:31.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:32 smithi146 ceph-mon[103278]: pgmap v9586: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:32 smithi078 ceph-mon[91136]: pgmap v9586: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:32 smithi078 ceph-mon[99959]: pgmap v9586: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:34 smithi146 ceph-mon[103278]: pgmap v9587: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[91136]: pgmap v9587: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[99959]: pgmap v9587: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:36 smithi146 ceph-mon[103278]: pgmap v9588: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:36 smithi078 ceph-mon[99959]: pgmap v9588: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:36 smithi078 ceph-mon[91136]: pgmap v9588: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:38.227 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:39:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:38 smithi078 ceph-mon[99959]: pgmap v9589: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:38 smithi078 ceph-mon[91136]: pgmap v9589: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 5h - - 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 5h - - 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 9s ago 5h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 4s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 9s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 4s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 9s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 9s ago 5h 520M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 4s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 9s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 9s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:39:38.753 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 4s ago 5h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 9s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 9s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 9s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 9s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 4s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 4s ago 5h 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 9s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 9s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:39:38.754 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:39:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:38 smithi146 ceph-mon[103278]: pgmap v9589: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:39:39.066 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:39:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:39:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:39:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:39:39.067 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:39:39.337 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:39:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:39:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:39:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:39:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:39:39.338 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:39:39.647 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:39:39.647 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:39:39.647 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:39:39.648 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:39 smithi078 ceph-mon[99959]: from='client.65787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:39.648 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:39 smithi078 ceph-mon[99959]: from='client.65793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:39.648 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/547628277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:39.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:39 smithi078 ceph-mon[91136]: from='client.65787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:39.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:39 smithi078 ceph-mon[91136]: from='client.65793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:39.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/547628277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:39 smithi146 ceph-mon[103278]: from='client.65787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:39 smithi146 ceph-mon[103278]: from='client.65793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/547628277' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:39:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:40 smithi146 ceph-mon[103278]: from='client.85718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:40 smithi146 ceph-mon[103278]: from='client.85730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:40 smithi146 ceph-mon[103278]: pgmap v9590: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3025198489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:39:40] "GET /metrics HTTP/1.1" 200 39012 "" "Prometheus/2.43.0" 2024-09-21T01:39:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[99959]: from='client.85718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[99959]: from='client.85730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[99959]: pgmap v9590: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3025198489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[91136]: from='client.85718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[91136]: from='client.85730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:39:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[91136]: pgmap v9590: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3025198489' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:39:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:41.487Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:41.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:41.488Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:42.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:42 smithi146 ceph-mon[103278]: pgmap v9591: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:42 smithi078 ceph-mon[91136]: pgmap v9591: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:42 smithi078 ceph-mon[99959]: pgmap v9591: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:44.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:44 smithi078 ceph-mon[91136]: pgmap v9592: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:44.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:44.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:44 smithi078 ceph-mon[99959]: pgmap v9592: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:44.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:44 smithi146 ceph-mon[103278]: pgmap v9592: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:46 smithi146 ceph-mon[103278]: pgmap v9593: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:46 smithi078 ceph-mon[91136]: pgmap v9593: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:46 smithi078 ceph-mon[99959]: pgmap v9593: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:48 smithi146 ceph-mon[103278]: pgmap v9594: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:48 smithi078 ceph-mon[99959]: pgmap v9594: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:48 smithi078 ceph-mon[91136]: pgmap v9594: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:50 smithi146 ceph-mon[103278]: pgmap v9595: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[91136]: pgmap v9595: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-mon[99959]: pgmap v9595: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:39:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:39:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:51.488Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:51.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:39:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:39:51.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:39:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:52 smithi146 ceph-mon[103278]: pgmap v9596: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:52 smithi078 ceph-mon[99959]: pgmap v9596: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:52 smithi078 ceph-mon[91136]: pgmap v9596: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:54.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:54 smithi146 ceph-mon[103278]: pgmap v9597: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:54.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[99959]: pgmap v9597: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[91136]: pgmap v9597: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:39:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:39:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:56 smithi146 ceph-mon[103278]: pgmap v9598: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:56 smithi078 ceph-mon[99959]: pgmap v9598: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:56 smithi078 ceph-mon[91136]: pgmap v9598: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:39:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:58 smithi146 ceph-mon[103278]: pgmap v9599: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:58 smithi078 ceph-mon[99959]: pgmap v9599: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:58 smithi078 ceph-mon[91136]: pgmap v9599: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:39:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:39:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:39:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:39:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T01:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:00 smithi146 ceph-mon[103278]: pgmap v9600: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:40:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:40:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:00 smithi078 ceph-mon[91136]: pgmap v9600: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:00 smithi078 ceph-mon[99959]: pgmap v9600: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:01.489Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:01.489Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:01.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:02.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:02 smithi078 ceph-mon[99959]: pgmap v9601: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:02.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:02 smithi078 ceph-mon[91136]: pgmap v9601: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:02 smithi146 ceph-mon[103278]: pgmap v9601: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:04 smithi146 ceph-mon[103278]: pgmap v9602: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:04 smithi078 ceph-mon[99959]: pgmap v9602: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:04 smithi078 ceph-mon[91136]: pgmap v9602: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:06.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:06 smithi078 ceph-mon[99959]: pgmap v9603: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:06.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:06.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:06 smithi078 ceph-mon[91136]: pgmap v9603: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:06.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:06 smithi146 ceph-mon[103278]: pgmap v9603: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:08 smithi146 ceph-mon[103278]: pgmap v9604: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:08 smithi078 ceph-mon[91136]: pgmap v9604: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:08 smithi078 ceph-mon[99959]: pgmap v9604: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:09.932 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:40:10.458 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:40:10.458 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 5h - - 2024-09-21T01:40:10.458 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 5h - - 2024-09-21T01:40:10.458 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 1s ago 5h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 16s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 1s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 16s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 1s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 1s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 16s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 1s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 1s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 16s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 1s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 1s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 1s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:40:10.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 1s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 16s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 16s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 16s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 16s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 1s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 1s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:40:10.460 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 16s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:40:10.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[99959]: pgmap v9605: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-mon[91136]: pgmap v9605: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:10 smithi146 ceph-mon[103278]: pgmap v9605: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:40:10.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:10.768 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:40:10.768 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:40:10.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:40:10.768 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:40:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:40:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:40:11.039 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:40:11.040 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:40:11.040 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:40:11.347 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:11.347 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:11.347 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:11 smithi146 ceph-mon[103278]: from='client.85742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:11 smithi146 ceph-mon[103278]: from='client.85748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:11 smithi146 ceph-mon[103278]: from='client.65835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2954432015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2396627646' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[91136]: from='client.85742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[91136]: from='client.85748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[91136]: from='client.65835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2954432015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2396627646' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:11.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[99959]: from='client.85742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[99959]: from='client.85748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[99959]: from='client.65835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2954432015' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2396627646' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:11.490Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:11.490Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:11.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:12 smithi146 ceph-mon[103278]: from='client.85766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:12 smithi146 ceph-mon[103278]: pgmap v9606: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:12 smithi078 ceph-mon[99959]: from='client.85766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:12 smithi078 ceph-mon[99959]: pgmap v9606: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:12 smithi078 ceph-mon[91136]: from='client.85766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:12 smithi078 ceph-mon[91136]: pgmap v9606: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:14 smithi146 ceph-mon[103278]: pgmap v9607: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[99959]: pgmap v9607: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[91136]: pgmap v9607: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:15 smithi146 ceph-mon[103278]: pgmap v9608: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:15 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:15 smithi078 ceph-mon[91136]: pgmap v9608: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:15 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:15 smithi078 ceph-mon[99959]: pgmap v9608: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:15 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:18 smithi146 ceph-mon[103278]: pgmap v9609: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:18 smithi078 ceph-mon[99959]: pgmap v9609: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:18 smithi078 ceph-mon[91136]: pgmap v9609: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:20 smithi146 ceph-mon[103278]: pgmap v9610: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:20 smithi078 ceph-mon[91136]: pgmap v9610: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:20 smithi078 ceph-mon[99959]: pgmap v9610: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:40:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:40:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:21.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:21.491Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:21.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:22.701 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:22 smithi146 ceph-mon[103278]: pgmap v9611: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:22 smithi078 ceph-mon[91136]: pgmap v9611: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:22 smithi078 ceph-mon[99959]: pgmap v9611: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:24.532 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:24 smithi146 ceph-mon[103278]: pgmap v9612: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:24 smithi078 ceph-mon[99959]: pgmap v9612: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:24 smithi078 ceph-mon[91136]: pgmap v9612: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:26 smithi146 ceph-mon[103278]: pgmap v9613: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:26 smithi078 ceph-mon[91136]: pgmap v9613: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:26 smithi078 ceph-mon[99959]: pgmap v9613: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:27.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:28 smithi146 ceph-mon[103278]: pgmap v9614: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:28 smithi078 ceph-mon[91136]: pgmap v9614: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:28 smithi078 ceph-mon[99959]: pgmap v9614: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:30 smithi146 ceph-mon[103278]: pgmap v9615: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:40:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:40:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[99959]: pgmap v9615: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:30.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[91136]: pgmap v9615: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:31.491Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:31.492Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:31.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:32 smithi146 ceph-mon[103278]: pgmap v9616: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:32 smithi078 ceph-mon[91136]: pgmap v9616: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:32 smithi078 ceph-mon[99959]: pgmap v9616: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:34 smithi146 ceph-mon[103278]: pgmap v9617: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[91136]: pgmap v9617: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[99959]: pgmap v9617: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:36 smithi146 ceph-mon[103278]: pgmap v9618: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:36 smithi078 ceph-mon[99959]: pgmap v9618: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:36 smithi078 ceph-mon[91136]: pgmap v9618: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:38 smithi146 ceph-mon[103278]: pgmap v9619: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:38 smithi078 ceph-mon[91136]: pgmap v9619: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:38 smithi078 ceph-mon[99959]: pgmap v9619: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:40 smithi078 ceph-mon[91136]: pgmap v9620: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:40 smithi078 ceph-mon[99959]: pgmap v9620: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:40:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:40:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:40 smithi146 ceph-mon[103278]: pgmap v9620: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:41.644 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:40:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:41.492Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:41.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:41.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:40:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 5h - - 2024-09-21T01:40:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 5h - - 2024-09-21T01:40:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 12s ago 5h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:40:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 7s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:40:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 12s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 7s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 12s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 12s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 7s ago 5h 351M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 12s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 12s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 7s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 12s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 12s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 12s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 12s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 7s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 7s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:40:42.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:40:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 7s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:40:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 12s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:40:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:40:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 12s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:40:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:40:42.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:40:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:40:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:40:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:40:42.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:40:42.479 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:40:42.748 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:40:42.749 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:40:42.749 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:40:42.749 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:40:42.749 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:40:42.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:42 smithi078 ceph-mon[99959]: pgmap v9621: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:42.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:42 smithi078 ceph-mon[99959]: from='client.65859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:42.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1897929162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:42.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:42 smithi078 ceph-mon[91136]: pgmap v9621: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:42.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:42 smithi078 ceph-mon[91136]: from='client.65859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:42.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1897929162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:42 smithi146 ceph-mon[103278]: pgmap v9621: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:42 smithi146 ceph-mon[103278]: from='client.65859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1897929162' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:40:43.058 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:40:43.059 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:40:43.059 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:40:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[91136]: from='client.65865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[91136]: from='client.85790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3853480544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[99959]: from='client.65865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[99959]: from='client.85790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3853480544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:43.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:43 smithi146 ceph-mon[103278]: from='client.65865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:43 smithi146 ceph-mon[103278]: from='client.85790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:43.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3853480544' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:40:43.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:44 smithi146 ceph-mon[103278]: from='client.85802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:44 smithi146 ceph-mon[103278]: pgmap v9622: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:44 smithi078 ceph-mon[99959]: from='client.85802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:44 smithi078 ceph-mon[99959]: pgmap v9622: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:44 smithi078 ceph-mon[91136]: from='client.85802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:40:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:44 smithi078 ceph-mon[91136]: pgmap v9622: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:46.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:46 smithi078 ceph-mon[99959]: pgmap v9623: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:46.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:46 smithi078 ceph-mon[91136]: pgmap v9623: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:46 smithi146 ceph-mon[103278]: pgmap v9623: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:47.827 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:48.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:48 smithi078 ceph-mon[91136]: pgmap v9624: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:48.925 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:48 smithi078 ceph-mon[99959]: pgmap v9624: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:48 smithi146 ceph-mon[103278]: pgmap v9624: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:50 smithi146 ceph-mon[103278]: pgmap v9625: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:40:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:40:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[99959]: pgmap v9625: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[91136]: pgmap v9625: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:40:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:51.493Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:51.493Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:40:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:40:51.494Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:40:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:52 smithi146 ceph-mon[103278]: pgmap v9626: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:52 smithi078 ceph-mon[99959]: pgmap v9626: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:52 smithi078 ceph-mon[91136]: pgmap v9626: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:40:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:54 smithi146 ceph-mon[103278]: pgmap v9627: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:40:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[99959]: pgmap v9627: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:40:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[91136]: pgmap v9627: 161 pgs: 161 active+clean; 585 KiB data, 305 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:40:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:40:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:56 smithi146 ceph-mon[103278]: pgmap v9628: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:40:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:56 smithi078 ceph-mon[99959]: pgmap v9628: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:40:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:56 smithi078 ceph-mon[91136]: pgmap v9628: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T01:40:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:40:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:58 smithi146 ceph-mon[103278]: pgmap v9629: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:40:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:58 smithi078 ceph-mon[99959]: pgmap v9629: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:40:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:58 smithi078 ceph-mon[91136]: pgmap v9629: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:40:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:00 smithi146 ceph-mon[103278]: pgmap v9630: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:41:01.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:00 smithi078 ceph-mon[99959]: pgmap v9630: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:41:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:00 smithi078 ceph-mon[91136]: pgmap v9630: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:41:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:41:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:41:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:01.494Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:01.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:01.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:02 smithi146 ceph-mon[103278]: pgmap v9631: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:41:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:02 smithi078 ceph-mon[91136]: pgmap v9631: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:41:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:02 smithi078 ceph-mon[99959]: pgmap v9631: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:41:04.936 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:04 smithi146 ceph-mon[103278]: pgmap v9632: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:41:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:04 smithi078 ceph-mon[91136]: pgmap v9632: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:41:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:04 smithi078 ceph-mon[99959]: pgmap v9632: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:41:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:06 smithi146 ceph-mon[103278]: pgmap v9633: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T01:41:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:06 smithi078 ceph-mon[91136]: pgmap v9633: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T01:41:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:06 smithi078 ceph-mon[99959]: pgmap v9633: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T01:41:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:08.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:08 smithi078 ceph-mon[99959]: pgmap v9634: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:41:08.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:08 smithi078 ceph-mon[91136]: pgmap v9634: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:41:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:08 smithi146 ceph-mon[103278]: pgmap v9634: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:41:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:10 smithi146 ceph-mon[103278]: pgmap v9635: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:41:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:41:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:41:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[91136]: pgmap v9635: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[99959]: pgmap v9635: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:11.495Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:11.495Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:11.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:12 smithi146 ceph-mon[103278]: pgmap v9636: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:41:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:12 smithi078 ceph-mon[91136]: pgmap v9636: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:41:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:12 smithi078 ceph-mon[99959]: pgmap v9636: 161 pgs: 161 active+clean; 585 KiB data, 322 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:41:13.345 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 5h - - 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 5h - - 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 4s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 19s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 4s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 19s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 4s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 4s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:41:13.873 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 19s ago 5h 352M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 4s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 4s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 19s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 4s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 4s ago 5h 94.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 4s ago 5h 79.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 4s ago 5h 94.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 19s ago 5h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 19s ago 5h 85.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 19s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 19s ago 5h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 19s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 4s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:41:13.874 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 4s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:41:13.875 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 19s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:41:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:14.186 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:41:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:41:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:41:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:41:14.186 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:41:14.187 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:41:14.453 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:41:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:41:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:41:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:41:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:41:14.454 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:41:14.763 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:14.764 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:14.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: from='client.85814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:14.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: pgmap v9637: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T01:41:14.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: from='client.65901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:14.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2478834297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:14.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: from='client.85814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: pgmap v9637: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: from='client.65901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2478834297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.765 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: from='client.85814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: pgmap v9637: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: from='client.65901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2478834297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:15 smithi146 ceph-mon[103278]: from='client.85826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:15 smithi146 ceph-mon[103278]: from='client.65919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1896212958' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:15 smithi078 ceph-mon[91136]: from='client.85826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:15 smithi078 ceph-mon[91136]: from='client.65919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1896212958' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:15 smithi078 ceph-mon[99959]: from='client.85826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:15 smithi078 ceph-mon[99959]: from='client.65919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1896212958' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:16 smithi146 ceph-mon[103278]: pgmap v9638: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:41:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:16 smithi078 ceph-mon[99959]: pgmap v9638: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:41:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:16 smithi078 ceph-mon[91136]: pgmap v9638: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T01:41:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:17 smithi146 ceph-mon[103278]: pgmap v9639: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T01:41:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:17 smithi078 ceph-mon[99959]: pgmap v9639: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T01:41:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:17 smithi078 ceph-mon[91136]: pgmap v9639: 161 pgs: 161 active+clean; 585 KiB data, 326 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T01:41:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:20 smithi146 ceph-mon[103278]: pgmap v9640: 161 pgs: 161 active+clean; 585 KiB data, 330 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:41:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:20 smithi078 ceph-mon[99959]: pgmap v9640: 161 pgs: 161 active+clean; 585 KiB data, 330 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:41:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:20 smithi078 ceph-mon[91136]: pgmap v9640: 161 pgs: 161 active+clean; 585 KiB data, 330 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:41:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:41:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:41:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:21.496Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:21.496Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:21.497Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:22 smithi146 ceph-mon[103278]: pgmap v9641: 161 pgs: 161 active+clean; 585 KiB data, 330 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T01:41:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:22 smithi078 ceph-mon[99959]: pgmap v9641: 161 pgs: 161 active+clean; 585 KiB data, 330 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T01:41:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:22 smithi078 ceph-mon[91136]: pgmap v9641: 161 pgs: 161 active+clean; 585 KiB data, 330 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T01:41:24.685 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:24 smithi146 ceph-mon[103278]: pgmap v9642: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:41:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:24 smithi078 ceph-mon[91136]: pgmap v9642: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:41:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:24 smithi078 ceph-mon[99959]: pgmap v9642: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:41:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:26 smithi146 ceph-mon[103278]: pgmap v9643: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:41:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:26 smithi078 ceph-mon[99959]: pgmap v9643: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:41:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:26 smithi078 ceph-mon[91136]: pgmap v9643: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T01:41:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:28 smithi146 ceph-mon[103278]: pgmap v9644: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:41:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:28 smithi078 ceph-mon[91136]: pgmap v9644: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:41:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:28 smithi078 ceph-mon[99959]: pgmap v9644: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:41:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:30 smithi146 ceph-mon[103278]: pgmap v9645: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:41:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:41:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[99959]: pgmap v9645: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[91136]: pgmap v9645: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:30.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:31.497Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:31.497Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:31.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:32 smithi146 ceph-mon[103278]: pgmap v9646: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:41:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:32 smithi078 ceph-mon[91136]: pgmap v9646: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:41:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:32 smithi078 ceph-mon[99959]: pgmap v9646: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:41:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:34 smithi146 ceph-mon[103278]: pgmap v9647: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:41:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:34 smithi078 ceph-mon[99959]: pgmap v9647: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:41:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:34 smithi078 ceph-mon[91136]: pgmap v9647: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:41:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.041 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.042 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:36.043 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:36 smithi146 ceph-mon[103278]: pgmap v9648: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:36 smithi078 ceph-mon[99959]: pgmap v9648: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:36 smithi078 ceph-mon[91136]: pgmap v9648: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:38 smithi146 ceph-mon[103278]: pgmap v9649: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:38 smithi078 ceph-mon[91136]: pgmap v9649: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:38 smithi078 ceph-mon[99959]: pgmap v9649: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:40 smithi146 ceph-mon[103278]: pgmap v9650: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:40 smithi078 ceph-mon[99959]: pgmap v9650: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:40 smithi078 ceph-mon[91136]: pgmap v9650: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:41:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:41:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:41.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:41.498Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:41.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:42 smithi146 ceph-mon[103278]: pgmap v9651: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:42 smithi078 ceph-mon[91136]: pgmap v9651: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:42 smithi078 ceph-mon[99959]: pgmap v9651: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:44.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:44 smithi146 ceph-mon[103278]: pgmap v9652: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:44 smithi078 ceph-mon[91136]: pgmap v9652: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:44 smithi078 ceph-mon[99959]: pgmap v9652: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:45.065 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:41:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:41:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 5h - - 2024-09-21T01:41:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T01:41:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 15s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:41:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:41:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 15s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 15s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 15s ago 5h 521M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 10s ago 5h 353M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 15s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 15s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 15s ago 5h 90.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 15s ago 5h 95.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:41:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 15s ago 5h 80.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:41:45.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 15s ago 5h 94.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:41:45.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 10s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:41:45.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 10s ago 5h 85.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:41:45.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 10s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:41:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 10s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:41:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 10s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:41:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 15s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:41:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:41:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 15s ago 5h 90.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:41:45.593 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 10s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:41:45.901 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:41:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:41:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:41:45.903 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:41:46.170 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:41:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:41:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:41:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:41:46.171 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:41:46.478 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:41:46.478 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:46.478 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:41:46.478 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:46.478 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:41:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[91136]: from='client.65931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[91136]: from='client.65937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[91136]: pgmap v9653: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[91136]: from='client.85862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1502657528' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1561733979' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[99959]: from='client.65931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[99959]: from='client.65937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[99959]: pgmap v9653: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[99959]: from='client.85862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1502657528' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1561733979' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:46.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:46 smithi146 ceph-mon[103278]: from='client.65931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:46 smithi146 ceph-mon[103278]: from='client.65937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:46 smithi146 ceph-mon[103278]: pgmap v9653: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:46.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:46 smithi146 ceph-mon[103278]: from='client.85862 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:46.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1502657528' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:41:46.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1561733979' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:41:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:47 smithi146 ceph-mon[103278]: from='client.65955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:47 smithi078 ceph-mon[91136]: from='client.65955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:47 smithi078 ceph-mon[99959]: from='client.65955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:41:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:48 smithi078 ceph-mon[91136]: pgmap v9654: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:48 smithi078 ceph-mon[99959]: pgmap v9654: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:48 smithi146 ceph-mon[103278]: pgmap v9654: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:50 smithi146 ceph-mon[103278]: pgmap v9655: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:41:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:41:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[91136]: pgmap v9655: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[99959]: pgmap v9655: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:51.498Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:51.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:41:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:41:51.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:41:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:52 smithi078 ceph-mon[91136]: pgmap v9656: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:52 smithi078 ceph-mon[99959]: pgmap v9656: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:52 smithi146 ceph-mon[103278]: pgmap v9656: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:54 smithi146 ceph-mon[103278]: pgmap v9657: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:54 smithi078 ceph-mon[91136]: pgmap v9657: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:54 smithi078 ceph-mon[99959]: pgmap v9657: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:41:55.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:55.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:55.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:55.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:55 smithi146 ceph-mon[103278]: pgmap v9658: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[91136]: pgmap v9658: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:41:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:55 smithi078 ceph-mon[99959]: pgmap v9658: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:41:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:57 smithi146 ceph-mon[103278]: pgmap v9659: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:57 smithi078 ceph-mon[91136]: pgmap v9659: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:57 smithi078 ceph-mon[99959]: pgmap v9659: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:41:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:41:59 smithi078 ceph-mon[91136]: pgmap v9660: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:41:59 smithi078 ceph-mon[99959]: pgmap v9660: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:41:59 smithi146 ceph-mon[103278]: pgmap v9660: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:42:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:42:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:01.499Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:01.499Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:01.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:02.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:02 smithi078 ceph-mon[91136]: pgmap v9661: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:02.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:02 smithi078 ceph-mon[99959]: pgmap v9661: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:02.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:02 smithi146 ceph-mon[103278]: pgmap v9661: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:04 smithi146 ceph-mon[103278]: pgmap v9662: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:04 smithi078 ceph-mon[91136]: pgmap v9662: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:04.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:04 smithi078 ceph-mon[99959]: pgmap v9662: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:06.739 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:06 smithi146 ceph-mon[103278]: pgmap v9663: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:06 smithi078 ceph-mon[91136]: pgmap v9663: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:06 smithi078 ceph-mon[99959]: pgmap v9663: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:07.682 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:07.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:08 smithi146 ceph-mon[103278]: pgmap v9664: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:08 smithi078 ceph-mon[91136]: pgmap v9664: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:08 smithi078 ceph-mon[99959]: pgmap v9664: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:10.736 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:42:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:42:10.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[91136]: pgmap v9665: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:10.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[99959]: pgmap v9665: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:10.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:10 smithi146 ceph-mon[103278]: pgmap v9665: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:11.724 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:11.499Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:11.724 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:11.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:11.724 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:11.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:12 smithi146 ceph-mon[103278]: pgmap v9666: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:12 smithi078 ceph-mon[99959]: pgmap v9666: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:12 smithi078 ceph-mon[91136]: pgmap v9666: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:14 smithi146 ceph-mon[103278]: pgmap v9667: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:14 smithi078 ceph-mon[91136]: pgmap v9667: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:14 smithi078 ceph-mon[99959]: pgmap v9667: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[99959]: pgmap v9668: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:15 smithi078 ceph-mon[91136]: pgmap v9668: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:15 smithi146 ceph-mon[103278]: pgmap v9668: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:16.776 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:42:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:17.301 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:42:17.301 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 5h - - 2024-09-21T01:42:17.301 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 5h - - 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 7s ago 5h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 2s ago 5h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 7s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 7s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 7s ago 5h 522M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 5h 354M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 7s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 7s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 2s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 7s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 7s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 7s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:42:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 7s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 2s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 2s ago 5h 85.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 2s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 2s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 7s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 7s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:42:17.303 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 2s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:42:17.610 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:42:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:42:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:42:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:42:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:42:17.611 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:42:17.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:42:17.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:42:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:17 smithi146 ceph-mon[103278]: from='client.65967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:17 smithi146 ceph-mon[103278]: from='client.65973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:17 smithi146 ceph-mon[103278]: from='client.85898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:17 smithi146 ceph-mon[103278]: pgmap v9669: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2226435515' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:18.186 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:42:18.186 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:18.187 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:18.187 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:18.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[91136]: from='client.65967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[91136]: from='client.65973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[91136]: from='client.85898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[91136]: pgmap v9669: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:18.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2226435515' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:18.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[99959]: from='client.65967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[99959]: from='client.65973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[99959]: from='client.85898 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:18.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[99959]: pgmap v9669: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:18.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2226435515' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:18 smithi146 ceph-mon[103278]: from='client.65991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3556811628' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:18 smithi078 ceph-mon[91136]: from='client.65991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3556811628' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:18 smithi078 ceph-mon[99959]: from='client.65991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3556811628' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:19 smithi146 ceph-mon[103278]: pgmap v9670: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:20.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:19 smithi078 ceph-mon[99959]: pgmap v9670: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:19 smithi078 ceph-mon[91136]: pgmap v9670: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:42:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:42:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:21.500Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:21.500Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:21.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:22 smithi146 ceph-mon[103278]: pgmap v9671: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:22 smithi078 ceph-mon[99959]: pgmap v9671: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:22 smithi078 ceph-mon[91136]: pgmap v9671: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:24 smithi146 ceph-mon[103278]: pgmap v9672: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:24 smithi078 ceph-mon[91136]: pgmap v9672: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:24 smithi078 ceph-mon[99959]: pgmap v9672: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:26.737 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:26 smithi146 ceph-mon[103278]: pgmap v9673: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:26 smithi078 ceph-mon[91136]: pgmap v9673: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:26 smithi078 ceph-mon[99959]: pgmap v9673: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:28 smithi146 ceph-mon[103278]: pgmap v9674: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:28 smithi078 ceph-mon[91136]: pgmap v9674: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:28 smithi078 ceph-mon[99959]: pgmap v9674: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:29.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:30 smithi146 ceph-mon[103278]: pgmap v9675: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:42:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:42:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[91136]: pgmap v9675: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[99959]: pgmap v9675: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:31.500Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:31.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:31.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:32.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:32 smithi078 ceph-mon[91136]: pgmap v9676: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:32.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:32 smithi078 ceph-mon[99959]: pgmap v9676: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:32.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:32 smithi146 ceph-mon[103278]: pgmap v9676: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:34 smithi146 ceph-mon[103278]: pgmap v9677: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:34.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:34 smithi078 ceph-mon[91136]: pgmap v9677: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:34 smithi078 ceph-mon[99959]: pgmap v9677: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[99959]: pgmap v9678: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:36 smithi078 ceph-mon[91136]: pgmap v9678: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:36 smithi146 ceph-mon[103278]: pgmap v9678: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:37.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:37.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:38 smithi078 ceph-mon[91136]: pgmap v9679: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:38 smithi078 ceph-mon[99959]: pgmap v9679: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:38 smithi146 ceph-mon[103278]: pgmap v9679: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:40 smithi146 ceph-mon[103278]: pgmap v9680: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:40 smithi078 ceph-mon[99959]: pgmap v9680: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:40 smithi078 ceph-mon[91136]: pgmap v9680: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:42:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:42:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:41.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:41.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:41.501Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:42 smithi146 ceph-mon[103278]: pgmap v9681: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:42 smithi078 ceph-mon[91136]: pgmap v9681: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:42 smithi078 ceph-mon[99959]: pgmap v9681: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:44 smithi146 ceph-mon[103278]: pgmap v9682: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:44 smithi078 ceph-mon[91136]: pgmap v9682: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:44 smithi078 ceph-mon[99959]: pgmap v9682: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:46 smithi146 ceph-mon[103278]: pgmap v9683: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:46 smithi078 ceph-mon[91136]: pgmap v9683: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:46 smithi078 ceph-mon[99959]: pgmap v9683: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:47.618 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:48.488 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:42:48.656 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:48 smithi146 ceph-mon[103278]: pgmap v9684: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:48 smithi078 ceph-mon[91136]: pgmap v9684: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:48 smithi078 ceph-mon[99959]: pgmap v9684: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 5h - - 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 5h - - 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 18s ago 5h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 14s ago 5h 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 18s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 14s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 18s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 18s ago 5h 523M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 14s ago 5h 354M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 18s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:42:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 18s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 14s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 18s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 18s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 18s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 18s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 14s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 14s ago 5h 85.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 14s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 14s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 14s ago 5h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 18s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 18s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:42:49.013 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:49.323 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:42:49.324 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:42:49.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:42:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:42:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:42:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:42:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:42:49.597 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:42:49.597 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:49 smithi078 ceph-mon[99959]: from='client.85922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:49.598 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3898475353' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:49.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:49 smithi078 ceph-mon[91136]: from='client.85922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:49.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3898475353' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:49 smithi146 ceph-mon[103278]: from='client.85922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3898475353' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:42:49.907 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:42:49.907 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:42:49.908 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:42:50.706 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:42:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:42:50.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='client.85928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='client.85934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: pgmap v9685: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='client.85946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/44891844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='client.85928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='client.85934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: pgmap v9685: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='client.85946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/44891844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:50.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='client.85928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='client.85934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: pgmap v9685: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='client.85946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/44891844' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:51.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:51.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:42:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:42:51.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:42:52.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:52 smithi146 ceph-mon[103278]: pgmap v9686: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:52 smithi078 ceph-mon[91136]: pgmap v9686: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:52 smithi078 ceph-mon[99959]: pgmap v9686: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:54 smithi146 ceph-mon[103278]: pgmap v9687: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:54 smithi078 ceph-mon[91136]: pgmap v9687: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:54 smithi078 ceph-mon[99959]: pgmap v9687: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:42:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:56 smithi146 ceph-mon[103278]: pgmap v9688: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[91136]: pgmap v9688: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:42:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:56 smithi078 ceph-mon[99959]: pgmap v9688: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:57.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:42:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:58 smithi146 ceph-mon[103278]: pgmap v9689: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:58 smithi078 ceph-mon[91136]: pgmap v9689: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:58 smithi078 ceph-mon[99959]: pgmap v9689: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:42:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:00 smithi146 ceph-mon[103278]: pgmap v9690: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:00 smithi078 ceph-mon[91136]: pgmap v9690: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:00 smithi078 ceph-mon[99959]: pgmap v9690: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:43:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:43:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:01.501Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:01.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:01.502Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:02.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:02 smithi078 ceph-mon[91136]: pgmap v9691: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:02.756 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:02 smithi078 ceph-mon[99959]: pgmap v9691: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:02.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:02 smithi146 ceph-mon[103278]: pgmap v9691: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:04 smithi146 ceph-mon[103278]: pgmap v9692: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:04 smithi078 ceph-mon[91136]: pgmap v9692: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:04.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:04 smithi078 ceph-mon[99959]: pgmap v9692: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:06 smithi146 ceph-mon[103278]: pgmap v9693: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:06.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:06 smithi078 ceph-mon[91136]: pgmap v9693: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:06.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:06 smithi078 ceph-mon[99959]: pgmap v9693: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:07.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:08 smithi146 ceph-mon[103278]: pgmap v9694: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:08 smithi078 ceph-mon[99959]: pgmap v9694: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:08 smithi078 ceph-mon[91136]: pgmap v9694: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:10 smithi146 ceph-mon[103278]: pgmap v9695: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:10 smithi078 ceph-mon[99959]: pgmap v9695: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:10 smithi078 ceph-mon[91136]: pgmap v9695: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:43:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:43:11.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.760 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:11.503Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:11.760 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:11.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:11.760 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:11.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:12 smithi078 ceph-mon[91136]: pgmap v9696: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:12 smithi078 ceph-mon[99959]: pgmap v9696: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:12 smithi146 ceph-mon[103278]: pgmap v9696: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:13.645 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:13.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:14 smithi078 ceph-mon[91136]: pgmap v9697: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:14 smithi078 ceph-mon[99959]: pgmap v9697: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:14 smithi146 ceph-mon[103278]: pgmap v9697: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[91136]: pgmap v9698: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:16 smithi078 ceph-mon[99959]: pgmap v9698: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:16 smithi146 ceph-mon[103278]: pgmap v9698: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:17.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:17.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:18 smithi146 ceph-mon[103278]: pgmap v9699: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:18 smithi078 ceph-mon[99959]: pgmap v9699: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:18 smithi078 ceph-mon[91136]: pgmap v9699: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:20.203 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:43:20.732 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:43:20.732 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 5h - - 2024-09-21T01:43:20.732 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 5h - - 2024-09-21T01:43:20.732 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 10s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:43:20.732 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 5s ago 5h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:43:20.732 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 10s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 5s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 10s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 10s ago 5h 523M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 5s ago 5h 355M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 10s ago 5h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 10s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 5s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 10s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 10s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 10s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 10s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 5s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 5s ago 5h 85.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 5s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:43:20.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 5s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:43:20.734 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 5s ago 5h 73.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:43:20.734 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 10s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:43:20.734 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:43:20.734 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 10s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:43:20.734 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 5s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:43:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:20 smithi146 ceph-mon[103278]: pgmap v9700: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:20.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:20 smithi078 ceph-mon[99959]: pgmap v9700: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:20 smithi078 ceph-mon[91136]: pgmap v9700: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:43:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:43:21.041 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:43:21.042 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:43:21.307 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:43:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:43:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:43:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:43:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:43:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:43:21.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:43:21.561 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-mon[99959]: from='client.66039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:21.561 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-mon[99959]: from='client.66045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:21.561 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/103774653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:21.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-mon[91136]: from='client.66039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:21.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-mon[91136]: from='client.66045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:21.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/103774653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:21.562 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:21.503Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:21.562 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:21.503Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:21.562 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:21.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:43:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:21.630 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:21.630 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:21 smithi146 ceph-mon[103278]: from='client.66039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:21 smithi146 ceph-mon[103278]: from='client.66045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/103774653' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:22 smithi146 ceph-mon[103278]: from='client.66051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:22 smithi146 ceph-mon[103278]: from='client.66063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:22 smithi146 ceph-mon[103278]: pgmap v9701: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2058891712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[91136]: from='client.66051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[91136]: from='client.66063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[91136]: pgmap v9701: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2058891712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[99959]: from='client.66051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[99959]: from='client.66063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[99959]: pgmap v9701: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2058891712' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:24 smithi146 ceph-mon[103278]: pgmap v9702: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:24 smithi078 ceph-mon[99959]: pgmap v9702: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:24 smithi078 ceph-mon[91136]: pgmap v9702: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:26.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:26 smithi146 ceph-mon[103278]: pgmap v9703: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:26.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:26 smithi078 ceph-mon[91136]: pgmap v9703: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:26.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:26 smithi078 ceph-mon[99959]: pgmap v9703: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:27.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:27.805 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:28.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:28 smithi146 ceph-mon[103278]: pgmap v9704: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:28.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:28 smithi078 ceph-mon[99959]: pgmap v9704: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:28 smithi078 ceph-mon[91136]: pgmap v9704: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:30.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:30 smithi078 ceph-mon[91136]: pgmap v9705: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:30 smithi078 ceph-mon[99959]: pgmap v9705: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:43:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:43:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:30 smithi146 ceph-mon[103278]: pgmap v9705: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:31.504Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:31.504Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:31.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:32 smithi146 ceph-mon[103278]: pgmap v9706: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:32 smithi078 ceph-mon[91136]: pgmap v9706: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:32 smithi078 ceph-mon[99959]: pgmap v9706: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:34.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:34 smithi078 ceph-mon[91136]: pgmap v9707: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:34.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:34 smithi078 ceph-mon[99959]: pgmap v9707: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:34 smithi146 ceph-mon[103278]: pgmap v9707: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:36.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[91136]: pgmap v9708: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:36 smithi078 ceph-mon[99959]: pgmap v9708: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:36 smithi146 ceph-mon[103278]: pgmap v9708: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:38 smithi146 ceph-mon[103278]: pgmap v9709: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:38 smithi078 ceph-mon[99959]: pgmap v9709: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:38 smithi078 ceph-mon[91136]: pgmap v9709: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:40 smithi146 ceph-mon[103278]: pgmap v9710: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:40 smithi078 ceph-mon[99959]: pgmap v9710: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:40 smithi078 ceph-mon[91136]: pgmap v9710: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:43:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:43:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:41.505Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:41.505Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:41.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:42.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:42 smithi146 ceph-mon[103278]: pgmap v9711: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:42 smithi078 ceph-mon[91136]: pgmap v9711: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:42 smithi078 ceph-mon[99959]: pgmap v9711: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:43.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:44 smithi146 ceph-mon[103278]: pgmap v9712: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:44 smithi078 ceph-mon[91136]: pgmap v9712: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:44 smithi078 ceph-mon[99959]: pgmap v9712: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:46 smithi146 ceph-mon[103278]: pgmap v9713: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:46 smithi078 ceph-mon[91136]: pgmap v9713: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:46 smithi078 ceph-mon[99959]: pgmap v9713: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:48 smithi078 ceph-mon[99959]: pgmap v9714: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:48 smithi078 ceph-mon[91136]: pgmap v9714: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:48.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:48 smithi146 ceph-mon[103278]: pgmap v9714: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:50 smithi146 ceph-mon[103278]: pgmap v9715: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:50.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:50 smithi078 ceph-mon[99959]: pgmap v9715: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:50.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:50 smithi078 ceph-mon[91136]: pgmap v9715: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:43:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:43:51.776 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:51.506Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:51.776 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:51.506Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:43:51.777 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:43:51.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:43:51.937 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:43:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[91136]: pgmap v9716: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:52.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:51 smithi078 ceph-mon[99959]: pgmap v9716: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:51 smithi146 ceph-mon[103278]: pgmap v9716: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:52.474 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:43:52.474 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 5h - - 2024-09-21T01:43:52.474 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 5h - - 2024-09-21T01:43:52.474 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 1s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 17s ago 5h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 1s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 17s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 1s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 1s ago 5h 525M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 17s ago 5h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 1s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 1s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 17s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 1s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 1s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 1s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 1s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:43:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 17s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 17s ago 5h 85.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 17s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 17s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 17s ago 5h 73.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 1s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 1s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:43:52.476 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:43:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:43:52.802 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:43:53.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:52 smithi078 ceph-mon[91136]: from='client.85994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:52 smithi078 ceph-mon[91136]: from='client.86000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:52 smithi078 ceph-mon[91136]: from='client.66087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:52 smithi078 ceph-mon[99959]: from='client.85994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:52 smithi078 ceph-mon[99959]: from='client.86000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:52 smithi078 ceph-mon[99959]: from='client.66087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:43:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:43:53.086 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:43:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:52 smithi146 ceph-mon[103278]: from='client.85994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:52 smithi146 ceph-mon[103278]: from='client.86000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:52 smithi146 ceph-mon[103278]: from='client.66087 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:53.412 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:43:53.412 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:43:53.412 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:43:54.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/431015303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:54.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[99959]: from='client.86018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:54.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1994475870' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:54.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[99959]: pgmap v9717: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:54.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/431015303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:54.096 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[91136]: from='client.86018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:54.096 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1994475870' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:54.096 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:53 smithi078 ceph-mon[91136]: pgmap v9717: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/431015303' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:43:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:53 smithi146 ceph-mon[103278]: from='client.86018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:43:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1994475870' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:43:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:53 smithi146 ceph-mon[103278]: pgmap v9717: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:43:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:56 smithi146 ceph-mon[103278]: pgmap v9718: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[91136]: pgmap v9718: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[99959]: pgmap v9718: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:43:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:43:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:58 smithi146 ceph-mon[103278]: pgmap v9719: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:58 smithi078 ceph-mon[99959]: pgmap v9719: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:58 smithi078 ceph-mon[91136]: pgmap v9719: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:43:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:00 smithi146 ceph-mon[103278]: pgmap v9720: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:01.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:00 smithi078 ceph-mon[99959]: pgmap v9720: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:00 smithi078 ceph-mon[91136]: pgmap v9720: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:44:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:44:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:01.506Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:01.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:01.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:02.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:02 smithi078 ceph-mon[91136]: pgmap v9721: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:02.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:02 smithi078 ceph-mon[99959]: pgmap v9721: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:02 smithi146 ceph-mon[103278]: pgmap v9721: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:04 smithi146 ceph-mon[103278]: pgmap v9722: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:04 smithi078 ceph-mon[99959]: pgmap v9722: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:04 smithi078 ceph-mon[91136]: pgmap v9722: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:06 smithi146 ceph-mon[103278]: pgmap v9723: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:06 smithi078 ceph-mon[91136]: pgmap v9723: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:06 smithi078 ceph-mon[99959]: pgmap v9723: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:08.850 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:08 smithi146 ceph-mon[103278]: pgmap v9724: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:08 smithi078 ceph-mon[99959]: pgmap v9724: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:08 smithi078 ceph-mon[91136]: pgmap v9724: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:10.736 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:10 smithi146 ceph-mon[103278]: pgmap v9725: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:10.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:10 smithi078 ceph-mon[99959]: pgmap v9725: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:10.924 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:44:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:44:10.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:10 smithi078 ceph-mon[91136]: pgmap v9725: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:11.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:11.508Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:11.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:12.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:11 smithi146 ceph-mon[103278]: pgmap v9726: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[91136]: pgmap v9726: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:11 smithi078 ceph-mon[99959]: pgmap v9726: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:13.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:13.731 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:14 smithi078 ceph-mon[99959]: pgmap v9727: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:14 smithi078 ceph-mon[91136]: pgmap v9727: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:14 smithi146 ceph-mon[103278]: pgmap v9727: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:16 smithi146 ceph-mon[103278]: pgmap v9728: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[99959]: pgmap v9728: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[91136]: pgmap v9728: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:18 smithi146 ceph-mon[103278]: pgmap v9729: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:18 smithi078 ceph-mon[99959]: pgmap v9729: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:18 smithi078 ceph-mon[91136]: pgmap v9729: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:20 smithi146 ceph-mon[103278]: pgmap v9730: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:20 smithi078 ceph-mon[91136]: pgmap v9730: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:20 smithi078 ceph-mon[99959]: pgmap v9730: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:44:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T01:44:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:21.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:21.509Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:21.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:22 smithi146 ceph-mon[103278]: pgmap v9731: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:22 smithi078 ceph-mon[91136]: pgmap v9731: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:22 smithi078 ceph-mon[99959]: pgmap v9731: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:23.706 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 5h - - 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 5h - - 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 13s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 8s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 13s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 8s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 13s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 13s ago 5h 525M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 8s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:44:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 13s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 13s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 8s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 13s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 13s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 13s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 13s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 8s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 8s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 8s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 8s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 8s ago 5h 73.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 13s ago 5h 89.2M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:44:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 13s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:44:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 8s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:44:24.542 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:44:24.542 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:44:24.542 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:44:24.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:44:24.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:44:24.544 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:44:24.813 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:44:24.814 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:44:24.815 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:44:24.815 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:44:24.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:24 smithi078 ceph-mon[99959]: pgmap v9732: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:24.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:24 smithi078 ceph-mon[99959]: from='client.86030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:24.815 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3797044594' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:24.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:24 smithi078 ceph-mon[91136]: pgmap v9732: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:24.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:24 smithi078 ceph-mon[91136]: from='client.86030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:24.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3797044594' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:24 smithi146 ceph-mon[103278]: pgmap v9732: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:24 smithi146 ceph-mon[103278]: from='client.86030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3797044594' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:25.122 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:44:25.122 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:25.122 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:25 smithi146 ceph-mon[103278]: from='client.66117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:25 smithi146 ceph-mon[103278]: from='client.66123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1114190429' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:25 smithi078 ceph-mon[99959]: from='client.66117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:25 smithi078 ceph-mon[99959]: from='client.66123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1114190429' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:25 smithi078 ceph-mon[91136]: from='client.66117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:25 smithi078 ceph-mon[91136]: from='client.66123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1114190429' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:26.196 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 01:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T01:44:25.848639814Z level=info msg="Completed cleanup jobs" duration=12.627549ms 2024-09-21T01:44:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:26 smithi146 ceph-mon[103278]: from='client.66135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:26 smithi146 ceph-mon[103278]: pgmap v9733: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:26 smithi078 ceph-mon[99959]: from='client.66135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:26 smithi078 ceph-mon[99959]: pgmap v9733: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:26 smithi078 ceph-mon[91136]: from='client.66135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:26 smithi078 ceph-mon[91136]: pgmap v9733: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:28 smithi146 ceph-mon[103278]: pgmap v9734: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:28 smithi078 ceph-mon[91136]: pgmap v9734: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:28 smithi078 ceph-mon[99959]: pgmap v9734: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:30.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:30 smithi146 ceph-mon[103278]: pgmap v9735: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:30 smithi078 ceph-mon[99959]: pgmap v9735: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:30 smithi078 ceph-mon[91136]: pgmap v9735: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:44:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:44:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:31.509Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:31.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:31.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[91136]: pgmap v9736: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:32 smithi078 ceph-mon[99959]: pgmap v9736: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:32 smithi146 ceph-mon[103278]: pgmap v9736: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:34.563 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:34 smithi078 ceph-mon[99959]: pgmap v9737: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:34.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:34 smithi078 ceph-mon[91136]: pgmap v9737: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:34 smithi146 ceph-mon[103278]: pgmap v9737: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:36.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[99959]: pgmap v9738: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[91136]: pgmap v9738: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:36 smithi146 ceph-mon[103278]: pgmap v9738: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:38 smithi078 ceph-mon[99959]: pgmap v9739: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:38 smithi078 ceph-mon[91136]: pgmap v9739: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:38 smithi146 ceph-mon[103278]: pgmap v9739: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:40 smithi078 ceph-mon[99959]: pgmap v9740: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:40 smithi078 ceph-mon[91136]: pgmap v9740: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:44:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:44:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:40 smithi146 ceph-mon[103278]: pgmap v9740: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:41.510Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:41.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:41.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:42 smithi078 ceph-mon[91136]: pgmap v9741: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:42 smithi078 ceph-mon[99959]: pgmap v9741: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:42 smithi146 ceph-mon[103278]: pgmap v9741: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:45.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:44 smithi146 ceph-mon[103278]: pgmap v9742: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:44 smithi078 ceph-mon[99959]: pgmap v9742: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:44 smithi078 ceph-mon[91136]: pgmap v9742: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:46 smithi146 ceph-mon[103278]: pgmap v9743: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:46 smithi078 ceph-mon[99959]: pgmap v9743: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:46 smithi078 ceph-mon[91136]: pgmap v9743: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:48 smithi146 ceph-mon[103278]: pgmap v9744: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:48 smithi078 ceph-mon[91136]: pgmap v9744: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:48 smithi078 ceph-mon[99959]: pgmap v9744: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:50.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:50 smithi146 ceph-mon[103278]: pgmap v9745: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:50 smithi078 ceph-mon[99959]: pgmap v9745: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:50 smithi078 ceph-mon[91136]: pgmap v9745: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:44:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:44:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:51.511Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:51.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:44:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:44:51.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:44:52.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:52 smithi146 ceph-mon[103278]: pgmap v9746: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:52.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[99959]: pgmap v9746: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:52.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:52 smithi078 ceph-mon[91136]: pgmap v9746: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:54 smithi078 ceph-mon[91136]: pgmap v9747: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:54 smithi078 ceph-mon[99959]: pgmap v9747: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:54 smithi146 ceph-mon[103278]: pgmap v9747: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:44:55.422 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 5h - - 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 5h - - 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 4s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 0s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 4s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:44:55.972 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 0s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 4s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 4s ago 5h 526M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 0s ago 5h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 4s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 4s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 0s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 4s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 4s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 4s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 4s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 0s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 0s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 0s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:44:55.973 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 0s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:44:55.974 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 0s ago 5h 68.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:44:55.974 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 4s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:44:55.974 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:44:55.974 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 4s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:44:55.974 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:44:56.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:44:56.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:44:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: from='client.86066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: pgmap v9748: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: from='client.66153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:56.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3510727340' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: from='client.86066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: pgmap v9748: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: from='client.66153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3510727340' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:44:56.568 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:44:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:44:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:44:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:44:56.569 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:44:56.569 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:44:56.886 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:44:56.886 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:44:56.886 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: from='client.86066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: pgmap v9748: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: from='client.66153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:44:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3510727340' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:44:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[91136]: from='client.86078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[91136]: from='client.66171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/251160846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[99959]: from='client.86078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[99959]: from='client.66171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/251160846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:44:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:57 smithi146 ceph-mon[103278]: from='client.86078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:57 smithi146 ceph-mon[103278]: from='client.66171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:44:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/251160846' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:44:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:58 smithi078 ceph-mon[99959]: pgmap v9749: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:58 smithi078 ceph-mon[91136]: pgmap v9749: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:58 smithi146 ceph-mon[103278]: pgmap v9749: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:44:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:00 smithi078 ceph-mon[91136]: pgmap v9750: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:00 smithi078 ceph-mon[99959]: pgmap v9750: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:45:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:45:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:00 smithi146 ceph-mon[103278]: pgmap v9750: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:01.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:01.512Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:01.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:02 smithi146 ceph-mon[103278]: pgmap v9751: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:02 smithi078 ceph-mon[91136]: pgmap v9751: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:02 smithi078 ceph-mon[99959]: pgmap v9751: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:04 smithi146 ceph-mon[103278]: pgmap v9752: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:04 smithi078 ceph-mon[91136]: pgmap v9752: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:04 smithi078 ceph-mon[99959]: pgmap v9752: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:06 smithi146 ceph-mon[103278]: pgmap v9753: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:06 smithi078 ceph-mon[91136]: pgmap v9753: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:06 smithi078 ceph-mon[99959]: pgmap v9753: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:07.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:08 smithi146 ceph-mon[103278]: pgmap v9754: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:08 smithi078 ceph-mon[91136]: pgmap v9754: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:08 smithi078 ceph-mon[99959]: pgmap v9754: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:45:10.866 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:10 smithi146 ceph-mon[103278]: pgmap v9755: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:10 smithi078 ceph-mon[91136]: pgmap v9755: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:10 smithi078 ceph-mon[99959]: pgmap v9755: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:45:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:45:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:11.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:11.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:11.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:12 smithi146 ceph-mon[103278]: pgmap v9756: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[99959]: pgmap v9756: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:12 smithi078 ceph-mon[91136]: pgmap v9756: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:14 smithi146 ceph-mon[103278]: pgmap v9757: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:14 smithi078 ceph-mon[91136]: pgmap v9757: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:14 smithi078 ceph-mon[99959]: pgmap v9757: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[91136]: pgmap v9758: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[99959]: pgmap v9758: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:16 smithi146 ceph-mon[103278]: pgmap v9758: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:18.658 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:18 smithi078 ceph-mon[91136]: pgmap v9759: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:18.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:18 smithi078 ceph-mon[99959]: pgmap v9759: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:18 smithi146 ceph-mon[103278]: pgmap v9759: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:20 smithi146 ceph-mon[103278]: pgmap v9760: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:20 smithi078 ceph-mon[91136]: pgmap v9760: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:20 smithi078 ceph-mon[99959]: pgmap v9760: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:45:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:45:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:21.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:21.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:21.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:21.911 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:21.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:22 smithi146 ceph-mon[103278]: pgmap v9761: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:22 smithi078 ceph-mon[91136]: pgmap v9761: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:22 smithi078 ceph-mon[99959]: pgmap v9761: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:24 smithi146 ceph-mon[103278]: pgmap v9762: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:24 smithi078 ceph-mon[99959]: pgmap v9762: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:24 smithi078 ceph-mon[91136]: pgmap v9762: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:26.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:26 smithi146 ceph-mon[103278]: pgmap v9763: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:26 smithi078 ceph-mon[91136]: pgmap v9763: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:26 smithi078 ceph-mon[99959]: pgmap v9763: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:27.176 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 5h - - 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 16s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 11s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:45:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 11s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 5h 525M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 11s ago 5h 356M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 16s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 11s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 5h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 11s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 11s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 11s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 11s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:45:27.705 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 11s ago 5h 68.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:45:27.706 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 16s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:45:27.706 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:45:27.706 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 16s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:45:27.706 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 11s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:45:27.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:28.016 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:45:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:45:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:45:28.018 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:45:28.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:28.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:45:28.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:45:28.290 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:45:28.290 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:45:28.290 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:45:28.290 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:45:28.290 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:45:28.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:45:28.601 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:45:28.601 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:45:28.601 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:45:28.601 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:45:28.601 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:45:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:28 smithi146 ceph-mon[103278]: from='client.66183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:28 smithi146 ceph-mon[103278]: from='client.86108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:28 smithi146 ceph-mon[103278]: pgmap v9764: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:28 smithi146 ceph-mon[103278]: from='client.66195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3727821008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[91136]: from='client.66183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[91136]: from='client.86108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[91136]: pgmap v9764: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[91136]: from='client.66195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3727821008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[99959]: from='client.66183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[99959]: from='client.86108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[99959]: pgmap v9764: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[99959]: from='client.66195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3727821008' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:45:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:30.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:29 smithi146 ceph-mon[103278]: from='client.66207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:30.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1434113618' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:29 smithi078 ceph-mon[99959]: from='client.66207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1434113618' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:29 smithi078 ceph-mon[91136]: from='client.66207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:45:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1434113618' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:45:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:30 smithi146 ceph-mon[103278]: pgmap v9765: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:30 smithi078 ceph-mon[99959]: pgmap v9765: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:30 smithi078 ceph-mon[91136]: pgmap v9765: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:45:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:45:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:31.512Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:31.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:31.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:32.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:32 smithi146 ceph-mon[103278]: pgmap v9766: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[91136]: pgmap v9766: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:32 smithi078 ceph-mon[99959]: pgmap v9766: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:34 smithi146 ceph-mon[103278]: pgmap v9767: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:34 smithi078 ceph-mon[91136]: pgmap v9767: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:34 smithi078 ceph-mon[99959]: pgmap v9767: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:36.826 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[99959]: pgmap v9768: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:36.826 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:36.826 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:36.826 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:36.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[91136]: pgmap v9768: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:36.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:36.827 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:36.827 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:36 smithi146 ceph-mon[103278]: pgmap v9768: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:38 smithi146 ceph-mon[103278]: pgmap v9769: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:38 smithi078 ceph-mon[91136]: pgmap v9769: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:38 smithi078 ceph-mon[99959]: pgmap v9769: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:40.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:40 smithi078 ceph-mon[99959]: pgmap v9770: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:40 smithi078 ceph-mon[91136]: pgmap v9770: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:40.840 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:45:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:45:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:40 smithi146 ceph-mon[103278]: pgmap v9770: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:41.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:41.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:41.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:42 smithi146 ceph-mon[103278]: pgmap v9771: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:42 smithi078 ceph-mon[91136]: pgmap v9771: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:42 smithi078 ceph-mon[99959]: pgmap v9771: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:44 smithi146 ceph-mon[103278]: pgmap v9772: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:44 smithi078 ceph-mon[91136]: pgmap v9772: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:44 smithi078 ceph-mon[99959]: pgmap v9772: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:46 smithi146 ceph-mon[103278]: pgmap v9773: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:46 smithi078 ceph-mon[91136]: pgmap v9773: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:46 smithi078 ceph-mon[99959]: pgmap v9773: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:48 smithi146 ceph-mon[103278]: pgmap v9774: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:48 smithi078 ceph-mon[91136]: pgmap v9774: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:48 smithi078 ceph-mon[99959]: pgmap v9774: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:50 smithi078 ceph-mon[91136]: pgmap v9775: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:50 smithi078 ceph-mon[99959]: pgmap v9775: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:50.911 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:45:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:45:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:50 smithi146 ceph-mon[103278]: pgmap v9775: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:51.541 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:51.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:51.541 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:51.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:45:51.541 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:45:51.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:45:52.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:52 smithi146 ceph-mon[103278]: pgmap v9776: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:52.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.695 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[99959]: pgmap v9776: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[91136]: pgmap v9776: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:54 smithi146 ceph-mon[103278]: pgmap v9777: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:54 smithi078 ceph-mon[99959]: pgmap v9777: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:54 smithi078 ceph-mon[91136]: pgmap v9777: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:45:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:56 smithi146 ceph-mon[103278]: pgmap v9778: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[91136]: pgmap v9778: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[99959]: pgmap v9778: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:45:57.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:57.608 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:58.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:45:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:58 smithi078 ceph-mon[99959]: pgmap v9779: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:58 smithi078 ceph-mon[91136]: pgmap v9779: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:58.897 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:45:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:58 smithi146 ceph-mon[103278]: pgmap v9779: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:45:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 5h - - 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 5h - - 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 7s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 3s ago 5h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 7s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 3s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 7s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 7s ago 5h 526M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 3s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:45:59.445 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 7s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 7s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 3s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 7s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 7s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 7s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 7s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 3s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 3s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 3s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 3s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 3s ago 5h 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 7s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:45:59.446 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:45:59.447 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 7s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:45:59.447 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:45:59.766 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:45:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:45:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:45:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:45:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:45:59.768 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:46:00.045 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:46:00.046 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:46:00.046 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:46:00.369 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:46:00.370 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:00.370 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:00.370 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:00.370 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:46:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:46:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[99959]: from='client.66219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[99959]: from='client.66225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[99959]: from='client.86150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[99959]: pgmap v9780: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1346735580' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3928475071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[91136]: from='client.66219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[91136]: from='client.66225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[91136]: from='client.86150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[91136]: pgmap v9780: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1346735580' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3928475071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:00 smithi146 ceph-mon[103278]: from='client.66219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:00 smithi146 ceph-mon[103278]: from='client.66225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:00 smithi146 ceph-mon[103278]: from='client.86150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:00 smithi146 ceph-mon[103278]: pgmap v9780: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1346735580' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3928475071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:01.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:01 smithi078 ceph-mon[99959]: from='client.86162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:01 smithi078 ceph-mon[91136]: from='client.86162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:01.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:01.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:01.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:01 smithi146 ceph-mon[103278]: from='client.86162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:02 smithi146 ceph-mon[103278]: pgmap v9781: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:02 smithi078 ceph-mon[91136]: pgmap v9781: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:02 smithi078 ceph-mon[99959]: pgmap v9781: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:04 smithi146 ceph-mon[103278]: pgmap v9782: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:04 smithi078 ceph-mon[91136]: pgmap v9782: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:04 smithi078 ceph-mon[99959]: pgmap v9782: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:06 smithi146 ceph-mon[103278]: pgmap v9783: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:06 smithi078 ceph-mon[91136]: pgmap v9783: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:06 smithi078 ceph-mon[99959]: pgmap v9783: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:08 smithi146 ceph-mon[103278]: pgmap v9784: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:08 smithi078 ceph-mon[91136]: pgmap v9784: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:08 smithi078 ceph-mon[99959]: pgmap v9784: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:10 smithi146 ceph-mon[103278]: pgmap v9785: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:10 smithi078 ceph-mon[91136]: pgmap v9785: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:10 smithi078 ceph-mon[99959]: pgmap v9785: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:46:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:46:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:11.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:11.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:11.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:12.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:12 smithi146 ceph-mon[103278]: pgmap v9786: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:12.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:12.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:12.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[91136]: pgmap v9786: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[99959]: pgmap v9786: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:14 smithi146 ceph-mon[103278]: pgmap v9787: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:14 smithi078 ceph-mon[91136]: pgmap v9787: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:14 smithi078 ceph-mon[99959]: pgmap v9787: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:16 smithi146 ceph-mon[103278]: pgmap v9788: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[91136]: pgmap v9788: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[99959]: pgmap v9788: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:18 smithi146 ceph-mon[103278]: pgmap v9789: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:18 smithi078 ceph-mon[91136]: pgmap v9789: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:18 smithi078 ceph-mon[99959]: pgmap v9789: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:19 smithi078 ceph-mon[99959]: pgmap v9790: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:19 smithi078 ceph-mon[91136]: pgmap v9790: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:19 smithi146 ceph-mon[103278]: pgmap v9790: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:20.700 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:46:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:46:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:21.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:21.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:21.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:22 smithi146 ceph-mon[103278]: pgmap v9791: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:22 smithi078 ceph-mon[91136]: pgmap v9791: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:22 smithi078 ceph-mon[99959]: pgmap v9791: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:24.817 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:24 smithi146 ceph-mon[103278]: pgmap v9792: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:24 smithi078 ceph-mon[91136]: pgmap v9792: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:24 smithi078 ceph-mon[99959]: pgmap v9792: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:26 smithi146 ceph-mon[103278]: pgmap v9793: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:26 smithi078 ceph-mon[91136]: pgmap v9793: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:26 smithi078 ceph-mon[99959]: pgmap v9793: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:27.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:28.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:28 smithi146 ceph-mon[103278]: pgmap v9794: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:28.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:28 smithi078 ceph-mon[91136]: pgmap v9794: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:28 smithi078 ceph-mon[99959]: pgmap v9794: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:46:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:30 smithi146 ceph-mon[103278]: pgmap v9795: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:46:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:46:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:30 smithi078 ceph-mon[99959]: pgmap v9795: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:30 smithi078 ceph-mon[91136]: pgmap v9795: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 5h - - 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 14s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 14s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 19s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:46:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 14s ago 5h 358M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 14s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 14s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 14s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 14s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 14s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 14s ago 5h 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 19s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:46:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 19s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:46:31.197 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 14s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:46:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:46:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:46:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:46:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:46:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:46:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:46:31.503 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:46:31.775 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:46:31.776 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:46:31.777 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-mon[99959]: from='client.66255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:31.777 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3687436897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:31.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-mon[91136]: from='client.66255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:31.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3687436897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:31.777 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:31.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:31.778 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:31.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:31.778 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:31.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:31 smithi146 ceph-mon[103278]: from='client.66255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3687436897' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:46:32.096 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:46:32.096 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:32.096 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:46:32.096 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:32.097 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: from='client.66261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: from='client.86186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: pgmap v9796: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:32.890 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/314115877' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: from='client.66261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: from='client.86186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: pgmap v9796: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/314115877' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: from='client.66261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: from='client.86186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: pgmap v9796: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/314115877' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:46:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:33 smithi146 ceph-mon[103278]: from='client.86198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:33 smithi078 ceph-mon[91136]: from='client.86198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:33 smithi078 ceph-mon[99959]: from='client.86198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:46:34.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:34 smithi146 ceph-mon[103278]: pgmap v9797: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:34 smithi078 ceph-mon[99959]: pgmap v9797: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:34 smithi078 ceph-mon[91136]: pgmap v9797: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:35 smithi146 ceph-mon[103278]: pgmap v9798: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:35 smithi078 ceph-mon[91136]: pgmap v9798: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:35 smithi078 ceph-mon[99959]: pgmap v9798: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:38 smithi146 ceph-mon[103278]: pgmap v9799: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:38.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:38 smithi078 ceph-mon[91136]: pgmap v9799: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:38.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:38 smithi078 ceph-mon[99959]: pgmap v9799: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:40.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:40 smithi078 ceph-mon[91136]: pgmap v9800: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:40.740 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:46:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:46:40.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:40 smithi078 ceph-mon[99959]: pgmap v9800: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:40 smithi146 ceph-mon[103278]: pgmap v9800: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:41.517Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:41.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:41.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:41.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:41.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:42 smithi146 ceph-mon[103278]: pgmap v9801: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:42 smithi078 ceph-mon[99959]: pgmap v9801: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:42 smithi078 ceph-mon[91136]: pgmap v9801: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:44 smithi146 ceph-mon[103278]: pgmap v9802: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:44 smithi078 ceph-mon[99959]: pgmap v9802: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:44 smithi078 ceph-mon[91136]: pgmap v9802: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:46 smithi146 ceph-mon[103278]: pgmap v9803: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:46 smithi078 ceph-mon[99959]: pgmap v9803: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:46 smithi078 ceph-mon[91136]: pgmap v9803: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:48 smithi146 ceph-mon[103278]: pgmap v9804: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:48 smithi078 ceph-mon[99959]: pgmap v9804: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:48 smithi078 ceph-mon[91136]: pgmap v9804: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:50 smithi146 ceph-mon[103278]: pgmap v9805: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:50 smithi078 ceph-mon[99959]: pgmap v9805: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:50 smithi078 ceph-mon[91136]: pgmap v9805: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:46:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:46:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:51.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:51.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:46:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:46:51.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:46:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:52 smithi146 ceph-mon[103278]: pgmap v9806: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[91136]: pgmap v9806: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[99959]: pgmap v9806: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T01:46:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:54.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:54 smithi146 ceph-mon[103278]: pgmap v9807: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:54 smithi078 ceph-mon[91136]: pgmap v9807: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:54 smithi078 ceph-mon[99959]: pgmap v9807: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:46:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:56 smithi146 ceph-mon[103278]: pgmap v9808: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:56 smithi078 ceph-mon[91136]: pgmap v9808: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:56 smithi078 ceph-mon[99959]: pgmap v9808: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:46:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:46:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:58 smithi146 ceph-mon[103278]: pgmap v9809: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:46:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:58 smithi078 ceph-mon[91136]: pgmap v9809: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:46:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:58 smithi078 ceph-mon[99959]: pgmap v9809: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:46:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:46:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:00.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:00 smithi078 ceph-mon[99959]: pgmap v9810: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:00.898 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:00 smithi078 ceph-mon[91136]: pgmap v9810: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:00.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:47:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:47:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:00 smithi146 ceph-mon[103278]: pgmap v9810: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:01.519Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:01.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:01.520Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:02.410 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:47:02.942 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:47:02.942 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 5h - - 2024-09-21T01:47:02.942 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 5h - - 2024-09-21T01:47:02.942 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 10s ago 5h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:47:02.942 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 6s ago 5h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 10s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 6s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 10s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 10s ago 5h 526M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 6s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 10s ago 5h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 10s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 6s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 10s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 10s ago 5h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 10s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 10s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 6s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 6s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:47:02.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 6s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:47:02.944 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 6s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:47:02.944 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 6s ago 5h 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:47:02.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 10s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:47:02.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:47:02.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 10s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:47:02.944 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:47:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:02 smithi146 ceph-mon[103278]: pgmap v9811: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:02 smithi078 ceph-mon[99959]: pgmap v9811: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:02 smithi078 ceph-mon[91136]: pgmap v9811: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:47:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:47:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:47:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:47:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:47:03.254 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:47:03.523 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:47:03.523 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:47:03.523 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:47:03.524 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:47:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:03 smithi078 ceph-mon[91136]: from='client.86210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:03 smithi078 ceph-mon[91136]: from='client.66297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/832943673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:03 smithi078 ceph-mon[99959]: from='client.86210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:03 smithi078 ceph-mon[99959]: from='client.66297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/832943673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:03.835 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:47:03.835 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:47:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:47:03.835 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:47:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:47:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:03 smithi146 ceph-mon[103278]: from='client.86210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:03 smithi146 ceph-mon[103278]: from='client.66297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/832943673' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:04 smithi146 ceph-mon[103278]: from='client.66303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:04 smithi146 ceph-mon[103278]: pgmap v9812: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:04 smithi146 ceph-mon[103278]: from='client.86234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1480434833' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[91136]: from='client.66303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[91136]: pgmap v9812: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[91136]: from='client.86234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1480434833' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[99959]: from='client.66303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[99959]: pgmap v9812: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:05.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[99959]: from='client.86234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:05.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1480434833' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:06 smithi146 ceph-mon[103278]: pgmap v9813: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:06 smithi078 ceph-mon[99959]: pgmap v9813: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:06 smithi078 ceph-mon[91136]: pgmap v9813: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:08 smithi146 ceph-mon[103278]: pgmap v9814: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:08 smithi078 ceph-mon[99959]: pgmap v9814: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:08 smithi078 ceph-mon[91136]: pgmap v9814: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:09 smithi146 ceph-mon[103278]: pgmap v9815: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:09 smithi078 ceph-mon[99959]: pgmap v9815: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:09 smithi078 ceph-mon[91136]: pgmap v9815: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:47:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:47:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:11.520Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:11.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:11.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:12 smithi146 ceph-mon[103278]: pgmap v9816: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[91136]: pgmap v9816: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[99959]: pgmap v9816: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:13.856 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:14 smithi146 ceph-mon[103278]: pgmap v9817: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:14 smithi078 ceph-mon[91136]: pgmap v9817: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:14 smithi078 ceph-mon[99959]: pgmap v9817: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:16.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:16 smithi146 ceph-mon[103278]: pgmap v9818: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:16 smithi078 ceph-mon[91136]: pgmap v9818: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:16 smithi078 ceph-mon[99959]: pgmap v9818: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:18 smithi146 ceph-mon[103278]: pgmap v9819: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:18 smithi078 ceph-mon[91136]: pgmap v9819: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:18 smithi078 ceph-mon[99959]: pgmap v9819: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:20.841 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:20 smithi078 ceph-mon[99959]: pgmap v9820: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:20.842 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:47:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:47:20.842 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:20 smithi078 ceph-mon[91136]: pgmap v9820: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:20 smithi146 ceph-mon[103278]: pgmap v9820: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:21.534 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:21.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:21.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:21.535 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:21.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:22 smithi146 ceph-mon[103278]: pgmap v9821: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:22 smithi078 ceph-mon[91136]: pgmap v9821: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:22 smithi078 ceph-mon[99959]: pgmap v9821: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:24 smithi146 ceph-mon[103278]: pgmap v9822: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:24 smithi078 ceph-mon[99959]: pgmap v9822: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:24 smithi078 ceph-mon[91136]: pgmap v9822: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:26 smithi146 ceph-mon[103278]: pgmap v9823: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:26 smithi078 ceph-mon[99959]: pgmap v9823: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:26 smithi078 ceph-mon[91136]: pgmap v9823: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:28.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:28 smithi146 ceph-mon[103278]: pgmap v9824: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:28.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:28 smithi078 ceph-mon[91136]: pgmap v9824: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:28 smithi078 ceph-mon[99959]: pgmap v9824: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:30 smithi146 ceph-mon[103278]: pgmap v9825: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:30 smithi078 ceph-mon[91136]: pgmap v9825: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:30 smithi078 ceph-mon[99959]: pgmap v9825: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:47:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:47:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:31.521Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:31.521Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:31.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:32 smithi146 ceph-mon[103278]: pgmap v9826: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[99959]: pgmap v9826: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[91136]: pgmap v9826: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:34.130 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:47:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:47:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 5h - - 2024-09-21T01:47:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 5h - - 2024-09-21T01:47:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 2s ago 5h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:47:34.654 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 17s ago 5h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 2s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 17s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 2s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 17s ago 5h 357M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 2s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 17s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 2s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 17s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 17s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 17s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:47:34.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 17s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:47:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 17s ago 5h 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:47:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 2s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:47:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:47:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 2s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:47:34.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 17s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:47:34.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:34 smithi146 ceph-mon[103278]: pgmap v9827: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:34.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:47:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:47:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:47:34.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:47:34.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:47:34.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:34 smithi078 ceph-mon[99959]: pgmap v9827: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:34.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:34 smithi078 ceph-mon[91136]: pgmap v9827: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:47:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:47:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:47:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:47:35.239 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:47:35.548 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:47:35.548 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:47:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:47:35.548 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:47:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:47:35.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:35 smithi146 ceph-mon[103278]: from='client.66327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:35.623 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:35 smithi146 ceph-mon[103278]: from='client.66333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:35.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:35 smithi146 ceph-mon[103278]: from='client.86258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:35.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2600768949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:35.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2697159063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[99959]: from='client.66327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[99959]: from='client.66333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[99959]: from='client.86258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2600768949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2697159063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[91136]: from='client.66327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[91136]: from='client.66333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[91136]: from='client.86258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2600768949' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:47:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2697159063' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:47:36.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:36 smithi146 ceph-mon[103278]: from='client.86270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:36.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:36 smithi146 ceph-mon[103278]: pgmap v9828: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:36 smithi078 ceph-mon[99959]: from='client.86270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:36 smithi078 ceph-mon[99959]: pgmap v9828: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:36 smithi078 ceph-mon[91136]: from='client.86270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:47:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:36 smithi078 ceph-mon[91136]: pgmap v9828: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:38 smithi146 ceph-mon[103278]: pgmap v9829: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:38 smithi078 ceph-mon[99959]: pgmap v9829: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:38 smithi078 ceph-mon[91136]: pgmap v9829: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:40 smithi146 ceph-mon[103278]: pgmap v9830: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:40 smithi078 ceph-mon[91136]: pgmap v9830: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:40 smithi078 ceph-mon[99959]: pgmap v9830: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:47:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:47:41.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:41.522Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:41.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:41.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:41.729 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:41.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:42.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:42 smithi078 ceph-mon[91136]: pgmap v9831: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:42.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:42 smithi078 ceph-mon[99959]: pgmap v9831: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:42 smithi146 ceph-mon[103278]: pgmap v9831: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:43.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:44 smithi146 ceph-mon[103278]: pgmap v9832: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:44 smithi078 ceph-mon[99959]: pgmap v9832: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:44 smithi078 ceph-mon[91136]: pgmap v9832: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:46 smithi146 ceph-mon[103278]: pgmap v9833: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:46 smithi078 ceph-mon[99959]: pgmap v9833: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:46 smithi078 ceph-mon[91136]: pgmap v9833: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:48 smithi146 ceph-mon[103278]: pgmap v9834: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:48 smithi078 ceph-mon[99959]: pgmap v9834: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:48 smithi078 ceph-mon[91136]: pgmap v9834: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:50 smithi146 ceph-mon[103278]: pgmap v9835: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:50 smithi078 ceph-mon[99959]: pgmap v9835: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:50 smithi078 ceph-mon[91136]: pgmap v9835: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:47:50] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T01:47:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:51.522Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:51.522Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:47:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:47:51.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:47:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:52 smithi146 ceph-mon[103278]: pgmap v9836: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[91136]: pgmap v9836: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[99959]: pgmap v9836: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:54.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:54 smithi078 ceph-mon[99959]: pgmap v9837: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:54 smithi078 ceph-mon[91136]: pgmap v9837: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:54 smithi146 ceph-mon[103278]: pgmap v9837: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:47:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:56 smithi146 ceph-mon[103278]: pgmap v9838: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:56 smithi078 ceph-mon[91136]: pgmap v9838: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:56 smithi078 ceph-mon[99959]: pgmap v9838: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:47:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:47:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:58 smithi146 ceph-mon[103278]: pgmap v9839: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:47:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:58 smithi078 ceph-mon[91136]: pgmap v9839: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:47:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:58 smithi078 ceph-mon[99959]: pgmap v9839: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:47:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:47:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:00 smithi146 ceph-mon[103278]: pgmap v9840: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:00 smithi078 ceph-mon[91136]: pgmap v9840: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:00 smithi078 ceph-mon[99959]: pgmap v9840: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:48:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:48:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:01.522Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:01.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:01.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:02 smithi146 ceph-mon[103278]: pgmap v9841: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:02 smithi078 ceph-mon[99959]: pgmap v9841: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:02 smithi078 ceph-mon[91136]: pgmap v9841: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:04.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:04 smithi078 ceph-mon[99959]: pgmap v9842: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:04.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:04 smithi078 ceph-mon[91136]: pgmap v9842: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:04 smithi146 ceph-mon[103278]: pgmap v9842: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:05.847 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:48:06.374 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 5h - - 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 5h - - 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 14s ago 5h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 9s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 14s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 9s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 14s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 14s ago 5h 527M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:48:06.375 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 9s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 14s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 14s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 9s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 14s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 14s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 14s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 14s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 9s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 9s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 9s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 9s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 9s ago 5h 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 14s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:48:06.376 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 14s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:48:06.377 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:06.689 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:48:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:48:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:48:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:48:06.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:48:06.690 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:48:06.957 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:48:06.958 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:48:06.958 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:48:06.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:06 smithi078 ceph-mon[99959]: pgmap v9843: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:06.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:06 smithi078 ceph-mon[91136]: pgmap v9843: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:06 smithi146 ceph-mon[103278]: pgmap v9843: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:07.264 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:48:07.264 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:07.264 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:07.264 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:07 smithi146 ceph-mon[103278]: from='client.86282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:07 smithi146 ceph-mon[103278]: from='client.66369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:07 smithi146 ceph-mon[103278]: from='client.86294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2894938961' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1512551061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[91136]: from='client.86282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[91136]: from='client.66369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[91136]: from='client.86294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2894938961' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1512551061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[99959]: from='client.86282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[99959]: from='client.66369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[99959]: from='client.86294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2894938961' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1512551061' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:08 smithi146 ceph-mon[103278]: from='client.86306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:08 smithi146 ceph-mon[103278]: pgmap v9844: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:08 smithi078 ceph-mon[91136]: from='client.86306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:08 smithi078 ceph-mon[91136]: pgmap v9844: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:08 smithi078 ceph-mon[99959]: from='client.86306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:08 smithi078 ceph-mon[99959]: pgmap v9844: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:10 smithi146 ceph-mon[103278]: pgmap v9845: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:10 smithi078 ceph-mon[99959]: pgmap v9845: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:10 smithi078 ceph-mon[91136]: pgmap v9845: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:48:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:48:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:11.523Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:11.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:11.524Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:12.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[91136]: pgmap v9846: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[99959]: pgmap v9846: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:12 smithi146 ceph-mon[103278]: pgmap v9846: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:14 smithi146 ceph-mon[103278]: pgmap v9847: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:14 smithi078 ceph-mon[99959]: pgmap v9847: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:14 smithi078 ceph-mon[91136]: pgmap v9847: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:16.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:16 smithi146 ceph-mon[103278]: pgmap v9848: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:16 smithi078 ceph-mon[91136]: pgmap v9848: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:16 smithi078 ceph-mon[99959]: pgmap v9848: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:18 smithi146 ceph-mon[103278]: pgmap v9849: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:18 smithi078 ceph-mon[91136]: pgmap v9849: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:18 smithi078 ceph-mon[99959]: pgmap v9849: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:20 smithi146 ceph-mon[103278]: pgmap v9850: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:20 smithi078 ceph-mon[99959]: pgmap v9850: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:48:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:48:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:20 smithi078 ceph-mon[91136]: pgmap v9850: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:21.524Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:21.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:21.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:21.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:22.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:22 smithi078 ceph-mon[91136]: pgmap v9851: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:22.974 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:22 smithi078 ceph-mon[99959]: pgmap v9851: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:22 smithi146 ceph-mon[103278]: pgmap v9851: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:24 smithi146 ceph-mon[103278]: pgmap v9852: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:24 smithi078 ceph-mon[91136]: pgmap v9852: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:24 smithi078 ceph-mon[99959]: pgmap v9852: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:26 smithi146 ceph-mon[103278]: pgmap v9853: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:26 smithi078 ceph-mon[91136]: pgmap v9853: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:26 smithi078 ceph-mon[99959]: pgmap v9853: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:28 smithi146 ceph-mon[103278]: pgmap v9854: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:28 smithi078 ceph-mon[99959]: pgmap v9854: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:28 smithi078 ceph-mon[91136]: pgmap v9854: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:30 smithi146 ceph-mon[103278]: pgmap v9855: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:30 smithi078 ceph-mon[99959]: pgmap v9855: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:30 smithi078 ceph-mon[91136]: pgmap v9855: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:48:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:48:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:31.525Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:31.525Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:31.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:32 smithi146 ceph-mon[103278]: pgmap v9856: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[91136]: pgmap v9856: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[99959]: pgmap v9856: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:34 smithi146 ceph-mon[103278]: pgmap v9857: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:34 smithi078 ceph-mon[91136]: pgmap v9857: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:34 smithi078 ceph-mon[99959]: pgmap v9857: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:36.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:36 smithi146 ceph-mon[103278]: pgmap v9858: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:36 smithi078 ceph-mon[91136]: pgmap v9858: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:36 smithi078 ceph-mon[99959]: pgmap v9858: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:37.561 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:48:38.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:38.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.002 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 5h - - 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 5h - - 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 5s ago 5h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 0s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 5s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 0s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 5s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 5s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 0s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 5s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 5s ago 5h 16.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 0s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:48:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 5s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 5s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 5s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 5s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 0s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 0s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 0s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 0s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 0s ago 5h 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 5s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 5s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:48:38.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 0s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:48:38.399 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:48:38.400 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:48:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:48:38.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:48:38.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:48:38.667 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:48:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:48:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:48:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:48:38.668 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:48:38.978 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:48:38.979 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:48:38.979 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:48:38.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:38 smithi078 ceph-mon[99959]: from='client.86318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:38.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:38 smithi078 ceph-mon[99959]: pgmap v9859: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:38.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2477991110' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:38 smithi078 ceph-mon[91136]: from='client.86318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:38 smithi078 ceph-mon[91136]: pgmap v9859: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:38.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2477991110' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:38 smithi146 ceph-mon[103278]: from='client.86318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:38 smithi146 ceph-mon[103278]: pgmap v9859: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2477991110' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:48:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:39 smithi146 ceph-mon[103278]: from='client.86324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:39 smithi146 ceph-mon[103278]: from='client.66411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:39 smithi146 ceph-mon[103278]: from='client.86342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2804780247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[91136]: from='client.86324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[91136]: from='client.66411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[91136]: from='client.86342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2804780247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[99959]: from='client.86324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[99959]: from='client.66411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[99959]: from='client.86342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:48:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2804780247' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:48:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:40 smithi146 ceph-mon[103278]: pgmap v9860: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:40 smithi078 ceph-mon[91136]: pgmap v9860: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:40 smithi078 ceph-mon[99959]: pgmap v9860: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:48:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:48:41.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:41 smithi078 ceph-mon[91136]: pgmap v9861: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:41.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:41 smithi078 ceph-mon[99959]: pgmap v9861: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:41.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:41.526Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:41.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:41 smithi146 ceph-mon[103278]: pgmap v9861: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:43.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:43.821 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:44 smithi078 ceph-mon[91136]: pgmap v9862: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:44 smithi078 ceph-mon[99959]: pgmap v9862: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:44 smithi146 ceph-mon[103278]: pgmap v9862: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:46 smithi146 ceph-mon[103278]: pgmap v9863: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:46 smithi078 ceph-mon[99959]: pgmap v9863: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:46 smithi078 ceph-mon[91136]: pgmap v9863: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:48 smithi146 ceph-mon[103278]: pgmap v9864: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:48 smithi078 ceph-mon[91136]: pgmap v9864: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:48 smithi078 ceph-mon[99959]: pgmap v9864: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:50 smithi146 ceph-mon[103278]: pgmap v9865: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:50 smithi078 ceph-mon[91136]: pgmap v9865: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:50 smithi078 ceph-mon[99959]: pgmap v9865: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:48:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:48:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:51.526Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:51.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:48:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:48:51.527Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:48:52.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:52 smithi078 ceph-mon[91136]: pgmap v9866: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:52.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:52 smithi078 ceph-mon[99959]: pgmap v9866: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:52 smithi146 ceph-mon[103278]: pgmap v9866: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:54 smithi146 ceph-mon[103278]: pgmap v9867: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:54 smithi078 ceph-mon[91136]: pgmap v9867: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:54 smithi078 ceph-mon[99959]: pgmap v9867: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:48:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:56 smithi146 ceph-mon[103278]: pgmap v9868: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:56 smithi078 ceph-mon[91136]: pgmap v9868: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:56 smithi078 ceph-mon[99959]: pgmap v9868: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:48:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:48:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:58 smithi146 ceph-mon[103278]: pgmap v9869: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:58 smithi078 ceph-mon[91136]: pgmap v9869: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:58 smithi078 ceph-mon[99959]: pgmap v9869: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:48:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:59.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:48:59 smithi146 ceph-mon[103278]: pgmap v9870: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:48:59 smithi078 ceph-mon[91136]: pgmap v9870: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:48:59 smithi078 ceph-mon[99959]: pgmap v9870: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:49:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:49:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:01.527Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:01.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:01.528Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:02 smithi078 ceph-mon[91136]: pgmap v9871: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:02 smithi078 ceph-mon[99959]: pgmap v9871: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:02 smithi146 ceph-mon[103278]: pgmap v9871: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:04.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:04 smithi078 ceph-mon[99959]: pgmap v9872: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:04.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:04 smithi078 ceph-mon[91136]: pgmap v9872: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:04 smithi146 ceph-mon[103278]: pgmap v9872: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:06.919 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:06 smithi078 ceph-mon[91136]: pgmap v9873: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:06.920 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:06 smithi078 ceph-mon[99959]: pgmap v9873: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:06 smithi146 ceph-mon[103278]: pgmap v9873: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:07.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:08 smithi146 ceph-mon[103278]: pgmap v9874: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:08 smithi078 ceph-mon[99959]: pgmap v9874: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:08 smithi078 ceph-mon[91136]: pgmap v9874: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:09.276 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 5h - - 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 5h - - 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 17s ago 5h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 12s ago 5h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 17s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 12s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 17s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:49:09.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 17s ago 5h 528M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 5h 360M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 17s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 17s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 12s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 17s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 17s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 17s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 17s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 12s ago 5h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:49:09.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 17s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:49:09.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:49:09.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 17s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:49:09.803 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:49:10.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:49:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:49:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:49:10.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:49:10.382 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:49:10.383 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:49:10.696 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:49:10.696 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:10.696 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:10.696 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:10.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:10.697 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:49:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:49:10.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[91136]: from='client.66435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:10.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[91136]: from='client.66441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:10.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[91136]: pgmap v9875: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:10.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2652012231' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:10.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[99959]: from='client.66435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:10.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[99959]: from='client.66441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:10.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[99959]: pgmap v9875: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:10.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2652012231' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:10 smithi146 ceph-mon[103278]: from='client.66435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:10 smithi146 ceph-mon[103278]: from='client.66441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:10 smithi146 ceph-mon[103278]: pgmap v9875: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2652012231' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-mon[91136]: from='client.66447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-mon[91136]: from='client.66459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3111430537' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-mon[99959]: from='client.66447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-mon[99959]: from='client.66459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3111430537' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:11.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:11.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:11.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:11 smithi146 ceph-mon[103278]: from='client.66447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:11 smithi146 ceph-mon[103278]: from='client.66459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3111430537' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:12 smithi146 ceph-mon[103278]: pgmap v9876: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:12 smithi078 ceph-mon[91136]: pgmap v9876: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:12 smithi078 ceph-mon[99959]: pgmap v9876: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:13 smithi146 ceph-mon[103278]: pgmap v9877: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[91136]: pgmap v9877: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:13 smithi078 ceph-mon[99959]: pgmap v9877: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:16 smithi146 ceph-mon[103278]: pgmap v9878: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:17.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:16 smithi078 ceph-mon[99959]: pgmap v9878: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:17.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:16 smithi078 ceph-mon[91136]: pgmap v9878: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:18 smithi146 ceph-mon[103278]: pgmap v9879: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:18 smithi078 ceph-mon[91136]: pgmap v9879: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:18 smithi078 ceph-mon[99959]: pgmap v9879: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:20 smithi146 ceph-mon[103278]: pgmap v9880: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:20 smithi078 ceph-mon[91136]: pgmap v9880: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:20 smithi078 ceph-mon[99959]: pgmap v9880: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:49:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:49:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:21.528Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:21.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:21.529Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:22.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:22 smithi078 ceph-mon[99959]: pgmap v9881: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:22.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:22 smithi078 ceph-mon[91136]: pgmap v9881: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:22 smithi146 ceph-mon[103278]: pgmap v9881: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:24 smithi146 ceph-mon[103278]: pgmap v9882: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:24 smithi078 ceph-mon[99959]: pgmap v9882: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:24 smithi078 ceph-mon[91136]: pgmap v9882: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:26.218 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:25 smithi078 ceph-mon[91136]: pgmap v9883: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:26.218 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:25 smithi078 ceph-mon[99959]: pgmap v9883: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:25 smithi146 ceph-mon[103278]: pgmap v9883: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:27.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:27.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:27 smithi146 ceph-mon[103278]: pgmap v9884: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:27 smithi078 ceph-mon[91136]: pgmap v9884: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:27 smithi078 ceph-mon[99959]: pgmap v9884: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:29 smithi146 ceph-mon[103278]: pgmap v9885: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:29 smithi078 ceph-mon[91136]: pgmap v9885: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:29 smithi078 ceph-mon[99959]: pgmap v9885: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:49:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:49:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:31.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:31.530Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:31.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:32 smithi078 ceph-mon[91136]: pgmap v9886: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:32 smithi078 ceph-mon[99959]: pgmap v9886: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:32 smithi146 ceph-mon[103278]: pgmap v9886: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:33 smithi146 ceph-mon[103278]: pgmap v9887: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[91136]: pgmap v9887: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:33 smithi078 ceph-mon[99959]: pgmap v9887: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:36.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:36 smithi146 ceph-mon[103278]: pgmap v9888: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:36 smithi078 ceph-mon[91136]: pgmap v9888: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:36 smithi078 ceph-mon[99959]: pgmap v9888: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:38.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:38 smithi146 ceph-mon[103278]: pgmap v9889: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:38.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:38.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:38.861 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[91136]: pgmap v9889: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[99959]: pgmap v9889: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:40 smithi146 ceph-mon[103278]: pgmap v9890: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:40.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:40 smithi078 ceph-mon[99959]: pgmap v9890: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:40.993 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:49:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T01:49:40.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:40 smithi078 ceph-mon[91136]: pgmap v9890: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:40.993 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 5h - - 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 5h - - 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 8s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 3s ago 5h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 8s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:49:41.519 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 3s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 8s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 8s ago 5h 529M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 3s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 8s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 8s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 3s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 8s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 8s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 8s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 8s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 3s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 3s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 3s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 3s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 3s ago 5h 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:49:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 8s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:49:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:49:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 8s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:49:41.521 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 3s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:49:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:41.530Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:41.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:41.531Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:49:41.832 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:49:41.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:49:41.833 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:49:41.833 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:49:41.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:49:41.833 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:49:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:49:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:49:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:49:42.104 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:49:42.412 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:49:42.412 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:49:42.413 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:42.413 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:49:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:42 smithi146 ceph-mon[103278]: from='client.66471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:42 smithi146 ceph-mon[103278]: from='client.66477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:42 smithi146 ceph-mon[103278]: from='client.66483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:42 smithi146 ceph-mon[103278]: pgmap v9891: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2457521090' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2787803683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[91136]: from='client.66471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[91136]: from='client.66477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[91136]: from='client.66483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[91136]: pgmap v9891: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2457521090' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2787803683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[99959]: from='client.66471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[99959]: from='client.66477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[99959]: from='client.66483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[99959]: pgmap v9891: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2457521090' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:49:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2787803683' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:49:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:43 smithi146 ceph-mon[103278]: from='client.66495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:43 smithi078 ceph-mon[91136]: from='client.66495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:43 smithi078 ceph-mon[99959]: from='client.66495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:49:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:44 smithi146 ceph-mon[103278]: pgmap v9892: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:44 smithi078 ceph-mon[91136]: pgmap v9892: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:44 smithi078 ceph-mon[99959]: pgmap v9892: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:46.997 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:46 smithi078 ceph-mon[99959]: pgmap v9893: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:46.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:46 smithi078 ceph-mon[91136]: pgmap v9893: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:46 smithi146 ceph-mon[103278]: pgmap v9893: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:47.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:47.831 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:48 smithi146 ceph-mon[103278]: pgmap v9894: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:48 smithi078 ceph-mon[91136]: pgmap v9894: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:48 smithi078 ceph-mon[99959]: pgmap v9894: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:49 smithi146 ceph-mon[103278]: pgmap v9895: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:49 smithi078 ceph-mon[91136]: pgmap v9895: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:49 smithi078 ceph-mon[99959]: pgmap v9895: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:49:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:49:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:51.531Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:51.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:49:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:49:51.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:49:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:52 smithi146 ceph-mon[103278]: pgmap v9896: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:52 smithi078 ceph-mon[99959]: pgmap v9896: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:52 smithi078 ceph-mon[91136]: pgmap v9896: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:54 smithi146 ceph-mon[103278]: pgmap v9897: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[99959]: pgmap v9897: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:54 smithi078 ceph-mon[91136]: pgmap v9897: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:49:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:56 smithi146 ceph-mon[103278]: pgmap v9898: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:56 smithi078 ceph-mon[99959]: pgmap v9898: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:56 smithi078 ceph-mon[91136]: pgmap v9898: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:49:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:58 smithi146 ceph-mon[103278]: pgmap v9899: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:49:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[91136]: pgmap v9899: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[99959]: pgmap v9899: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:49:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:49:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:49:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T01:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:00 smithi146 ceph-mon[103278]: pgmap v9900: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:50:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:50:01.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:00 smithi078 ceph-mon[99959]: pgmap v9900: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:01.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:00 smithi078 ceph-mon[91136]: pgmap v9900: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:01.531Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:01.532Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:01.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:01.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:02 smithi146 ceph-mon[103278]: pgmap v9901: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:02 smithi078 ceph-mon[99959]: pgmap v9901: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:02 smithi078 ceph-mon[91136]: pgmap v9901: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:04 smithi146 ceph-mon[103278]: pgmap v9902: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:04 smithi078 ceph-mon[99959]: pgmap v9902: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:04 smithi078 ceph-mon[91136]: pgmap v9902: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:06.945 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:06 smithi078 ceph-mon[99959]: pgmap v9903: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:06.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:06 smithi078 ceph-mon[91136]: pgmap v9903: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:06 smithi146 ceph-mon[103278]: pgmap v9903: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:08.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:08.011 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:08 smithi078 ceph-mon[91136]: pgmap v9904: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:08 smithi078 ceph-mon[99959]: pgmap v9904: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:08 smithi146 ceph-mon[103278]: pgmap v9904: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:10 smithi146 ceph-mon[103278]: pgmap v9905: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:10 smithi078 ceph-mon[91136]: pgmap v9905: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:10 smithi078 ceph-mon[99959]: pgmap v9905: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:50:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:50:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:11.532Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:11.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:11.533Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:12.711 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:50:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:12 smithi146 ceph-mon[103278]: pgmap v9906: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:12 smithi078 ceph-mon[91136]: pgmap v9906: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:12 smithi078 ceph-mon[99959]: pgmap v9906: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T01:50:13.275 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 5h - - 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 5h - - 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 0s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 15s ago 5h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 0s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 15s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 0s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 0s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 15s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 0s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 0s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 15s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:50:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 0s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 0s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 0s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 0s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 15s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 15s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 15s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 15s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 15s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 0s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 0s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:50:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 15s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:50:13.588 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:50:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:50:13.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:50:13.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:50:13.590 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:50:13.860 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:50:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:50:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:50:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:50:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:50:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:50:13.861 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:50:13.861 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[91136]: from='client.66507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:13.861 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3797332460' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[99959]: from='client.66507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:13.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:13.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:13.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3797332460' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:13 smithi146 ceph-mon[103278]: from='client.66507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3797332460' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:14.174 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:14.174 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:14.174 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:14.174 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:14 smithi146 ceph-mon[103278]: from='client.86432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:14 smithi146 ceph-mon[103278]: from='client.86438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:14 smithi146 ceph-mon[103278]: pgmap v9907: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3337049892' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[91136]: from='client.86432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[91136]: from='client.86438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[91136]: pgmap v9907: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3337049892' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[99959]: from='client.86432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[99959]: from='client.86438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[99959]: pgmap v9907: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3337049892' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:15 smithi146 ceph-mon[103278]: from='client.86450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:15 smithi078 ceph-mon[91136]: from='client.86450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:15 smithi078 ceph-mon[99959]: from='client.86450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:16 smithi146 ceph-mon[103278]: pgmap v9908: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:16 smithi078 ceph-mon[91136]: pgmap v9908: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:16 smithi078 ceph-mon[99959]: pgmap v9908: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:18.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:18 smithi146 ceph-mon[103278]: pgmap v9909: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[99959]: pgmap v9909: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[91136]: pgmap v9909: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:20 smithi146 ceph-mon[103278]: pgmap v9910: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:20 smithi078 ceph-mon[91136]: pgmap v9910: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:20 smithi078 ceph-mon[99959]: pgmap v9910: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:50:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:50:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:21.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:21.534Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:21.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:22.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:22 smithi078 ceph-mon[99959]: pgmap v9911: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:22.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:22 smithi078 ceph-mon[91136]: pgmap v9911: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:22 smithi146 ceph-mon[103278]: pgmap v9911: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:24 smithi146 ceph-mon[103278]: pgmap v9912: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:24 smithi078 ceph-mon[91136]: pgmap v9912: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:24 smithi078 ceph-mon[99959]: pgmap v9912: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:26 smithi146 ceph-mon[103278]: pgmap v9913: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:26 smithi078 ceph-mon[99959]: pgmap v9913: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:26 smithi078 ceph-mon[91136]: pgmap v9913: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:28.027 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:28.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:28.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:28 smithi078 ceph-mon[99959]: pgmap v9914: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:28.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:28 smithi078 ceph-mon[91136]: pgmap v9914: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:28.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:28 smithi146 ceph-mon[103278]: pgmap v9914: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:30 smithi146 ceph-mon[103278]: pgmap v9915: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:30 smithi078 ceph-mon[91136]: pgmap v9915: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:30 smithi078 ceph-mon[99959]: pgmap v9915: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:50:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:50:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:31.534Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:31.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:31.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:32.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:32 smithi078 ceph-mon[99959]: pgmap v9916: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:32.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:32 smithi078 ceph-mon[91136]: pgmap v9916: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:32 smithi146 ceph-mon[103278]: pgmap v9916: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:34 smithi146 ceph-mon[103278]: pgmap v9917: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[99959]: pgmap v9917: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:34 smithi078 ceph-mon[91136]: pgmap v9917: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:36 smithi146 ceph-mon[103278]: pgmap v9918: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:36 smithi078 ceph-mon[99959]: pgmap v9918: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:36 smithi078 ceph-mon[91136]: pgmap v9918: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:38 smithi146 ceph-mon[103278]: pgmap v9919: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[99959]: pgmap v9919: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[91136]: pgmap v9919: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:40.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:40 smithi146 ceph-mon[103278]: pgmap v9920: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:40 smithi078 ceph-mon[91136]: pgmap v9920: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:50:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:50:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:40 smithi078 ceph-mon[99959]: pgmap v9920: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:41.535Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:41.535Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:41.536Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:42.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:42 smithi146 ceph-mon[103278]: pgmap v9921: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:42 smithi078 ceph-mon[91136]: pgmap v9921: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:42 smithi078 ceph-mon[99959]: pgmap v9921: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:44.474 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:50:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:44 smithi078 ceph-mon[99959]: pgmap v9922: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:44 smithi078 ceph-mon[91136]: pgmap v9922: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:45.004 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 5h - - 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 5h - - 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 11s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 6s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 11s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 6s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 11s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 11s ago 5h 532M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 6s ago 5h 359M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 11s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 11s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 6s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:50:45.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 11s ago 5h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 11s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 11s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 11s ago 5h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 6s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 6s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 6s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 6s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 6s ago 5h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 11s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 11s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:50:45.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 6s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:50:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:44 smithi146 ceph-mon[103278]: pgmap v9922: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:45.314 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:50:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:50:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:50:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:50:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:50:45.316 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:50:45.584 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:50:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:50:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:50:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:50:45.585 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:50:45.900 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:50:45.901 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:50:45.901 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:50:45.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:45 smithi078 ceph-mon[91136]: from='client.86462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:45.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:45 smithi078 ceph-mon[91136]: from='client.86468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:45.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2120654952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:45.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:45 smithi078 ceph-mon[99959]: from='client.86462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:45.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:45 smithi078 ceph-mon[99959]: from='client.86468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:45.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2120654952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:45 smithi146 ceph-mon[103278]: from='client.86462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:45 smithi146 ceph-mon[103278]: from='client.86468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2120654952' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:50:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:46 smithi146 ceph-mon[103278]: from='client.66555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:46 smithi146 ceph-mon[103278]: from='client.86486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:46 smithi146 ceph-mon[103278]: pgmap v9923: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1542319506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[91136]: from='client.66555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[91136]: from='client.86486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[91136]: pgmap v9923: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1542319506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[99959]: from='client.66555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[99959]: from='client.86486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:50:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[99959]: pgmap v9923: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1542319506' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:50:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:48.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:48 smithi078 ceph-mon[99959]: pgmap v9924: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:48.716 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:48 smithi078 ceph-mon[91136]: pgmap v9924: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:48 smithi146 ceph-mon[103278]: pgmap v9924: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:50.685 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:50:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:50:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:50 smithi146 ceph-mon[103278]: pgmap v9925: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:50 smithi078 ceph-mon[91136]: pgmap v9925: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:50 smithi078 ceph-mon[99959]: pgmap v9925: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:50:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:51.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:51.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:50:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:50:51.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:50:52.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:52 smithi078 ceph-mon[91136]: pgmap v9926: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:52.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:52 smithi078 ceph-mon[99959]: pgmap v9926: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:52 smithi146 ceph-mon[103278]: pgmap v9926: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:50:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:54 smithi146 ceph-mon[103278]: pgmap v9927: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:50:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[91136]: pgmap v9927: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:50:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:54 smithi078 ceph-mon[99959]: pgmap v9927: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T01:50:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:56 smithi146 ceph-mon[103278]: pgmap v9928: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:50:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:56 smithi078 ceph-mon[91136]: pgmap v9928: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:50:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:56 smithi078 ceph-mon[99959]: pgmap v9928: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:50:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:50:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:58 smithi146 ceph-mon[103278]: pgmap v9929: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:50:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[91136]: pgmap v9929: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:50:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:59.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[99959]: pgmap v9929: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:50:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:50:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:00 smithi146 ceph-mon[103278]: pgmap v9930: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:51:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:00 smithi078 ceph-mon[91136]: pgmap v9930: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:51:01.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:00 smithi078 ceph-mon[99959]: pgmap v9930: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:51:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:51:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:51:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:01.536Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:01.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:01.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:02 smithi146 ceph-mon[103278]: pgmap v9931: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:51:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:02 smithi078 ceph-mon[99959]: pgmap v9931: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:51:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:02 smithi078 ceph-mon[91136]: pgmap v9931: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T01:51:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:04 smithi146 ceph-mon[103278]: pgmap v9932: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:51:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:04 smithi078 ceph-mon[91136]: pgmap v9932: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:51:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:04 smithi078 ceph-mon[99959]: pgmap v9932: 161 pgs: 161 active+clean; 585 KiB data, 318 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T01:51:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:06 smithi146 ceph-mon[103278]: pgmap v9933: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T01:51:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:06 smithi078 ceph-mon[91136]: pgmap v9933: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T01:51:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:06 smithi078 ceph-mon[99959]: pgmap v9933: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T01:51:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:08.946 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:08 smithi078 ceph-mon[91136]: pgmap v9934: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:51:08.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:08 smithi078 ceph-mon[99959]: pgmap v9934: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:51:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:08 smithi146 ceph-mon[103278]: pgmap v9934: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T01:51:10.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:10 smithi078 ceph-mon[91136]: pgmap v9935: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T01:51:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:51:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:51:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:10 smithi078 ceph-mon[99959]: pgmap v9935: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T01:51:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:10 smithi146 ceph-mon[103278]: pgmap v9935: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T01:51:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:11.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:11.537Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:11.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:12 smithi146 ceph-mon[103278]: pgmap v9936: 161 pgs: 161 active+clean; 585 KiB data, 348 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:51:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:12 smithi078 ceph-mon[99959]: pgmap v9936: 161 pgs: 161 active+clean; 585 KiB data, 348 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:51:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:12 smithi078 ceph-mon[91136]: pgmap v9936: 161 pgs: 161 active+clean; 585 KiB data, 348 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T01:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:14 smithi146 ceph-mon[103278]: pgmap v9937: 161 pgs: 161 active+clean; 585 KiB data, 348 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T01:51:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:14 smithi078 ceph-mon[99959]: pgmap v9937: 161 pgs: 161 active+clean; 585 KiB data, 348 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T01:51:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:14 smithi078 ceph-mon[91136]: pgmap v9937: 161 pgs: 161 active+clean; 585 KiB data, 348 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T01:51:16.199 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 5h - - 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 18s ago 5h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 3s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 18s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 3s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:51:16.725 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 3s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 18s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 3s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 18s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 3s ago 5h 91.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 3s ago 5h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 3s ago 5h 80.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 3s ago 5h 95.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 5h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 5h 86.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 5h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 18s ago 5h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:51:16.726 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 3s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:51:16.727 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:51:16.727 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 3s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:51:16.727 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:51:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:16 smithi146 ceph-mon[103278]: pgmap v9938: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:17.033 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:51:17.034 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:51:17.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:51:17.034 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:51:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:16 smithi078 ceph-mon[99959]: pgmap v9938: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-21T01:51:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:16 smithi078 ceph-mon[91136]: pgmap v9938: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 145 op/s 2024-09-21T01:51:17.300 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:51:17.300 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:51:17.300 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:51:17.300 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:51:17.300 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:51:17.301 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:51:17.605 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:51:17.605 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:17.605 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:17.605 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:17.605 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:17 smithi146 ceph-mon[103278]: from='client.66579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:17 smithi146 ceph-mon[103278]: from='client.66585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:17 smithi146 ceph-mon[103278]: from='client.66591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3646519916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/806724983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[99959]: from='client.66579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[99959]: from='client.66585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[99959]: from='client.66591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3646519916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/806724983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[91136]: from='client.66579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[91136]: from='client.66585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[91136]: from='client.66591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3646519916' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/806724983' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:18 smithi146 ceph-mon[103278]: from='client.86522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:18 smithi146 ceph-mon[103278]: pgmap v9939: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:51:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[99959]: from='client.86522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[99959]: pgmap v9939: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:51:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[91136]: from='client.86522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[91136]: pgmap v9939: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T01:51:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:20 smithi146 ceph-mon[103278]: pgmap v9940: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:51:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:20 smithi078 ceph-mon[99959]: pgmap v9940: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:51:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:20 smithi078 ceph-mon[91136]: pgmap v9940: 161 pgs: 161 active+clean; 585 KiB data, 352 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T01:51:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:51:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:51:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:21.537Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:21.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:21.538Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:22.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:22 smithi146 ceph-mon[103278]: pgmap v9941: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T01:51:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:22 smithi078 ceph-mon[99959]: pgmap v9941: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T01:51:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:22 smithi078 ceph-mon[91136]: pgmap v9941: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T01:51:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:24 smithi146 ceph-mon[103278]: pgmap v9942: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:51:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:24 smithi078 ceph-mon[99959]: pgmap v9942: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:51:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:24 smithi078 ceph-mon[91136]: pgmap v9942: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T01:51:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:26 smithi146 ceph-mon[103278]: pgmap v9943: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:51:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:26 smithi078 ceph-mon[99959]: pgmap v9943: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:51:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:26 smithi078 ceph-mon[91136]: pgmap v9943: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 98 op/s 2024-09-21T01:51:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:28 smithi146 ceph-mon[103278]: pgmap v9944: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T01:51:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:28 smithi078 ceph-mon[99959]: pgmap v9944: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T01:51:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:28 smithi078 ceph-mon[91136]: pgmap v9944: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T01:51:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:30.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:30 smithi078 ceph-mon[91136]: pgmap v9945: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T01:51:30.973 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:51:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:51:30.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:30 smithi078 ceph-mon[99959]: pgmap v9945: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T01:51:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:30 smithi146 ceph-mon[103278]: pgmap v9945: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 63 op/s 2024-09-21T01:51:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:31.538Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:31.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:31.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:32 smithi146 ceph-mon[103278]: pgmap v9946: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:51:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:32 smithi078 ceph-mon[99959]: pgmap v9946: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:51:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:32 smithi078 ceph-mon[91136]: pgmap v9946: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T01:51:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:34 smithi146 ceph-mon[103278]: pgmap v9947: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:51:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[99959]: pgmap v9947: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:51:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[91136]: pgmap v9947: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T01:51:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:36 smithi146 ceph-mon[103278]: pgmap v9948: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:36 smithi078 ceph-mon[91136]: pgmap v9948: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:36 smithi078 ceph-mon[99959]: pgmap v9948: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:38 smithi146 ceph-mon[103278]: pgmap v9949: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[99959]: pgmap v9949: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[91136]: pgmap v9949: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:40 smithi146 ceph-mon[103278]: pgmap v9950: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:40 smithi078 ceph-mon[99959]: pgmap v9950: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:40 smithi078 ceph-mon[91136]: pgmap v9950: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:51:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:51:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:41.539Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:41.539Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:41.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:42.837 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:42 smithi146 ceph-mon[103278]: pgmap v9951: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:42 smithi078 ceph-mon[99959]: pgmap v9951: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:42 smithi078 ceph-mon[91136]: pgmap v9951: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:43.852 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:44.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:44 smithi146 ceph-mon[103278]: pgmap v9952: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:44 smithi078 ceph-mon[91136]: pgmap v9952: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:44 smithi078 ceph-mon[99959]: pgmap v9952: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:45 smithi146 ceph-mon[103278]: pgmap v9953: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:45 smithi078 ceph-mon[99959]: pgmap v9953: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:45 smithi078 ceph-mon[91136]: pgmap v9953: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:47.900 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:51:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:47 smithi146 ceph-mon[103278]: pgmap v9954: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:47 smithi078 ceph-mon[91136]: pgmap v9954: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:47 smithi078 ceph-mon[99959]: pgmap v9954: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 5h - - 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 5h - - 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 14s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 9s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 14s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:51:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 9s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 14s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 14s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 9s ago 5h 362M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 14s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 14s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 9s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 14s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 14s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 14s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 14s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 9s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 9s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 9s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:51:48.428 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 9s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:51:48.429 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 9s ago 5h 70.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:51:48.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 14s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:51:48.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:51:48.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 14s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:51:48.429 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 9s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:51:48.738 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:51:48.738 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:51:48.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:51:48.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:48.738 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:51:48.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:51:48.739 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:51:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:51:49.012 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:51:49.012 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:51:49.012 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:51:49.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[91136]: from='client.86534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[91136]: from='client.86540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[91136]: from='client.86546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1042975216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:49.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[99959]: from='client.86534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[99959]: from='client.86540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[99959]: from='client.86546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1042975216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:48 smithi146 ceph-mon[103278]: from='client.86534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:48 smithi146 ceph-mon[103278]: from='client.86540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:48 smithi146 ceph-mon[103278]: from='client.86546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1042975216' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:51:49.320 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:51:49.320 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:49.320 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:51:49.321 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:49.321 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:51:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:49 smithi146 ceph-mon[103278]: from='client.86558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2210622012' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:49 smithi146 ceph-mon[103278]: pgmap v9955: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:49 smithi078 ceph-mon[91136]: from='client.86558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2210622012' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:49 smithi078 ceph-mon[91136]: pgmap v9955: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:49 smithi078 ceph-mon[99959]: from='client.86558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:51:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2210622012' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:51:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:49 smithi078 ceph-mon[99959]: pgmap v9955: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:50.662 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:51:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:51:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:51.539Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:51.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:51:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:51:51.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:51:52.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:52 smithi078 ceph-mon[99959]: pgmap v9956: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:52.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:52 smithi078 ceph-mon[91136]: pgmap v9956: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:52 smithi146 ceph-mon[103278]: pgmap v9956: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:54 smithi146 ceph-mon[103278]: pgmap v9957: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[91136]: pgmap v9957: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[99959]: pgmap v9957: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:51:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:55 smithi146 ceph-mon[103278]: pgmap v9958: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:55 smithi078 ceph-mon[91136]: pgmap v9958: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:55 smithi078 ceph-mon[99959]: pgmap v9958: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:57.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:51:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:57 smithi146 ceph-mon[103278]: pgmap v9959: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:57 smithi078 ceph-mon[99959]: pgmap v9959: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:57 smithi078 ceph-mon[91136]: pgmap v9959: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:51:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:51:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:51:59 smithi146 ceph-mon[103278]: pgmap v9960: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:51:59 smithi078 ceph-mon[91136]: pgmap v9960: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:51:59 smithi078 ceph-mon[99959]: pgmap v9960: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:52:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:52:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:01.540Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:01.540Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:01.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:02 smithi146 ceph-mon[103278]: pgmap v9961: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:03.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:02 smithi078 ceph-mon[91136]: pgmap v9961: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:03.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:02 smithi078 ceph-mon[99959]: pgmap v9961: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:04 smithi146 ceph-mon[103278]: pgmap v9962: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:04 smithi078 ceph-mon[99959]: pgmap v9962: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:04 smithi078 ceph-mon[91136]: pgmap v9962: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:06 smithi146 ceph-mon[103278]: pgmap v9963: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:06 smithi078 ceph-mon[91136]: pgmap v9963: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:06 smithi078 ceph-mon[99959]: pgmap v9963: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:08 smithi146 ceph-mon[103278]: pgmap v9964: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:08 smithi078 ceph-mon[91136]: pgmap v9964: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:08 smithi078 ceph-mon[99959]: pgmap v9964: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:10 smithi146 ceph-mon[103278]: pgmap v9965: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:10 smithi078 ceph-mon[91136]: pgmap v9965: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:52:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:52:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:10 smithi078 ceph-mon[99959]: pgmap v9965: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:11.541 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:11.541Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:11.541 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:11.541Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:11.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:12 smithi146 ceph-mon[103278]: pgmap v9966: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:12 smithi078 ceph-mon[99959]: pgmap v9966: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:12 smithi078 ceph-mon[91136]: pgmap v9966: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:14.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:14.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:14 smithi146 ceph-mon[103278]: pgmap v9967: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[99959]: pgmap v9967: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[91136]: pgmap v9967: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:16 smithi146 ceph-mon[103278]: pgmap v9968: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:16 smithi078 ceph-mon[91136]: pgmap v9968: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:16 smithi078 ceph-mon[99959]: pgmap v9968: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:18 smithi146 ceph-mon[103278]: pgmap v9969: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:18 smithi078 ceph-mon[99959]: pgmap v9969: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:18 smithi078 ceph-mon[91136]: pgmap v9969: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:19.612 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:52:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 5h - - 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 5h - - 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 6s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 1s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 6s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:52:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 1s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 6s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 6s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 1s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 6s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 6s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 1s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 6s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 6s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 6s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 6s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:52:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:52:20.142 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 1s ago 5h 70.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:52:20.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 6s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:52:20.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:52:20.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 6s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:52:20.142 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:52:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:20.451 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:52:20.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:52:20.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:52:20.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:20.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:52:20.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:52:20.452 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:52:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:52:20.722 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:52:20.722 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:52:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:52:20.723 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:52:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:52:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-mon[91136]: from='client.86570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-mon[91136]: pgmap v9970: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1193558473' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-mon[99959]: from='client.86570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-mon[99959]: pgmap v9970: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1193558473' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:21.036 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:52:21.037 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:21.037 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:21.037 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:21.037 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:20 smithi146 ceph-mon[103278]: from='client.86570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:20 smithi146 ceph-mon[103278]: pgmap v9970: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1193558473' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:21.775 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:21.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:21.776 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:21.542Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:21.776 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:21.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:22.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[99959]: from='client.86576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[99959]: from='client.86582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[99959]: from='client.86594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4198883177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[91136]: from='client.86576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[91136]: from='client.86582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[91136]: from='client.86594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4198883177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:21 smithi146 ceph-mon[103278]: from='client.86576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:21 smithi146 ceph-mon[103278]: from='client.86582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:21 smithi146 ceph-mon[103278]: from='client.86594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4198883177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:22 smithi078 ceph-mon[91136]: pgmap v9971: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:22 smithi078 ceph-mon[99959]: pgmap v9971: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:22 smithi146 ceph-mon[103278]: pgmap v9971: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:23.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:23 smithi146 ceph-mon[103278]: pgmap v9972: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:23 smithi078 ceph-mon[91136]: pgmap v9972: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:23 smithi078 ceph-mon[99959]: pgmap v9972: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:26 smithi146 ceph-mon[103278]: pgmap v9973: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:26 smithi078 ceph-mon[99959]: pgmap v9973: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:26 smithi078 ceph-mon[91136]: pgmap v9973: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:28 smithi146 ceph-mon[103278]: pgmap v9974: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:28 smithi078 ceph-mon[91136]: pgmap v9974: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:28 smithi078 ceph-mon[99959]: pgmap v9974: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:30 smithi146 ceph-mon[103278]: pgmap v9975: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:30 smithi078 ceph-mon[91136]: pgmap v9975: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:30 smithi078 ceph-mon[99959]: pgmap v9975: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:52:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:52:31.831 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:31.542Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:31.831 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:31.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:31.831 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:31.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:32 smithi146 ceph-mon[103278]: pgmap v9976: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:32 smithi078 ceph-mon[99959]: pgmap v9976: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:32 smithi078 ceph-mon[91136]: pgmap v9976: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:34 smithi146 ceph-mon[103278]: pgmap v9977: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[91136]: pgmap v9977: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[99959]: pgmap v9977: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:36 smithi146 ceph-mon[103278]: pgmap v9978: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:36 smithi078 ceph-mon[91136]: pgmap v9978: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:36 smithi078 ceph-mon[99959]: pgmap v9978: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:38 smithi146 ceph-mon[103278]: pgmap v9979: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:38 smithi078 ceph-mon[99959]: pgmap v9979: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:38 smithi078 ceph-mon[91136]: pgmap v9979: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:39 smithi146 ceph-mon[103278]: pgmap v9980: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[99959]: pgmap v9980: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:39 smithi078 ceph-mon[91136]: pgmap v9980: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:52:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:52:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:41.543Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:41.543Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:41.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:42 smithi146 ceph-mon[103278]: pgmap v9981: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:42 smithi078 ceph-mon[91136]: pgmap v9981: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:42 smithi078 ceph-mon[99959]: pgmap v9981: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:44 smithi146 ceph-mon[103278]: pgmap v9982: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:44 smithi078 ceph-mon[91136]: pgmap v9982: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:44 smithi078 ceph-mon[99959]: pgmap v9982: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:46 smithi146 ceph-mon[103278]: pgmap v9983: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:46 smithi078 ceph-mon[91136]: pgmap v9983: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:46 smithi078 ceph-mon[99959]: pgmap v9983: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:48 smithi146 ceph-mon[103278]: pgmap v9984: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:48 smithi078 ceph-mon[91136]: pgmap v9984: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:48 smithi078 ceph-mon[99959]: pgmap v9984: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:50 smithi146 ceph-mon[103278]: pgmap v9985: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:50 smithi078 ceph-mon[91136]: pgmap v9985: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:50 smithi078 ceph-mon[99959]: pgmap v9985: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:52:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:52:51.331 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:52:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:51.544Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:51.544Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:52:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:52:51.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:52:51.867 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:52:51.867 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 5h - - 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 5h - - 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 17s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 12s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 17s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 12s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 17s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 17s ago 5h 530M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 17s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 17s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 12s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 17s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 17s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 17s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 17s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 12s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 17s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 17s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 5h 89.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:52:52.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:52:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:52:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:52:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:52:52.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:52:52.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:52:52.488 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:52:52.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:52:52.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[99959]: from='client.66687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:52.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[99959]: from='client.86612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:52.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[99959]: pgmap v9986: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:52.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3946877471' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:52.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[91136]: from='client.66687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:52.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[91136]: from='client.86612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:52.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[91136]: pgmap v9986: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:52.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3946877471' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:52:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:52:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:52:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:52 smithi146 ceph-mon[103278]: from='client.66687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:52 smithi146 ceph-mon[103278]: from='client.86612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:52 smithi146 ceph-mon[103278]: pgmap v9986: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3946877471' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:52:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:53 smithi078 ceph-mon[91136]: from='client.66699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:53 smithi078 ceph-mon[91136]: from='client.66711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1556548840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:53 smithi078 ceph-mon[99959]: from='client.66699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:53 smithi078 ceph-mon[99959]: from='client.66711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1556548840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:53 smithi146 ceph-mon[103278]: from='client.66699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:53 smithi146 ceph-mon[103278]: from='client.66711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:52:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1556548840' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:52:54.850 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[99959]: pgmap v9987: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[91136]: pgmap v9987: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:54.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:54 smithi146 ceph-mon[103278]: pgmap v9987: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:52:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:52:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:55 smithi146 ceph-mon[103278]: pgmap v9988: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:55 smithi078 ceph-mon[99959]: pgmap v9988: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:55 smithi078 ceph-mon[91136]: pgmap v9988: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:57.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:52:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:57 smithi146 ceph-mon[103278]: pgmap v9989: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:57 smithi078 ceph-mon[91136]: pgmap v9989: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:57 smithi078 ceph-mon[99959]: pgmap v9989: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:52:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:59.288 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:59.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[91136]: pgmap v9990: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-mon[99959]: pgmap v9990: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:00 smithi146 ceph-mon[103278]: pgmap v9990: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:53:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:53:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:01.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:01.545Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:01.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:02.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:02 smithi078 ceph-mon[99959]: pgmap v9991: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:02 smithi078 ceph-mon[91136]: pgmap v9991: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:02 smithi146 ceph-mon[103278]: pgmap v9991: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:04 smithi146 ceph-mon[103278]: pgmap v9992: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:04 smithi078 ceph-mon[91136]: pgmap v9992: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:04 smithi078 ceph-mon[99959]: pgmap v9992: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:06.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:06 smithi146 ceph-mon[103278]: pgmap v9993: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:06 smithi078 ceph-mon[91136]: pgmap v9993: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:06 smithi078 ceph-mon[99959]: pgmap v9993: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:07.854 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:08 smithi146 ceph-mon[103278]: pgmap v9994: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:08 smithi078 ceph-mon[91136]: pgmap v9994: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:08 smithi078 ceph-mon[99959]: pgmap v9994: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:10 smithi146 ceph-mon[103278]: pgmap v9995: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:10 smithi078 ceph-mon[91136]: pgmap v9995: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:10 smithi078 ceph-mon[99959]: pgmap v9995: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:53:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:53:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:11.545Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:11.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:11.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:12.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:12 smithi078 ceph-mon[91136]: pgmap v9996: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:12.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:12 smithi078 ceph-mon[99959]: pgmap v9996: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:12 smithi146 ceph-mon[103278]: pgmap v9996: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:14 smithi146 ceph-mon[103278]: pgmap v9997: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[91136]: pgmap v9997: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[99959]: pgmap v9997: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:16 smithi146 ceph-mon[103278]: pgmap v9998: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:16 smithi078 ceph-mon[91136]: pgmap v9998: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:16 smithi078 ceph-mon[99959]: pgmap v9998: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:18 smithi146 ceph-mon[103278]: pgmap v9999: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:18 smithi078 ceph-mon[91136]: pgmap v9999: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:18 smithi078 ceph-mon[99959]: pgmap v9999: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:20 smithi146 ceph-mon[103278]: pgmap v10000: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[91136]: pgmap v10000: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-mon[99959]: pgmap v10000: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:53:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:53:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:21.546Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:21.546Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:21.547Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:22 smithi146 ceph-mon[103278]: pgmap v10001: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:22 smithi078 ceph-mon[91136]: pgmap v10001: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:22 smithi078 ceph-mon[99959]: pgmap v10001: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:23.120 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:53:23.645 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 5h - - 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 5h - - 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 9s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 4s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 9s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 4s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 9s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 9s ago 5h 531M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 4s ago 5h 361M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 9s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:53:23.646 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 9s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 4s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 9s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 9s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 9s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 9s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 4s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 4s ago 5h 70.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 9s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 9s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:53:23.647 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:53:23.956 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:53:23.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:53:23.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:53:23.959 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:53:24.227 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:53:24.228 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:53:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:53:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:53:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:24.538 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:24.538 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:24 smithi146 ceph-mon[103278]: from='client.66723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.021 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:24 smithi146 ceph-mon[103278]: from='client.66729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:24 smithi146 ceph-mon[103278]: from='client.66735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:24 smithi146 ceph-mon[103278]: pgmap v10002: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:25.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1907886630' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:25.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4294309453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[91136]: from='client.66723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[91136]: from='client.66729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[91136]: from='client.66735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[91136]: pgmap v10002: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1907886630' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4294309453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[99959]: from='client.66723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[99959]: from='client.66729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[99959]: from='client.66735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[99959]: pgmap v10002: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1907886630' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4294309453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:25.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:25 smithi146 ceph-mon[103278]: from='client.86666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:25 smithi078 ceph-mon[91136]: from='client.86666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:25 smithi078 ceph-mon[99959]: from='client.86666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:26.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:26 smithi146 ceph-mon[103278]: pgmap v10003: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:26 smithi078 ceph-mon[91136]: pgmap v10003: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:26 smithi078 ceph-mon[99959]: pgmap v10003: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:28 smithi146 ceph-mon[103278]: pgmap v10004: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:28 smithi078 ceph-mon[99959]: pgmap v10004: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:28 smithi078 ceph-mon[91136]: pgmap v10004: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:30 smithi146 ceph-mon[103278]: pgmap v10005: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:30 smithi078 ceph-mon[91136]: pgmap v10005: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:30 smithi078 ceph-mon[99959]: pgmap v10005: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:53:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:53:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:31.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:31.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:31.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:32 smithi146 ceph-mon[103278]: pgmap v10006: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:32 smithi078 ceph-mon[99959]: pgmap v10006: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:32 smithi078 ceph-mon[91136]: pgmap v10006: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:35.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[99959]: pgmap v10007: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:35.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[91136]: pgmap v10007: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:35.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.008 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:34 smithi146 ceph-mon[103278]: pgmap v10007: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:36 smithi146 ceph-mon[103278]: pgmap v10008: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:36 smithi078 ceph-mon[91136]: pgmap v10008: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:36 smithi078 ceph-mon[99959]: pgmap v10008: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:38 smithi146 ceph-mon[103278]: pgmap v10009: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:38 smithi078 ceph-mon[91136]: pgmap v10009: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:38 smithi078 ceph-mon[99959]: pgmap v10009: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[91136]: pgmap v10010: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.613 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-mon[99959]: pgmap v10010: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:40 smithi146 ceph-mon[103278]: pgmap v10010: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:53:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:53:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:41.547Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:41.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:41.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:42 smithi146 ceph-mon[103278]: pgmap v10011: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:42 smithi078 ceph-mon[99959]: pgmap v10011: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:42 smithi078 ceph-mon[91136]: pgmap v10011: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:44 smithi146 ceph-mon[103278]: pgmap v10012: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:44 smithi078 ceph-mon[91136]: pgmap v10012: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:44 smithi078 ceph-mon[99959]: pgmap v10012: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:46 smithi146 ceph-mon[103278]: pgmap v10013: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:46 smithi078 ceph-mon[91136]: pgmap v10013: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:46 smithi078 ceph-mon[99959]: pgmap v10013: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:48 smithi146 ceph-mon[103278]: pgmap v10014: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:48 smithi078 ceph-mon[99959]: pgmap v10014: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:48 smithi078 ceph-mon[91136]: pgmap v10014: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:50 smithi146 ceph-mon[103278]: pgmap v10015: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:50 smithi078 ceph-mon[99959]: pgmap v10015: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:50 smithi078 ceph-mon[91136]: pgmap v10015: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:53:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:53:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:51.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:51.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:53:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:53:51.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:53:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:52 smithi146 ceph-mon[103278]: pgmap v10016: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:52 smithi078 ceph-mon[91136]: pgmap v10016: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:52 smithi078 ceph-mon[99959]: pgmap v10016: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:54.839 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:53:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:54 smithi146 ceph-mon[103278]: pgmap v10017: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[99959]: pgmap v10017: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[91136]: pgmap v10017: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:53:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 5h - - 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 5h - - 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 0s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 16s ago 5h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 0s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 16s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 0s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:53:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 0s ago 5h 531M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 16s ago 5h 363M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 0s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 0s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 16s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 0s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 0s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 0s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 0s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 16s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 16s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 16s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 16s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 5h 70.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 0s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:53:55.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:53:55.383 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 0s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:53:55.383 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:53:55.732 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:53:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:53:55.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:53:55.733 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:53:55.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1293717539' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:55.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1293717539' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:53:56.011 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:53:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1293717539' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:53:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:53:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:56.343 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:53:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:56.344 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:53:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:56 smithi146 ceph-mon[103278]: from='client.66759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:56 smithi146 ceph-mon[103278]: from='client.86684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:56 smithi146 ceph-mon[103278]: from='client.86690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:56 smithi146 ceph-mon[103278]: pgmap v10018: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:57.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3147917900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:57.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[99959]: from='client.66759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[99959]: from='client.86684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[99959]: from='client.86690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[99959]: pgmap v10018: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:57.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3147917900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:57.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[91136]: from='client.66759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[91136]: from='client.86684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[91136]: from='client.86690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:57.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[91136]: pgmap v10018: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:57.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3147917900' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:53:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:57 smithi146 ceph-mon[103278]: from='client.66783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:57 smithi078 ceph-mon[91136]: from='client.66783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:57 smithi078 ceph-mon[99959]: from='client.66783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:53:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:53:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:58 smithi146 ceph-mon[103278]: pgmap v10019: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:53:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:58 smithi078 ceph-mon[91136]: pgmap v10019: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:53:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:58 smithi078 ceph-mon[99959]: pgmap v10019: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:53:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:53:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:00 smithi146 ceph-mon[103278]: pgmap v10020: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:54:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[99959]: pgmap v10020: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:00 smithi078 ceph-mon[91136]: pgmap v10020: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:01.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:01.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:01.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:02 smithi146 ceph-mon[103278]: pgmap v10021: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:02 smithi078 ceph-mon[91136]: pgmap v10021: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:02 smithi078 ceph-mon[99959]: pgmap v10021: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:04 smithi146 ceph-mon[103278]: pgmap v10022: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:04 smithi078 ceph-mon[99959]: pgmap v10022: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:04 smithi078 ceph-mon[91136]: pgmap v10022: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:06.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:06 smithi146 ceph-mon[103278]: pgmap v10023: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:06 smithi078 ceph-mon[99959]: pgmap v10023: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:06 smithi078 ceph-mon[91136]: pgmap v10023: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:07.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:08 smithi146 ceph-mon[103278]: pgmap v10024: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:08 smithi078 ceph-mon[99959]: pgmap v10024: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:08 smithi078 ceph-mon[91136]: pgmap v10024: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:10 smithi146 ceph-mon[103278]: pgmap v10025: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:10 smithi078 ceph-mon[99959]: pgmap v10025: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:10 smithi078 ceph-mon[91136]: pgmap v10025: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:54:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:54:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:11.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:11.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:11.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:12 smithi146 ceph-mon[103278]: pgmap v10026: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:12 smithi078 ceph-mon[99959]: pgmap v10026: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:12 smithi078 ceph-mon[91136]: pgmap v10026: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:14 smithi146 ceph-mon[103278]: pgmap v10027: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:14 smithi078 ceph-mon[91136]: pgmap v10027: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:14 smithi078 ceph-mon[99959]: pgmap v10027: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:16 smithi078 ceph-mon[91136]: pgmap v10028: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:16 smithi078 ceph-mon[99959]: pgmap v10028: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:16 smithi146 ceph-mon[103278]: pgmap v10028: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:17.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:17.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:18 smithi146 ceph-mon[103278]: pgmap v10029: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:18 smithi078 ceph-mon[91136]: pgmap v10029: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:18 smithi078 ceph-mon[99959]: pgmap v10029: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:20 smithi146 ceph-mon[103278]: pgmap v10030: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:54:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:54:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[91136]: pgmap v10030: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:20 smithi078 ceph-mon[99959]: pgmap v10030: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:21.550Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:21.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:21.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:22.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:22 smithi078 ceph-mon[99959]: pgmap v10031: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:22 smithi078 ceph-mon[91136]: pgmap v10031: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:22 smithi146 ceph-mon[103278]: pgmap v10031: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:24 smithi146 ceph-mon[103278]: pgmap v10032: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:24 smithi078 ceph-mon[91136]: pgmap v10032: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:24 smithi078 ceph-mon[99959]: pgmap v10032: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 01:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T01:54:25.847165097Z level=info msg="Completed cleanup jobs" duration=11.959954ms 2024-09-21T01:54:26.634 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:54:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:26 smithi146 ceph-mon[103278]: pgmap v10033: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:26 smithi078 ceph-mon[91136]: pgmap v10033: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:26 smithi078 ceph-mon[99959]: pgmap v10033: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:27.163 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:54:27.163 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 5h - - 2024-09-21T01:54:27.163 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 5h - - 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 12s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 7s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 12s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 7s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 12s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 12s ago 5h 532M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 7s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 12s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 12s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 7s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 12s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 12s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 12s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 12s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 7s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 7s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:54:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:54:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:54:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 7s ago 5h 70.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:54:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 12s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:54:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:54:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 12s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:54:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:54:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:54:27.476 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:54:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:54:27.747 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:54:27.747 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:54:27.747 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:54:27.747 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:54:27.747 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:54:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:27 smithi146 ceph-mon[103278]: from='client.66795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/823253020' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:27 smithi078 ceph-mon[91136]: from='client.66795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/823253020' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:27 smithi078 ceph-mon[99959]: from='client.66795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/823253020' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:54:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:54:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:28.057 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:28.058 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:28.058 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:28 smithi146 ceph-mon[103278]: from='client.66801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:28 smithi146 ceph-mon[103278]: from='client.66807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:28 smithi146 ceph-mon[103278]: pgmap v10034: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:28 smithi146 ceph-mon[103278]: from='client.66819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/457029183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[91136]: from='client.66801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[91136]: from='client.66807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[91136]: pgmap v10034: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[91136]: from='client.66819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/457029183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[99959]: from='client.66801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[99959]: from='client.66807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[99959]: pgmap v10034: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[99959]: from='client.66819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/457029183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:54:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:31.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:30 smithi146 ceph-mon[103278]: pgmap v10035: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:30 smithi078 ceph-mon[91136]: pgmap v10035: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:30 smithi078 ceph-mon[99959]: pgmap v10035: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:54:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:54:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:31.551Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:31.551Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:31.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:32 smithi146 ceph-mon[103278]: pgmap v10036: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:32 smithi078 ceph-mon[91136]: pgmap v10036: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:32 smithi078 ceph-mon[99959]: pgmap v10036: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:34 smithi146 ceph-mon[103278]: pgmap v10037: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:34 smithi078 ceph-mon[91136]: pgmap v10037: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:34 smithi078 ceph-mon[99959]: pgmap v10037: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:36.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[99959]: pgmap v10038: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:35 smithi078 ceph-mon[91136]: pgmap v10038: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:35 smithi146 ceph-mon[103278]: pgmap v10038: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:37 smithi078 ceph-mon[91136]: pgmap v10039: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:37 smithi078 ceph-mon[99959]: pgmap v10039: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:37 smithi146 ceph-mon[103278]: pgmap v10039: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:40 smithi146 ceph-mon[103278]: pgmap v10040: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:54:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[99959]: pgmap v10040: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[91136]: pgmap v10040: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:41.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:41.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:41.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:42 smithi146 ceph-mon[103278]: pgmap v10041: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:42 smithi078 ceph-mon[91136]: pgmap v10041: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:42 smithi078 ceph-mon[99959]: pgmap v10041: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:44 smithi146 ceph-mon[103278]: pgmap v10042: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:44 smithi078 ceph-mon[91136]: pgmap v10042: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:45.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:44 smithi078 ceph-mon[99959]: pgmap v10042: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:46 smithi146 ceph-mon[103278]: pgmap v10043: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:46 smithi078 ceph-mon[99959]: pgmap v10043: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:46 smithi078 ceph-mon[91136]: pgmap v10043: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:47.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:48 smithi146 ceph-mon[103278]: pgmap v10044: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:48 smithi078 ceph-mon[99959]: pgmap v10044: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:48 smithi078 ceph-mon[91136]: pgmap v10044: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:50 smithi146 ceph-mon[103278]: pgmap v10045: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:50 smithi078 ceph-mon[99959]: pgmap v10045: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:50 smithi078 ceph-mon[91136]: pgmap v10045: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:54:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:54:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:51.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:51.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:54:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:54:51.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:54:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:52 smithi146 ceph-mon[103278]: pgmap v10046: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:52 smithi078 ceph-mon[91136]: pgmap v10046: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:52 smithi078 ceph-mon[99959]: pgmap v10046: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:54 smithi146 ceph-mon[103278]: pgmap v10047: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:54 smithi078 ceph-mon[99959]: pgmap v10047: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:54 smithi078 ceph-mon[91136]: pgmap v10047: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:54:56.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[99959]: pgmap v10048: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:56.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:56 smithi078 ceph-mon[91136]: pgmap v10048: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:54:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:56 smithi146 ceph-mon[103278]: pgmap v10048: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:57.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:54:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:58 smithi146 ceph-mon[103278]: pgmap v10049: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:58.321 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:58 smithi078 ceph-mon[91136]: pgmap v10049: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:58.322 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:58 smithi078 ceph-mon[99959]: pgmap v10049: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:54:58.349 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 5h - - 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 19s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 3s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 19s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 3s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 3s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 19s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 3s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 19s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:54:58.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 3s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 3s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 3s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 3s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 19s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 19s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 19s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 19s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 19s ago 5h 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 3s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 3s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:54:58.893 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:54:59.213 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:54:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:54:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:54:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:54:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:54:59.215 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:54:59.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:59 smithi078 ceph-mon[99959]: from='client.66831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:59.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:59.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:54:59 smithi078 ceph-mon[99959]: from='client.86756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:59.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:59 smithi078 ceph-mon[91136]: from='client.66831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:59.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:59.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:54:59 smithi078 ceph-mon[91136]: from='client.86756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:59.486 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:54:59.487 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:54:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:59 smithi146 ceph-mon[103278]: from='client.66831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:54:59 smithi146 ceph-mon[103278]: from='client.86756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:54:59.796 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:54:59.797 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:59.797 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:54:59.797 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:54:59.797 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='client.66843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2163520259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='client.86774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: pgmap v10050: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2145536384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='client.66843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2163520259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='client.86774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: pgmap v10050: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2145536384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='client.66843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2163520259' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='client.86774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: pgmap v10050: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2145536384' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:55:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:55:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:01.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:01.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:01.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:02 smithi146 ceph-mon[103278]: pgmap v10051: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:02 smithi078 ceph-mon[91136]: pgmap v10051: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:02 smithi078 ceph-mon[99959]: pgmap v10051: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:04 smithi146 ceph-mon[103278]: pgmap v10052: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:04 smithi078 ceph-mon[91136]: pgmap v10052: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:04 smithi078 ceph-mon[99959]: pgmap v10052: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:06 smithi146 ceph-mon[103278]: pgmap v10053: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:06 smithi078 ceph-mon[91136]: pgmap v10053: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:06 smithi078 ceph-mon[99959]: pgmap v10053: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:08 smithi146 ceph-mon[103278]: pgmap v10054: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:08 smithi078 ceph-mon[91136]: pgmap v10054: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:08 smithi078 ceph-mon[99959]: pgmap v10054: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:10.863 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:10 smithi146 ceph-mon[103278]: pgmap v10055: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:10 smithi078 ceph-mon[91136]: pgmap v10055: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:55:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:55:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:10 smithi078 ceph-mon[99959]: pgmap v10055: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:11.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:11.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:11.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:12 smithi146 ceph-mon[103278]: pgmap v10056: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:12 smithi078 ceph-mon[91136]: pgmap v10056: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:13.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:12 smithi078 ceph-mon[99959]: pgmap v10056: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:14 smithi146 ceph-mon[103278]: pgmap v10057: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:14 smithi078 ceph-mon[99959]: pgmap v10057: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:14 smithi078 ceph-mon[91136]: pgmap v10057: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[99959]: pgmap v10058: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:16 smithi078 ceph-mon[91136]: pgmap v10058: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:16 smithi146 ceph-mon[103278]: pgmap v10058: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:17.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:17.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:17.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:18 smithi146 ceph-mon[103278]: pgmap v10059: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:18.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:18 smithi078 ceph-mon[91136]: pgmap v10059: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:18.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:18 smithi078 ceph-mon[99959]: pgmap v10059: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:20.742 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:55:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:55:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:20 smithi146 ceph-mon[103278]: pgmap v10060: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[91136]: pgmap v10060: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[99959]: pgmap v10060: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:21.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:21.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:21.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:22 smithi146 ceph-mon[103278]: pgmap v10061: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:22 smithi078 ceph-mon[91136]: pgmap v10061: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:22 smithi078 ceph-mon[99959]: pgmap v10061: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:24 smithi146 ceph-mon[103278]: pgmap v10062: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:24 smithi078 ceph-mon[91136]: pgmap v10062: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:24 smithi078 ceph-mon[99959]: pgmap v10062: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:26 smithi146 ceph-mon[103278]: pgmap v10063: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:26 smithi078 ceph-mon[91136]: pgmap v10063: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:26 smithi078 ceph-mon[99959]: pgmap v10063: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:29.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:28 smithi146 ceph-mon[103278]: pgmap v10064: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:29.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:28 smithi078 ceph-mon[99959]: pgmap v10064: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:28 smithi078 ceph-mon[91136]: pgmap v10064: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:30.092 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 5h - - 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 15s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 15s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 15s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 15s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 10s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 15s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:55:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 15s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 15s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 15s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 15s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 15s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 10s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 10s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 10s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 10s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 10s ago 5h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 15s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 15s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:55:30.620 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:55:30.850 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:30 smithi146 ceph-mon[103278]: pgmap v10065: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:30.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:30 smithi078 ceph-mon[99959]: pgmap v10065: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:30.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:30 smithi078 ceph-mon[91136]: pgmap v10065: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:30.928 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:55:30] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:55:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:55:30.929 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:55:31.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:55:31.233 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:55:31.233 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:55:31.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:55:31.541 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:55:31.542 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:55:31.542 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:55:31.542 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:31.542 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:55:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[91136]: from='client.86786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[91136]: from='client.86792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[91136]: from='client.86798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1948997273' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/870983170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[99959]: from='client.86786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[99959]: from='client.86792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[99959]: from='client.86798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1948997273' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/870983170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:31.556Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:31.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:31.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:31 smithi146 ceph-mon[103278]: from='client.86786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:31 smithi146 ceph-mon[103278]: from='client.86792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:31 smithi146 ceph-mon[103278]: from='client.86798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1948997273' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:55:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/870983170' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:55:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:32 smithi078 ceph-mon[99959]: from='client.86810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:32 smithi078 ceph-mon[99959]: pgmap v10066: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:32 smithi078 ceph-mon[91136]: from='client.86810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:32 smithi078 ceph-mon[91136]: pgmap v10066: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:32 smithi146 ceph-mon[103278]: from='client.86810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:55:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:32 smithi146 ceph-mon[103278]: pgmap v10066: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:35.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:34 smithi078 ceph-mon[91136]: pgmap v10067: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:34 smithi078 ceph-mon[99959]: pgmap v10067: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:34 smithi146 ceph-mon[103278]: pgmap v10067: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:36 smithi146 ceph-mon[103278]: pgmap v10068: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[91136]: pgmap v10068: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:36 smithi078 ceph-mon[99959]: pgmap v10068: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:38 smithi078 ceph-mon[91136]: pgmap v10069: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:38 smithi078 ceph-mon[99959]: pgmap v10069: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:38.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:38 smithi146 ceph-mon[103278]: pgmap v10069: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:40.964 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:55:40] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[99959]: pgmap v10070: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[91136]: pgmap v10070: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:40.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:40 smithi146 ceph-mon[103278]: pgmap v10070: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:41.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:41.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:41.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:42 smithi146 ceph-mon[103278]: pgmap v10071: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:42 smithi078 ceph-mon[91136]: pgmap v10071: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:43.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:42 smithi078 ceph-mon[99959]: pgmap v10071: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:44 smithi146 ceph-mon[103278]: pgmap v10072: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:44 smithi078 ceph-mon[91136]: pgmap v10072: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:44 smithi078 ceph-mon[99959]: pgmap v10072: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:46 smithi146 ceph-mon[103278]: pgmap v10073: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:46 smithi078 ceph-mon[99959]: pgmap v10073: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:46 smithi078 ceph-mon[91136]: pgmap v10073: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:48 smithi078 ceph-mon[99959]: pgmap v10074: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:48 smithi078 ceph-mon[91136]: pgmap v10074: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:48 smithi146 ceph-mon[103278]: pgmap v10074: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:50.983 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:50 smithi146 ceph-mon[103278]: pgmap v10075: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:50 smithi078 ceph-mon[99959]: pgmap v10075: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:50 smithi078 ceph-mon[91136]: pgmap v10075: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:55:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:55:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:51.557Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:51.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:55:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:55:51.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:55:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:52 smithi146 ceph-mon[103278]: pgmap v10076: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:52 smithi078 ceph-mon[91136]: pgmap v10076: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:52 smithi078 ceph-mon[99959]: pgmap v10076: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:54 smithi078 ceph-mon[91136]: pgmap v10077: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:55.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:54 smithi078 ceph-mon[99959]: pgmap v10077: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:54 smithi146 ceph-mon[103278]: pgmap v10077: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:55:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:56 smithi146 ceph-mon[103278]: pgmap v10078: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[99959]: pgmap v10078: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:56.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:55:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:56 smithi078 ceph-mon[91136]: pgmap v10078: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:55:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:58 smithi146 ceph-mon[103278]: pgmap v10079: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:58 smithi078 ceph-mon[91136]: pgmap v10079: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:58.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:58 smithi078 ceph-mon[99959]: pgmap v10079: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:55:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:55:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:00 smithi146 ceph-mon[103278]: pgmap v10080: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:56:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:56:00.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[99959]: pgmap v10080: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[91136]: pgmap v10080: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:01.839 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:01.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:01.840 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:01.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:01.840 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:01.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:01.840 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:56:02.367 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:56:02.367 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 5h - - 2024-09-21T01:56:02.367 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 5h - - 2024-09-21T01:56:02.367 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 6s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 2s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 6s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 6s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 6s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 6s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 6s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 2s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 6s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 6s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 6s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 6s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:56:02.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 2s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 2s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 2s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 2s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 5h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 6s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 6s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:56:02.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:56:02.678 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:56:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:56:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:56:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:56:02.679 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:56:02.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:02 smithi078 ceph-mon[99959]: pgmap v10081: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:02.949 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4135655511' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:02.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:02 smithi078 ceph-mon[91136]: pgmap v10081: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:02.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4135655511' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:56:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:56:02.951 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:56:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:02 smithi146 ceph-mon[103278]: pgmap v10081: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4135655511' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:03.257 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:56:03.258 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:56:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:56:03.258 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:56:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:56:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:03 smithi146 ceph-mon[103278]: from='client.66903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:03 smithi146 ceph-mon[103278]: from='client.66909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:03 smithi146 ceph-mon[103278]: from='client.66915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3117539616' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[91136]: from='client.66903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[91136]: from='client.66909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[91136]: from='client.66915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3117539616' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[99959]: from='client.66903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[99959]: from='client.66909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[99959]: from='client.66915 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3117539616' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:04 smithi078 ceph-mon[91136]: from='client.66927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:04 smithi078 ceph-mon[91136]: pgmap v10082: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:04 smithi078 ceph-mon[99959]: from='client.66927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:04 smithi078 ceph-mon[99959]: pgmap v10082: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:04 smithi146 ceph-mon[103278]: from='client.66927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:04 smithi146 ceph-mon[103278]: pgmap v10082: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:06 smithi078 ceph-mon[99959]: pgmap v10083: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:06 smithi078 ceph-mon[91136]: pgmap v10083: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:06 smithi146 ceph-mon[103278]: pgmap v10083: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:08.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:08 smithi146 ceph-mon[103278]: pgmap v10084: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:08 smithi078 ceph-mon[91136]: pgmap v10084: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:09.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:08 smithi078 ceph-mon[99959]: pgmap v10084: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:10.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:10 smithi146 ceph-mon[103278]: pgmap v10085: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:10 smithi078 ceph-mon[91136]: pgmap v10085: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:10 smithi078 ceph-mon[99959]: pgmap v10085: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:56:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T01:56:11.988 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:11.558Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:11.988 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:11.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:11.988 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:11.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:12 smithi146 ceph-mon[103278]: pgmap v10086: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:12 smithi078 ceph-mon[99959]: pgmap v10086: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:12 smithi078 ceph-mon[91136]: pgmap v10086: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:14 smithi146 ceph-mon[103278]: pgmap v10087: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:14 smithi078 ceph-mon[99959]: pgmap v10087: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:14 smithi078 ceph-mon[91136]: pgmap v10087: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:16.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[91136]: pgmap v10088: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:16 smithi078 ceph-mon[99959]: pgmap v10088: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:16 smithi146 ceph-mon[103278]: pgmap v10088: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:18 smithi146 ceph-mon[103278]: pgmap v10089: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:18 smithi078 ceph-mon[99959]: pgmap v10089: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:18 smithi078 ceph-mon[91136]: pgmap v10089: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:20.905 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:56:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:56:20.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[99959]: pgmap v10090: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:20.905 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:20.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:20.906 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:20.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[91136]: pgmap v10090: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:20.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:20.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:20.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:20 smithi146 ceph-mon[103278]: pgmap v10090: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:21.559Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:21.559Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:21.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:22 smithi078 ceph-mon[91136]: pgmap v10091: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:23.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:22 smithi078 ceph-mon[99959]: pgmap v10091: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:22 smithi146 ceph-mon[103278]: pgmap v10091: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:24 smithi078 ceph-mon[99959]: pgmap v10092: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:24 smithi078 ceph-mon[91136]: pgmap v10092: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:24 smithi146 ceph-mon[103278]: pgmap v10092: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:26 smithi078 ceph-mon[99959]: pgmap v10093: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:26 smithi078 ceph-mon[91136]: pgmap v10093: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:27.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:26 smithi146 ceph-mon[103278]: pgmap v10093: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:28.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:28 smithi078 ceph-mon[99959]: pgmap v10094: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:28 smithi078 ceph-mon[91136]: pgmap v10094: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:28 smithi146 ceph-mon[103278]: pgmap v10094: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:31.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:30 smithi078 ceph-mon[99959]: pgmap v10095: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:30 smithi078 ceph-mon[91136]: pgmap v10095: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:56:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:56:31.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:30 smithi146 ceph-mon[103278]: pgmap v10095: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:31.891 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:31.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:31.891 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:31.560Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:31.891 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:31.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:32.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:32 smithi146 ceph-mon[103278]: pgmap v10096: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:33.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:32 smithi078 ceph-mon[91136]: pgmap v10096: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:33.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:32 smithi078 ceph-mon[99959]: pgmap v10096: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:33.557 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:56:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:56:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 5h - - 2024-09-21T01:56:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 5h - - 2024-09-21T01:56:34.089 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 18s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 13s ago 5h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 18s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 13s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 18s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 18s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 13s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 18s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 18s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 13s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 18s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 18s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 18s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:56:34.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 18s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 13s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 13s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 13s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 13s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 13s ago 5h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 18s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 18s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:56:34.091 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:56:34.399 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:56:34.399 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:56:34.400 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:56:34.666 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:56:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:56:34.666 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:56:34.667 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:56:34.980 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:56:34.980 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:56:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:56:34.980 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:56:34.980 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:56:34.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[99959]: from='client.66939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:34.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[99959]: pgmap v10097: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:34.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[99959]: from='client.86864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:34.980 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/595368685' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:34.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[91136]: from='client.66939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:34.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[91136]: pgmap v10097: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:34.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[91136]: from='client.86864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:34.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/595368685' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:34 smithi146 ceph-mon[103278]: from='client.66939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:34 smithi146 ceph-mon[103278]: pgmap v10097: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:34 smithi146 ceph-mon[103278]: from='client.86864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/595368685' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:56:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[99959]: from='client.66951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[99959]: from='client.86882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1252681914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[91136]: from='client.66951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[91136]: from='client.86882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1252681914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:35 smithi146 ceph-mon[103278]: from='client.66951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:35 smithi146 ceph-mon[103278]: from='client.86882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:56:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1252681914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:56:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:36 smithi146 ceph-mon[103278]: pgmap v10098: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:37.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:36 smithi078 ceph-mon[99959]: pgmap v10098: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:37.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:36 smithi078 ceph-mon[91136]: pgmap v10098: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:38 smithi146 ceph-mon[103278]: pgmap v10099: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:38 smithi078 ceph-mon[99959]: pgmap v10099: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:38 smithi078 ceph-mon[91136]: pgmap v10099: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:56:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:56:41.081 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[91136]: pgmap v10100: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:41.081 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.081 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.081 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.082 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[99959]: pgmap v10100: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:41.082 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.082 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.082 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:40 smithi146 ceph-mon[103278]: pgmap v10100: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:41.560Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:41.561Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:41.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:41.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:43.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:42 smithi078 ceph-mon[91136]: pgmap v10101: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:43.092 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:42 smithi078 ceph-mon[99959]: pgmap v10101: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:42 smithi146 ceph-mon[103278]: pgmap v10101: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:44 smithi146 ceph-mon[103278]: pgmap v10102: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:44 smithi078 ceph-mon[91136]: pgmap v10102: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:44 smithi078 ceph-mon[99959]: pgmap v10102: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:46 smithi146 ceph-mon[103278]: pgmap v10103: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:47.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:46 smithi078 ceph-mon[99959]: pgmap v10103: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:47.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:46 smithi078 ceph-mon[91136]: pgmap v10103: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:48 smithi146 ceph-mon[103278]: pgmap v10104: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:48 smithi078 ceph-mon[91136]: pgmap v10104: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:49.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:48 smithi078 ceph-mon[99959]: pgmap v10104: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:49 smithi146 ceph-mon[103278]: pgmap v10105: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:49 smithi078 ceph-mon[91136]: pgmap v10105: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:49 smithi078 ceph-mon[99959]: pgmap v10105: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:56:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:56:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:51.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:51.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:56:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:56:51.562Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:56:53.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:52 smithi146 ceph-mon[103278]: pgmap v10106: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:53.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:52 smithi078 ceph-mon[91136]: pgmap v10106: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:53.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:52 smithi078 ceph-mon[99959]: pgmap v10106: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:55.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:54 smithi078 ceph-mon[91136]: pgmap v10107: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:54 smithi078 ceph-mon[99959]: pgmap v10107: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:54 smithi146 ceph-mon[103278]: pgmap v10107: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:56:57.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:56 smithi146 ceph-mon[103278]: pgmap v10108: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:57.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[91136]: pgmap v10108: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[99959]: pgmap v10108: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:56:58.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:56:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:58 smithi146 ceph-mon[103278]: pgmap v10109: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:58 smithi078 ceph-mon[91136]: pgmap v10109: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:56:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:58 smithi078 ceph-mon[99959]: pgmap v10109: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:56:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:56:59 smithi146 ceph-mon[103278]: pgmap v10110: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:56:59 smithi078 ceph-mon[91136]: pgmap v10110: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:56:59 smithi078 ceph-mon[99959]: pgmap v10110: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:57:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:57:01.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.198 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:01.562Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:02.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:01.563Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:02.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:01.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:02.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:02 smithi146 ceph-mon[103278]: pgmap v10111: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:03.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:02 smithi078 ceph-mon[91136]: pgmap v10111: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:02 smithi078 ceph-mon[99959]: pgmap v10111: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:04 smithi078 ceph-mon[91136]: pgmap v10112: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:04 smithi078 ceph-mon[99959]: pgmap v10112: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:04 smithi146 ceph-mon[103278]: pgmap v10112: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:05.280 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 5h - - 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 5h - - 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 9s ago 5h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 5s ago 5h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 9s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 5s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:57:05.804 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 9s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 9s ago 5h 533M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 5s ago 5h 364M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 9s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 9s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 5s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 9s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 9s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 9s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 9s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 5s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 5s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 5s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 5s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 5s ago 5h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:57:05.805 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 9s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:57:05.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:57:05.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 9s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:57:05.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:57:06.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:06.115 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:57:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:57:06.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:57:06.116 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:57:06.383 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:57:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:57:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:57:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:57:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:57:06.384 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:57:06.384 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:57:06.691 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:57:06.691 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:06.691 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:06.691 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[91136]: from='client.86894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[91136]: from='client.86900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[91136]: pgmap v10113: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[91136]: from='client.66987 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2880696934' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2732886697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[99959]: from='client.86894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[99959]: from='client.86900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[99959]: pgmap v10113: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[99959]: from='client.66987 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2880696934' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2732886697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:06 smithi146 ceph-mon[103278]: from='client.86894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:06 smithi146 ceph-mon[103278]: from='client.86900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:06 smithi146 ceph-mon[103278]: pgmap v10113: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:06 smithi146 ceph-mon[103278]: from='client.66987 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2880696934' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2732886697' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:07 smithi146 ceph-mon[103278]: from='client.66999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:07 smithi078 ceph-mon[99959]: from='client.66999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:07 smithi078 ceph-mon[91136]: from='client.66999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:08 smithi146 ceph-mon[103278]: pgmap v10114: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:09.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:08 smithi078 ceph-mon[99959]: pgmap v10114: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:08 smithi078 ceph-mon[91136]: pgmap v10114: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:57:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:57:11.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:10 smithi078 ceph-mon[99959]: pgmap v10115: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:10 smithi078 ceph-mon[91136]: pgmap v10115: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:10 smithi146 ceph-mon[103278]: pgmap v10115: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:11.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:11.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:11.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:13.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:12 smithi146 ceph-mon[103278]: pgmap v10116: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:12 smithi078 ceph-mon[91136]: pgmap v10116: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:12 smithi078 ceph-mon[99959]: pgmap v10116: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:13.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:14 smithi146 ceph-mon[103278]: pgmap v10117: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:14 smithi078 ceph-mon[91136]: pgmap v10117: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:14 smithi078 ceph-mon[99959]: pgmap v10117: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:16 smithi146 ceph-mon[103278]: pgmap v10118: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[91136]: pgmap v10118: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[99959]: pgmap v10118: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:18 smithi146 ceph-mon[103278]: pgmap v10119: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:18 smithi078 ceph-mon[99959]: pgmap v10119: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:18 smithi078 ceph-mon[91136]: pgmap v10119: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:20 smithi146 ceph-mon[103278]: pgmap v10120: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:57:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:57:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[91136]: pgmap v10120: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[99959]: pgmap v10120: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:21.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:21.564Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:21.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:23.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:22 smithi078 ceph-mon[91136]: pgmap v10121: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:23.092 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:22 smithi078 ceph-mon[99959]: pgmap v10121: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:23.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:22 smithi146 ceph-mon[103278]: pgmap v10121: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:24 smithi078 ceph-mon[91136]: pgmap v10122: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:24 smithi078 ceph-mon[99959]: pgmap v10122: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:24 smithi146 ceph-mon[103278]: pgmap v10122: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:25 smithi146 ceph-mon[103278]: pgmap v10123: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:25 smithi078 ceph-mon[91136]: pgmap v10123: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:25 smithi078 ceph-mon[99959]: pgmap v10123: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:27.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:27.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:27.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:27 smithi146 ceph-mon[103278]: pgmap v10124: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:27 smithi078 ceph-mon[91136]: pgmap v10124: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:27 smithi078 ceph-mon[99959]: pgmap v10124: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:29 smithi146 ceph-mon[103278]: pgmap v10125: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:29 smithi078 ceph-mon[91136]: pgmap v10125: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:29 smithi078 ceph-mon[99959]: pgmap v10125: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:57:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:57:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:31.564Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:31.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:31.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:33.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:32 smithi146 ceph-mon[103278]: pgmap v10126: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:33.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:32 smithi078 ceph-mon[91136]: pgmap v10126: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:32 smithi078 ceph-mon[99959]: pgmap v10126: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:34 smithi146 ceph-mon[103278]: pgmap v10127: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:34 smithi078 ceph-mon[91136]: pgmap v10127: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:34 smithi078 ceph-mon[99959]: pgmap v10127: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[99959]: pgmap v10128: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[91136]: pgmap v10128: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:36.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:36.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:36.993 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:57:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:36 smithi146 ceph-mon[103278]: pgmap v10128: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 5h - - 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 5h - - 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 1s ago 5h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 16s ago 5h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 1s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 16s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 1s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 1s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 16s ago 5h 365M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:57:37.516 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 1s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 1s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 16s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 1s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 1s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 1s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 1s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 16s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 16s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 16s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 16s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 5h 73.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 1s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 1s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:57:37.517 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:57:37.829 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:57:37.829 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:57:37.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:57:37.831 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:57:38.100 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:57:38.100 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:57:38.100 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:57:38.100 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:57:38.100 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:57:38.101 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:57:38.101 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:38.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:38.410 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:57:38.410 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:38.410 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:57:38.410 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:38.410 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:57:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:38 smithi146 ceph-mon[103278]: from='client.67011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:38 smithi146 ceph-mon[103278]: from='client.86936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:38 smithi146 ceph-mon[103278]: from='client.67023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:38 smithi146 ceph-mon[103278]: pgmap v10129: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2162702926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1483958014' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[99959]: from='client.67011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[99959]: from='client.86936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[99959]: from='client.67023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[99959]: pgmap v10129: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2162702926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1483958014' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[91136]: from='client.67011 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[91136]: from='client.86936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[91136]: from='client.67023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[91136]: pgmap v10129: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2162702926' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:57:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1483958014' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:57:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:39 smithi146 ceph-mon[103278]: from='client.86954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:39 smithi078 ceph-mon[99959]: from='client.86954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:39 smithi078 ceph-mon[91136]: from='client.86954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:57:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:40 smithi078 ceph-mon[99959]: pgmap v10130: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:40 smithi078 ceph-mon[91136]: pgmap v10130: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:57:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:57:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:40 smithi146 ceph-mon[103278]: pgmap v10130: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:41.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:41.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:41.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:41.565Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:41.861 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:41.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:41 smithi146 ceph-mon[103278]: pgmap v10131: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[91136]: pgmap v10131: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:41 smithi078 ceph-mon[99959]: pgmap v10131: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:43.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:43.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:44 smithi146 ceph-mon[103278]: pgmap v10132: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:45.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:44 smithi078 ceph-mon[91136]: pgmap v10132: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:45.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:44 smithi078 ceph-mon[99959]: pgmap v10132: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:46 smithi078 ceph-mon[99959]: pgmap v10133: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:46 smithi078 ceph-mon[91136]: pgmap v10133: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:46 smithi146 ceph-mon[103278]: pgmap v10133: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:48 smithi146 ceph-mon[103278]: pgmap v10134: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:48 smithi078 ceph-mon[91136]: pgmap v10134: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:49.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:48 smithi078 ceph-mon[99959]: pgmap v10134: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:51.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:50 smithi078 ceph-mon[91136]: pgmap v10135: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:50 smithi078 ceph-mon[99959]: pgmap v10135: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:57:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:57:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:50 smithi146 ceph-mon[103278]: pgmap v10135: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:51.565Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:51.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:57:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:57:51.566Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:57:53.147 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:52 smithi146 ceph-mon[103278]: pgmap v10136: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:52 smithi078 ceph-mon[99959]: pgmap v10136: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:52 smithi078 ceph-mon[91136]: pgmap v10136: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:55.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:54 smithi146 ceph-mon[103278]: pgmap v10137: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:54 smithi078 ceph-mon[91136]: pgmap v10137: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:55.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:54 smithi078 ceph-mon[99959]: pgmap v10137: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:57:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[99959]: pgmap v10138: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[91136]: pgmap v10138: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:56 smithi146 ceph-mon[103278]: pgmap v10138: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:57:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:57:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:58 smithi146 ceph-mon[103278]: pgmap v10139: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:57:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:58 smithi078 ceph-mon[91136]: pgmap v10139: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:57:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:57:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:58 smithi078 ceph-mon[99959]: pgmap v10139: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:57:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:57:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:00 smithi078 ceph-mon[91136]: pgmap v10140: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:00 smithi078 ceph-mon[99959]: pgmap v10140: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:58:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:58:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:00 smithi146 ceph-mon[103278]: pgmap v10140: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:01.991 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:01.566Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:01.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:01.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:01.992 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:01.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:01 smithi146 ceph-mon[103278]: pgmap v10141: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[91136]: pgmap v10141: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:01 smithi078 ceph-mon[99959]: pgmap v10141: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:05.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:04 smithi078 ceph-mon[99959]: pgmap v10142: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:04 smithi078 ceph-mon[91136]: pgmap v10142: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:05.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:04 smithi146 ceph-mon[103278]: pgmap v10142: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:06 smithi146 ceph-mon[103278]: pgmap v10143: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:06 smithi078 ceph-mon[91136]: pgmap v10143: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:06 smithi078 ceph-mon[99959]: pgmap v10143: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:08.703 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:58:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:08 smithi078 ceph-mon[99959]: pgmap v10144: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:08 smithi078 ceph-mon[91136]: pgmap v10144: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:09.231 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:58:09.231 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 5h - - 2024-09-21T01:58:09.231 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 5h - - 2024-09-21T01:58:09.231 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 13s ago 5h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:58:09.231 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 8s ago 5h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:58:09.231 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 13s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 8s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 13s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 13s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 8s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 13s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 13s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 8s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 13s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 13s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 13s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 13s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 8s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 8s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:58:09.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 8s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:58:09.233 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 8s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:58:09.233 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 8s ago 5h 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:58:09.233 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 13s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:58:09.233 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:58:09.233 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 13s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:58:09.233 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:58:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:08 smithi146 ceph-mon[103278]: pgmap v10144: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:09.540 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:58:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:58:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:58:09.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:58:09.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:58:09.814 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:58:09.815 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:58:10.123 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:58:10.123 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:10.124 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:10.124 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:10.124 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:09 smithi078 ceph-mon[99959]: from='client.86966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:10.124 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2936378451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:10.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:09 smithi078 ceph-mon[91136]: from='client.86966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:10.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2936378451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:09 smithi146 ceph-mon[103278]: from='client.86966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2936378451' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:58:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:58:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[99959]: from='client.86972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[99959]: from='client.86978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[99959]: pgmap v10145: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[99959]: from='client.67071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1814506624' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[91136]: from='client.86972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[91136]: from='client.86978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[91136]: pgmap v10145: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[91136]: from='client.67071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1814506624' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:10 smithi146 ceph-mon[103278]: from='client.86972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:10 smithi146 ceph-mon[103278]: from='client.86978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:10 smithi146 ceph-mon[103278]: pgmap v10145: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:10 smithi146 ceph-mon[103278]: from='client.67071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1814506624' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:11.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:11.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:11.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:12 smithi146 ceph-mon[103278]: pgmap v10146: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:12 smithi078 ceph-mon[99959]: pgmap v10146: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:12 smithi078 ceph-mon[91136]: pgmap v10146: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:13.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:15.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:14 smithi146 ceph-mon[103278]: pgmap v10147: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:14 smithi078 ceph-mon[99959]: pgmap v10147: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:14 smithi078 ceph-mon[91136]: pgmap v10147: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[91136]: pgmap v10148: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[99959]: pgmap v10148: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:16 smithi146 ceph-mon[103278]: pgmap v10148: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:18 smithi146 ceph-mon[103278]: pgmap v10149: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:18 smithi078 ceph-mon[91136]: pgmap v10149: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:18 smithi078 ceph-mon[99959]: pgmap v10149: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:19 smithi146 ceph-mon[103278]: pgmap v10150: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:19 smithi078 ceph-mon[91136]: pgmap v10150: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:19 smithi078 ceph-mon[99959]: pgmap v10150: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:58:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T01:58:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:21.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:21.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:21.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[99959]: pgmap v10151: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:22 smithi078 ceph-mon[91136]: pgmap v10151: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:22 smithi146 ceph-mon[103278]: pgmap v10151: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:24 smithi078 ceph-mon[91136]: pgmap v10152: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:24 smithi078 ceph-mon[99959]: pgmap v10152: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:24 smithi146 ceph-mon[103278]: pgmap v10152: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:26 smithi078 ceph-mon[99959]: pgmap v10153: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:26 smithi078 ceph-mon[91136]: pgmap v10153: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:27.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:26 smithi146 ceph-mon[103278]: pgmap v10153: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:28.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:28 smithi146 ceph-mon[103278]: pgmap v10154: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:28 smithi078 ceph-mon[91136]: pgmap v10154: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:28 smithi078 ceph-mon[99959]: pgmap v10154: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:30 smithi078 ceph-mon[91136]: pgmap v10155: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:58:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:58:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:30 smithi078 ceph-mon[99959]: pgmap v10155: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:30 smithi146 ceph-mon[103278]: pgmap v10155: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:31.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:31.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:31.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:33.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:32 smithi078 ceph-mon[99959]: pgmap v10156: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:32 smithi078 ceph-mon[91136]: pgmap v10156: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:32 smithi146 ceph-mon[103278]: pgmap v10156: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:35.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:34 smithi146 ceph-mon[103278]: pgmap v10157: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:34 smithi078 ceph-mon[91136]: pgmap v10157: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:34 smithi078 ceph-mon[99959]: pgmap v10157: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:36 smithi146 ceph-mon[103278]: pgmap v10158: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[99959]: pgmap v10158: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:37.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[91136]: pgmap v10158: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:38 smithi146 ceph-mon[103278]: pgmap v10159: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:38 smithi078 ceph-mon[91136]: pgmap v10159: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:38 smithi078 ceph-mon[99959]: pgmap v10159: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:39 smithi146 ceph-mon[103278]: pgmap v10160: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:39 smithi078 ceph-mon[99959]: pgmap v10160: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:39 smithi078 ceph-mon[91136]: pgmap v10160: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:40.420 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:58:40.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:58:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 5h - - 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 5h - - 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 4s ago 5h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 19s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 4s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:58:40.949 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 19s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 4s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 4s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 19s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 4s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 4s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 19s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 4s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 4s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 4s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 4s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 19s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 19s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 19s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:58:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 19s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:58:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 19s ago 5h 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:58:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 4s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:58:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:58:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 4s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:58:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:58:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:40 smithi146 ceph-mon[103278]: from='client.87002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:40 smithi146 ceph-mon[103278]: from='client.87008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:41.266 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:58:41.266 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:58:41.266 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:58:41.267 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:58:41.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:40 smithi078 ceph-mon[99959]: from='client.87002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:41.269 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:40 smithi078 ceph-mon[99959]: from='client.87008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:41.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:40 smithi078 ceph-mon[91136]: from='client.87002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:41.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:40 smithi078 ceph-mon[91136]: from='client.87008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:41.534 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:58:41.534 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:58:41.535 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:58:41.535 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:58:41.535 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:58:41.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:58:41.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:58:41.536 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:58:41.536 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:58:41.536 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:58:41.536 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:58:41.536 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:58:41.536 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:58:41.537 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:58:41.855 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:58:41.856 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:41.856 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:58:41.856 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:41.856 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:58:41.856 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:41.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:41.856 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:41.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:41.856 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:41.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='client.87014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1277557608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='client.67107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: pgmap v10161: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/16292935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='client.87014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1277557608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='client.67107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: pgmap v10161: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/16292935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='client.87014 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1277557608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='client.67107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: pgmap v10161: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/16292935' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:58:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:44.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:44 smithi078 ceph-mon[91136]: pgmap v10162: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:44.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:44 smithi078 ceph-mon[99959]: pgmap v10162: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:44 smithi146 ceph-mon[103278]: pgmap v10162: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:46.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:46 smithi078 ceph-mon[91136]: pgmap v10163: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:46.911 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:46 smithi078 ceph-mon[99959]: pgmap v10163: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:46 smithi146 ceph-mon[103278]: pgmap v10163: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:48 smithi146 ceph-mon[103278]: pgmap v10164: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:49.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:48 smithi078 ceph-mon[99959]: pgmap v10164: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:48 smithi078 ceph-mon[91136]: pgmap v10164: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:51.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:50 smithi078 ceph-mon[99959]: pgmap v10165: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:50 smithi078 ceph-mon[91136]: pgmap v10165: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:58:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T01:58:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:50 smithi146 ceph-mon[103278]: pgmap v10165: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:51.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:51.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:58:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:58:51.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:58:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:52 smithi146 ceph-mon[103278]: pgmap v10166: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:53.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:52 smithi078 ceph-mon[99959]: pgmap v10166: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:53.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:52 smithi078 ceph-mon[91136]: pgmap v10166: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:54 smithi146 ceph-mon[103278]: pgmap v10167: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:55.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:54 smithi078 ceph-mon[99959]: pgmap v10167: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:54 smithi078 ceph-mon[91136]: pgmap v10167: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:58:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[91136]: pgmap v10168: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[99959]: pgmap v10168: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:56 smithi146 ceph-mon[103278]: pgmap v10168: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:58:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:58:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:58 smithi146 ceph-mon[103278]: pgmap v10169: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:58 smithi078 ceph-mon[91136]: pgmap v10169: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:58:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:58 smithi078 ceph-mon[99959]: pgmap v10169: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:58:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:58:59 smithi146 ceph-mon[103278]: pgmap v10170: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:58:59 smithi078 ceph-mon[99959]: pgmap v10170: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:58:59 smithi078 ceph-mon[91136]: pgmap v10170: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:59:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:59:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:01.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:01.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:01.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:02 smithi146 ceph-mon[103278]: pgmap v10171: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[91136]: pgmap v10171: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:02 smithi078 ceph-mon[99959]: pgmap v10171: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:05.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:04 smithi078 ceph-mon[99959]: pgmap v10172: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:05.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:04 smithi078 ceph-mon[91136]: pgmap v10172: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:04 smithi146 ceph-mon[103278]: pgmap v10172: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:07.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:06 smithi078 ceph-mon[99959]: pgmap v10173: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:07.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:06 smithi078 ceph-mon[91136]: pgmap v10173: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:06 smithi146 ceph-mon[103278]: pgmap v10173: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:08 smithi146 ceph-mon[103278]: pgmap v10174: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:09.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:08 smithi078 ceph-mon[99959]: pgmap v10174: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:08 smithi078 ceph-mon[91136]: pgmap v10174: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:11.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:10 smithi078 ceph-mon[91136]: pgmap v10175: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:10 smithi078 ceph-mon[99959]: pgmap v10175: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:59:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:59:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:10 smithi146 ceph-mon[103278]: pgmap v10175: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:11.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:11.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:11.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:12.156 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 5h - - 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 16s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 11s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 11s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:59:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 5h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 11s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 16s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 11s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 11s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 11s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 11s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 11s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:59:12.682 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 11s ago 5h 74.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:59:12.683 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 16s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:59:12.683 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:59:12.683 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 16s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:59:12.683 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:59:12.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:59:12.993 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:59:12.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:12 smithi078 ceph-mon[91136]: pgmap v10176: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:12.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:12 smithi078 ceph-mon[99959]: pgmap v10176: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:59:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:59:13.262 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:59:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:12 smithi146 ceph-mon[103278]: pgmap v10176: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:13.571 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:59:13.571 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:13.572 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:13.572 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:13 smithi146 ceph-mon[103278]: from='client.87038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:13 smithi146 ceph-mon[103278]: from='client.67125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:13 smithi146 ceph-mon[103278]: from='client.67131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4117815304' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4253548453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[91136]: from='client.87038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[91136]: from='client.67125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[91136]: from='client.67131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4117815304' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4253548453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[99959]: from='client.87038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[99959]: from='client.67125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[99959]: from='client.67131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4117815304' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4253548453' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:15.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:14 smithi146 ceph-mon[103278]: from='client.87062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:15.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:14 smithi146 ceph-mon[103278]: pgmap v10177: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:14 smithi078 ceph-mon[99959]: from='client.87062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:14 smithi078 ceph-mon[99959]: pgmap v10177: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:14 smithi078 ceph-mon[91136]: from='client.87062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:14 smithi078 ceph-mon[91136]: pgmap v10177: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:16.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:15 smithi146 ceph-mon[103278]: pgmap v10178: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:15 smithi078 ceph-mon[91136]: pgmap v10178: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:15 smithi078 ceph-mon[99959]: pgmap v10178: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:18 smithi146 ceph-mon[103278]: pgmap v10179: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:18 smithi078 ceph-mon[99959]: pgmap v10179: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:18 smithi078 ceph-mon[91136]: pgmap v10179: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:21.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:20 smithi078 ceph-mon[91136]: pgmap v10180: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:21.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:20 smithi078 ceph-mon[99959]: pgmap v10180: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:59:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T01:59:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:20 smithi146 ceph-mon[103278]: pgmap v10180: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:21.573Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:21.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:21.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:22.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[91136]: pgmap v10181: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:22 smithi078 ceph-mon[99959]: pgmap v10181: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:23.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:22 smithi146 ceph-mon[103278]: pgmap v10181: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:24 smithi146 ceph-mon[103278]: pgmap v10182: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:24 smithi078 ceph-mon[91136]: pgmap v10182: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:25.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:24 smithi078 ceph-mon[99959]: pgmap v10182: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:26.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:26 smithi078 ceph-mon[99959]: pgmap v10183: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:26.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:26 smithi078 ceph-mon[91136]: pgmap v10183: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:26 smithi146 ceph-mon[103278]: pgmap v10183: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:28.090 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:28.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:28.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:28 smithi146 ceph-mon[103278]: pgmap v10184: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:28 smithi078 ceph-mon[91136]: pgmap v10184: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:28 smithi078 ceph-mon[99959]: pgmap v10184: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:31.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:30 smithi078 ceph-mon[91136]: pgmap v10185: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:30 smithi078 ceph-mon[99959]: pgmap v10185: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:59:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:59:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:30 smithi146 ceph-mon[103278]: pgmap v10185: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:31.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:31.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:31.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:32 smithi146 ceph-mon[103278]: pgmap v10186: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:33.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:32 smithi078 ceph-mon[91136]: pgmap v10186: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:33.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:32 smithi078 ceph-mon[99959]: pgmap v10186: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:33 smithi146 ceph-mon[103278]: pgmap v10187: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:33 smithi078 ceph-mon[91136]: pgmap v10187: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:33 smithi078 ceph-mon[99959]: pgmap v10187: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:37.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:36 smithi146 ceph-mon[103278]: pgmap v10188: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:36 smithi078 ceph-mon[91136]: pgmap v10188: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:36 smithi078 ceph-mon[99959]: pgmap v10188: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:38.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:38.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:39.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:38 smithi078 ceph-mon[91136]: pgmap v10189: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:39.097 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:38 smithi078 ceph-mon[99959]: pgmap v10189: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:38 smithi146 ceph-mon[103278]: pgmap v10189: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:41.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:40 smithi078 ceph-mon[91136]: pgmap v10190: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:40 smithi078 ceph-mon[99959]: pgmap v10190: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:59:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T01:59:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:40 smithi146 ceph-mon[103278]: pgmap v10190: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:41.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:41.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:41.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:42 smithi146 ceph-mon[103278]: pgmap v10191: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[91136]: pgmap v10191: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:42 smithi078 ceph-mon[99959]: pgmap v10191: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:43.869 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T01:59:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 5h - - 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 5h - - 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 7s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 2s ago 5h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 7s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 7s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 7s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T01:59:44.392 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 7s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 7s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 2s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 7s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 7s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 7s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 7s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 2s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 2s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 2s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 2s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 5h 70.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 7s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T01:59:44.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 7s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T01:59:44.394 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T01:59:44.702 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:59:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T01:59:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T01:59:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T01:59:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T01:59:44.703 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T01:59:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T01:59:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T01:59:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T01:59:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T01:59:44.969 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T01:59:44.969 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:44 smithi078 ceph-mon[99959]: pgmap v10192: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:44.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:44 smithi078 ceph-mon[91136]: pgmap v10192: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:44 smithi146 ceph-mon[103278]: pgmap v10192: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:45.282 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T01:59:45.282 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T01:59:45.283 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T01:59:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:45 smithi146 ceph-mon[103278]: from='client.87074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:45 smithi146 ceph-mon[103278]: from='client.87080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:45 smithi146 ceph-mon[103278]: from='client.87086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3099320614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1215058616' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[99959]: from='client.87074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[99959]: from='client.87080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[99959]: from='client.87086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3099320614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1215058616' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[91136]: from='client.87074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[91136]: from='client.87080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[91136]: from='client.87086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3099320614' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T01:59:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1215058616' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T01:59:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:46 smithi146 ceph-mon[103278]: from='client.67179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:47.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:46 smithi146 ceph-mon[103278]: pgmap v10193: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:46 smithi078 ceph-mon[91136]: from='client.67179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:47.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:46 smithi078 ceph-mon[91136]: pgmap v10193: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:47.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:46 smithi078 ceph-mon[99959]: from='client.67179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:59:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:46 smithi078 ceph-mon[99959]: pgmap v10193: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:48.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:48 smithi078 ceph-mon[99959]: pgmap v10194: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:48.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:48 smithi078 ceph-mon[91136]: pgmap v10194: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:48 smithi146 ceph-mon[103278]: pgmap v10194: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 01:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:01:59:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T01:59:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:50 smithi146 ceph-mon[103278]: pgmap v10195: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:51.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:50 smithi078 ceph-mon[91136]: pgmap v10195: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:51.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:50 smithi078 ceph-mon[99959]: pgmap v10195: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:51.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:51.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T01:59:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 01:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T01:59:51.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T01:59:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:52 smithi146 ceph-mon[103278]: pgmap v10196: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:52 smithi078 ceph-mon[91136]: pgmap v10196: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:52 smithi078 ceph-mon[99959]: pgmap v10196: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:54 smithi146 ceph-mon[103278]: pgmap v10197: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:54 smithi078 ceph-mon[91136]: pgmap v10197: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:55.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:54 smithi078 ceph-mon[99959]: pgmap v10197: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T01:59:57.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:56 smithi078 ceph-mon[91136]: pgmap v10198: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:57.006 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:56 smithi078 ceph-mon[99959]: pgmap v10198: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:57.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:56 smithi146 ceph-mon[103278]: pgmap v10198: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:58.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:58.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:57 smithi146 ceph-mon[103278]: pgmap v10199: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[91136]: pgmap v10199: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T01:59:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:57 smithi078 ceph-mon[99959]: pgmap v10199: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T01:59:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 01:59:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 01:59:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:59:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 01:59:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:00 smithi146 ceph-mon[103278]: pgmap v10200: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:00 smithi078 ceph-mon[99959]: pgmap v10200: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T01:59:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:00 smithi078 ceph-mon[91136]: pgmap v10200: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:00:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:00:01.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:01.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:01.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:01.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:01.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:01.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[99959]: pgmap v10201: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[91136]: pgmap v10201: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:02 smithi146 ceph-mon[103278]: pgmap v10201: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:04 smithi146 ceph-mon[103278]: pgmap v10202: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:04 smithi078 ceph-mon[91136]: pgmap v10202: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:05.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:04 smithi078 ceph-mon[99959]: pgmap v10202: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:06 smithi078 ceph-mon[99959]: pgmap v10203: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:06 smithi078 ceph-mon[91136]: pgmap v10203: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:06 smithi146 ceph-mon[103278]: pgmap v10203: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:08.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:08 smithi078 ceph-mon[91136]: pgmap v10204: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:08.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:08 smithi078 ceph-mon[99959]: pgmap v10204: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:08 smithi146 ceph-mon[103278]: pgmap v10204: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:09.996 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:09 smithi078 ceph-mon[99959]: pgmap v10205: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:09.997 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:09 smithi078 ceph-mon[91136]: pgmap v10205: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:09 smithi146 ceph-mon[103278]: pgmap v10205: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:10.863 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:00:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:00:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:11.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:11.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:11.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:12 smithi146 ceph-mon[103278]: pgmap v10206: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:12 smithi078 ceph-mon[91136]: pgmap v10206: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:12 smithi078 ceph-mon[99959]: pgmap v10206: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:14 smithi146 ceph-mon[103278]: pgmap v10207: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:14 smithi078 ceph-mon[91136]: pgmap v10207: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:14 smithi078 ceph-mon[99959]: pgmap v10207: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:15.574 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:00:16.105 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 5h - - 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 14s ago 5h 92.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 14s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 19s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 14s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:00:16.106 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 14s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 14s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 14s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 14s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 14s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 14s ago 5h 71.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 19s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 19s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:00:16.107 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:00:16.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:16 smithi078 ceph-mon[91136]: from='client.67191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:16.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:16 smithi078 ceph-mon[91136]: pgmap v10208: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:16.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:16 smithi078 ceph-mon[91136]: from='client.87116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:16.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:16 smithi078 ceph-mon[99959]: from='client.67191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:16.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:16 smithi078 ceph-mon[99959]: pgmap v10208: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:16.416 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:16 smithi078 ceph-mon[99959]: from='client.87116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:16.416 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:00:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:00:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:00:16.417 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:00:16.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:16 smithi146 ceph-mon[103278]: from='client.67191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:16.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:16 smithi146 ceph-mon[103278]: pgmap v10208: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:16.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:16 smithi146 ceph-mon[103278]: from='client.87116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:00:16.683 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:00:16.684 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:00:16.684 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:00:16.684 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:00:16.684 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:00:16.991 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:16.991 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:16.991 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:17.185 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:17 smithi146 ceph-mon[103278]: from='client.67203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/92797371' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:17.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:17 smithi146 ceph-mon[103278]: from='client.67215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:17.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/162218211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[91136]: from='client.67203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/92797371' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[91136]: from='client.67215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/162218211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[99959]: from='client.67203 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/92797371' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[99959]: from='client.67215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/162218211' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:18 smithi146 ceph-mon[103278]: pgmap v10209: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[91136]: pgmap v10209: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:18 smithi078 ceph-mon[99959]: pgmap v10209: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:21.008 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:20 smithi078 ceph-mon[99959]: pgmap v10210: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:21.009 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:00:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T02:00:21.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:20 smithi078 ceph-mon[91136]: pgmap v10210: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:20 smithi146 ceph-mon[103278]: pgmap v10210: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:21.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:21.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:22.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:21.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:22 smithi146 ceph-mon[103278]: pgmap v10211: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[99959]: pgmap v10211: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:23.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[91136]: pgmap v10211: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:24 smithi146 ceph-mon[103278]: pgmap v10212: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:24 smithi078 ceph-mon[91136]: pgmap v10212: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:25.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:24 smithi078 ceph-mon[99959]: pgmap v10212: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:26 smithi146 ceph-mon[103278]: pgmap v10213: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:26 smithi078 ceph-mon[91136]: pgmap v10213: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:26 smithi078 ceph-mon[99959]: pgmap v10213: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:27.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:27.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:28 smithi146 ceph-mon[103278]: pgmap v10214: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:28 smithi078 ceph-mon[99959]: pgmap v10214: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:28 smithi078 ceph-mon[91136]: pgmap v10214: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:30 smithi078 ceph-mon[91136]: pgmap v10215: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:30 smithi078 ceph-mon[99959]: pgmap v10215: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:30 smithi146 ceph-mon[103278]: pgmap v10215: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:30.796 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:00:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:00:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:31.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:31.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:31.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:32 smithi146 ceph-mon[103278]: pgmap v10216: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:33.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:32 smithi078 ceph-mon[91136]: pgmap v10216: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:33.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:32 smithi078 ceph-mon[99959]: pgmap v10216: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:33 smithi146 ceph-mon[103278]: pgmap v10217: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:33 smithi078 ceph-mon[91136]: pgmap v10217: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:33 smithi078 ceph-mon[99959]: pgmap v10217: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:36 smithi078 ceph-mon[91136]: pgmap v10218: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:36 smithi078 ceph-mon[99959]: pgmap v10218: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:36 smithi146 ceph-mon[103278]: pgmap v10218: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:38.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:38.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:39.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:38 smithi146 ceph-mon[103278]: pgmap v10219: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:38 smithi078 ceph-mon[91136]: pgmap v10219: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:38 smithi078 ceph-mon[99959]: pgmap v10219: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:39 smithi146 ceph-mon[103278]: pgmap v10220: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:39 smithi078 ceph-mon[91136]: pgmap v10220: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:39 smithi078 ceph-mon[99959]: pgmap v10220: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:00:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:00:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:41.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:41.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:41.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:42 smithi146 ceph-mon[103278]: pgmap v10221: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[99959]: pgmap v10221: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[91136]: pgmap v10221: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:43 smithi146 ceph-mon[103278]: pgmap v10222: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:43 smithi078 ceph-mon[99959]: pgmap v10222: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:43 smithi078 ceph-mon[91136]: pgmap v10222: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:45 smithi146 ceph-mon[103278]: pgmap v10223: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:45 smithi078 ceph-mon[91136]: pgmap v10223: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:45 smithi078 ceph-mon[99959]: pgmap v10223: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:47.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:47.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:47.277 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 5h - - 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 5h - - 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 10s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 5s ago 5h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 10s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 5s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 10s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 10s ago 5h 534M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 5s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 10s ago 5h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:00:47.801 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 10s ago 5h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 5s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 10s ago 5h 91.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 10s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 10s ago 5h 81.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 10s ago 5h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 5s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 5s ago 5h 86.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 5s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 5s ago 5h 89.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 5s ago 5h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 10s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 10s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:00:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:00:48.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:00:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:00:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:00:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:00:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:00:48.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[99959]: from='client.67227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[99959]: from='client.67233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[99959]: from='client.87158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[99959]: pgmap v10224: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[91136]: from='client.67227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[91136]: from='client.67233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[91136]: from='client.87158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:47 smithi078 ceph-mon[91136]: pgmap v10224: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:47 smithi146 ceph-mon[103278]: from='client.67227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:47 smithi146 ceph-mon[103278]: from='client.67233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:47 smithi146 ceph-mon[103278]: from='client.87158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:47 smithi146 ceph-mon[103278]: pgmap v10224: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:00:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:00:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:00:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:00:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:00:48.383 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:00:48.687 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:00:48.688 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:48.688 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:00:48.688 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:48.688 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:00:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1656557954' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:48 smithi078 ceph-mon[91136]: from='client.67251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3373693277' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1656557954' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:48 smithi078 ceph-mon[99959]: from='client.67251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3373693277' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1656557954' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:00:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:48 smithi146 ceph-mon[103278]: from='client.67251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:00:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3373693277' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:00:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:49 smithi146 ceph-mon[103278]: pgmap v10225: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:50.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:49 smithi078 ceph-mon[99959]: pgmap v10225: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:50.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:49 smithi078 ceph-mon[91136]: pgmap v10225: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:00:50.943 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:00:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:00:51.831 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:51.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:51.831 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:51.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:00:51.831 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:00:51.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:00:52.103 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:51 smithi078 ceph-mon[99959]: pgmap v10226: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:52.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:51 smithi078 ceph-mon[91136]: pgmap v10226: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:51 smithi146 ceph-mon[103278]: pgmap v10226: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:00:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:53 smithi146 ceph-mon[103278]: pgmap v10227: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:00:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:53 smithi078 ceph-mon[91136]: pgmap v10227: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:00:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:53 smithi078 ceph-mon[99959]: pgmap v10227: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:00:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:55 smithi146 ceph-mon[103278]: pgmap v10228: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:00:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:55 smithi078 ceph-mon[91136]: pgmap v10228: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:00:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:55 smithi078 ceph-mon[99959]: pgmap v10228: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:00:57.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:57.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:57.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:00:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:58 smithi146 ceph-mon[103278]: pgmap v10229: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:00:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[91136]: pgmap v10229: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:00:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:58 smithi078 ceph-mon[99959]: pgmap v10229: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:00:59.653 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:00:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:00 smithi146 ceph-mon[103278]: pgmap v10230: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:01:00.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:00 smithi078 ceph-mon[91136]: pgmap v10230: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:01:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:00 smithi078 ceph-mon[99959]: pgmap v10230: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:01:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:01:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:01:01.906 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:01.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:01.906 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:01.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:01.906 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:01.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:02.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:01 smithi146 ceph-mon[103278]: pgmap v10231: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:01:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:01 smithi078 ceph-mon[91136]: pgmap v10231: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:01:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:01 smithi078 ceph-mon[99959]: pgmap v10231: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:01:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:04 smithi146 ceph-mon[103278]: pgmap v10232: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T02:01:04.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:04 smithi078 ceph-mon[99959]: pgmap v10232: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T02:01:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:04 smithi078 ceph-mon[91136]: pgmap v10232: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-09-21T02:01:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:05 smithi146 ceph-mon[103278]: pgmap v10233: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T02:01:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:05 smithi078 ceph-mon[91136]: pgmap v10233: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T02:01:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:05 smithi078 ceph-mon[99959]: pgmap v10233: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 70 op/s 2024-09-21T02:01:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:07.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:07 smithi146 ceph-mon[103278]: pgmap v10234: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:01:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:07 smithi078 ceph-mon[99959]: pgmap v10234: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:01:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:07 smithi078 ceph-mon[91136]: pgmap v10234: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-09-21T02:01:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:09 smithi146 ceph-mon[103278]: pgmap v10235: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T02:01:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:09 smithi078 ceph-mon[99959]: pgmap v10235: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T02:01:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:09 smithi078 ceph-mon[91136]: pgmap v10235: 161 pgs: 161 active+clean; 585 KiB data, 336 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-21T02:01:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:01:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:01:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:11.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:11.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:11.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:12.239 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:11 smithi078 ceph-mon[99959]: pgmap v10236: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T02:01:12.239 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:11 smithi078 ceph-mon[91136]: pgmap v10236: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T02:01:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:11 smithi146 ceph-mon[103278]: pgmap v10236: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T02:01:13.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:13.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:14 smithi146 ceph-mon[103278]: pgmap v10237: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:01:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:14 smithi078 ceph-mon[99959]: pgmap v10237: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:01:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:14 smithi078 ceph-mon[91136]: pgmap v10237: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:01:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:15 smithi146 ceph-mon[103278]: pgmap v10238: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T02:01:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:15 smithi078 ceph-mon[99959]: pgmap v10238: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T02:01:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:15 smithi078 ceph-mon[91136]: pgmap v10238: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-09-21T02:01:17.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:17.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:18 smithi146 ceph-mon[103278]: pgmap v10239: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T02:01:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[99959]: pgmap v10239: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T02:01:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:18 smithi078 ceph-mon[91136]: pgmap v10239: 161 pgs: 161 active+clean; 585 KiB data, 340 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-09-21T02:01:18.982 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:01:19.516 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:01:19.516 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 5h - - 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 5h - - 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 1s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 17s ago 5h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 1s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 17s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 2s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 17s ago 5h 366M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 1s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 17s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 5h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:01:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 5h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 5h 80.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 1s ago 5h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 17s ago 5h 110M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 17s ago 5h 86.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 17s ago 5h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 17s ago 5h 89.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 17s ago 5h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 1s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 1s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:01:19.518 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 17s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:01:19.870 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:01:19.870 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:01:19.871 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:01:20.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:19 smithi146 ceph-mon[103278]: from='client.67263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:19 smithi146 ceph-mon[103278]: from='client.87188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:19 smithi146 ceph-mon[103278]: from='client.67275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:19 smithi146 ceph-mon[103278]: pgmap v10240: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:01:20.033 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2104153401' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:01:20.140 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:01:20.141 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:01:20.141 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:01:20.141 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:01:20.141 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:01:20.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:01:20.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[99959]: from='client.67263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[99959]: from='client.87188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[99959]: from='client.67275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[99959]: pgmap v10240: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:01:20.141 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2104153401' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:20.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[91136]: from='client.67263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[91136]: from='client.87188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[91136]: from='client.67275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:20.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[91136]: pgmap v10240: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T02:01:20.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2104153401' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:20.451 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:01:20.451 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:20.451 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:20.452 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:20.452 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:01:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:01:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:20 smithi146 ceph-mon[103278]: from='client.87206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3786494442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:21.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:20 smithi078 ceph-mon[91136]: from='client.87206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:21.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3786494442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:20 smithi078 ceph-mon[99959]: from='client.87206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3786494442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:21.918 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:21.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:21.918 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:21.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:21.918 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:21.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:21 smithi146 ceph-mon[103278]: pgmap v10241: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T02:01:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:21 smithi078 ceph-mon[99959]: pgmap v10241: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T02:01:22.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:21 smithi078 ceph-mon[91136]: pgmap v10241: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 144 op/s 2024-09-21T02:01:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:24.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:24 smithi146 ceph-mon[103278]: pgmap v10242: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:01:24.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:24 smithi078 ceph-mon[91136]: pgmap v10242: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:01:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:24 smithi078 ceph-mon[99959]: pgmap v10242: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:01:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:25 smithi146 ceph-mon[103278]: pgmap v10243: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:01:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:25 smithi078 ceph-mon[91136]: pgmap v10243: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:01:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:25 smithi078 ceph-mon[99959]: pgmap v10243: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:01:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:27.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:27 smithi146 ceph-mon[103278]: pgmap v10244: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:01:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:27 smithi078 ceph-mon[91136]: pgmap v10244: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:01:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:27 smithi078 ceph-mon[99959]: pgmap v10244: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:01:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:29 smithi146 ceph-mon[103278]: pgmap v10245: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:01:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:29 smithi078 ceph-mon[91136]: pgmap v10245: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:01:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:29 smithi078 ceph-mon[99959]: pgmap v10245: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:01:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:01:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:01:31.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:31.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:31.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:31.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:31.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:31.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:32.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:31 smithi078 ceph-mon[91136]: pgmap v10246: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:01:32.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:31 smithi078 ceph-mon[99959]: pgmap v10246: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:01:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:31 smithi146 ceph-mon[103278]: pgmap v10246: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:01:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:33 smithi146 ceph-mon[103278]: pgmap v10247: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T02:01:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:33 smithi078 ceph-mon[99959]: pgmap v10247: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T02:01:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:33 smithi078 ceph-mon[91136]: pgmap v10247: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T02:01:36.178 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:35 smithi078 ceph-mon[99959]: pgmap v10248: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:36.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:35 smithi078 ceph-mon[91136]: pgmap v10248: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:35 smithi146 ceph-mon[103278]: pgmap v10248: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:38 smithi146 ceph-mon[103278]: pgmap v10249: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[91136]: pgmap v10249: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:38 smithi078 ceph-mon[99959]: pgmap v10249: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:40.208 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:39 smithi146 ceph-mon[103278]: pgmap v10250: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:39 smithi078 ceph-mon[91136]: pgmap v10250: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:39 smithi078 ceph-mon[99959]: pgmap v10250: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:01:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:01:41.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:41.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:41.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:41.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:41.913 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:41.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:41 smithi146 ceph-mon[103278]: pgmap v10251: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:41 smithi078 ceph-mon[91136]: pgmap v10251: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:41 smithi078 ceph-mon[99959]: pgmap v10251: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:44 smithi146 ceph-mon[103278]: pgmap v10252: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:44 smithi078 ceph-mon[91136]: pgmap v10252: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:44 smithi078 ceph-mon[99959]: pgmap v10252: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:45 smithi078 ceph-mon[91136]: pgmap v10253: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:45 smithi078 ceph-mon[99959]: pgmap v10253: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:45 smithi146 ceph-mon[103278]: pgmap v10253: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:47 smithi146 ceph-mon[103278]: pgmap v10254: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:47 smithi078 ceph-mon[91136]: pgmap v10254: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:47 smithi078 ceph-mon[99959]: pgmap v10254: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:49 smithi146 ceph-mon[103278]: pgmap v10255: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:49 smithi078 ceph-mon[91136]: pgmap v10255: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:49 smithi078 ceph-mon[99959]: pgmap v10255: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:50.754 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:01:50.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:01:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:01:51.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:50 smithi146 ceph-mon[103278]: from='client.67299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:51.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:50 smithi078 ceph-mon[99959]: from='client.67299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 5h - - 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 5h - - 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 13s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 8s ago 5h 88.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 13s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 8s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 13s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 13s ago 5h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 8s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:01:51.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 13s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 13s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 8s ago 5h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 13s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 13s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 13s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 13s ago 5h 97.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 8s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 8s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 8s ago 5h 93.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 8s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 8s ago 5h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 13s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:01:51.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 13s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:01:51.282 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:01:51.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:50 smithi078 ceph-mon[91136]: from='client.67299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:51.594 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:01:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:01:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:01:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:01:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:01:51.596 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:01:51.864 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:01:51.865 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:01:51.866 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:51.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:51.866 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:51.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:01:51.866 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:01:51.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:01:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:01:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:52.178 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:01:52.178 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:52.178 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[91136]: from='client.67305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[91136]: from='client.67311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/810959486' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[91136]: pgmap v10256: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[99959]: from='client.67305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[99959]: from='client.67311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/810959486' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:52.179 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:51 smithi078 ceph-mon[99959]: pgmap v10256: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:51 smithi146 ceph-mon[103278]: from='client.67305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:51 smithi146 ceph-mon[103278]: from='client.67311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/810959486' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:01:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:51 smithi146 ceph-mon[103278]: pgmap v10256: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:01:53.151 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:52 smithi078 ceph-mon[99959]: from='client.87242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:53.151 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4037446567' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:53.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:52 smithi078 ceph-mon[91136]: from='client.87242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:53.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4037446567' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:52 smithi146 ceph-mon[103278]: from='client.87242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:01:53.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4037446567' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:01:54.180 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:53 smithi078 ceph-mon[99959]: pgmap v10257: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:54.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:53 smithi078 ceph-mon[91136]: pgmap v10257: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:53 smithi146 ceph-mon[103278]: pgmap v10257: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:01:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:55 smithi146 ceph-mon[103278]: pgmap v10258: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:55 smithi078 ceph-mon[99959]: pgmap v10258: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:55 smithi078 ceph-mon[91136]: pgmap v10258: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:57.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:57.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:01:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[91136]: pgmap v10259: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[99959]: pgmap v10259: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:01:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:01:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:58 smithi146 ceph-mon[103278]: pgmap v10259: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:01:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:01:59 smithi146 ceph-mon[103278]: pgmap v10260: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:01:59 smithi078 ceph-mon[91136]: pgmap v10260: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:01:59 smithi078 ceph-mon[99959]: pgmap v10260: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:02:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:02:01.915 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:01.584Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:01.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:01.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:01.916 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:01.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:01.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:01 smithi146 ceph-mon[103278]: pgmap v10261: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:01 smithi078 ceph-mon[99959]: pgmap v10261: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:01 smithi078 ceph-mon[91136]: pgmap v10261: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:04 smithi146 ceph-mon[103278]: pgmap v10262: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:05.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:04 smithi078 ceph-mon[99959]: pgmap v10262: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:04 smithi078 ceph-mon[91136]: pgmap v10262: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:05 smithi146 ceph-mon[103278]: pgmap v10263: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:05 smithi078 ceph-mon[99959]: pgmap v10263: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:05 smithi078 ceph-mon[91136]: pgmap v10263: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:07.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:07 smithi146 ceph-mon[103278]: pgmap v10264: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:07 smithi078 ceph-mon[91136]: pgmap v10264: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:07 smithi078 ceph-mon[99959]: pgmap v10264: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:09 smithi146 ceph-mon[103278]: pgmap v10265: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:09 smithi078 ceph-mon[91136]: pgmap v10265: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:09 smithi078 ceph-mon[99959]: pgmap v10265: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:02:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:02:11.917 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:11.585Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:11.917 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:11.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:11.917 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:11.585Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:11 smithi146 ceph-mon[103278]: pgmap v10266: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:11 smithi078 ceph-mon[99959]: pgmap v10266: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:11 smithi078 ceph-mon[91136]: pgmap v10266: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:13.657 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:13.657 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:14.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:14 smithi078 ceph-mon[91136]: pgmap v10267: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:14.752 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:14 smithi078 ceph-mon[99959]: pgmap v10267: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:14 smithi146 ceph-mon[103278]: pgmap v10267: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:16.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:15 smithi078 ceph-mon[91136]: pgmap v10268: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:16.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:15 smithi078 ceph-mon[99959]: pgmap v10268: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:15 smithi146 ceph-mon[103278]: pgmap v10268: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:17.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:17 smithi146 ceph-mon[103278]: pgmap v10269: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[91136]: pgmap v10269: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[99959]: pgmap v10269: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:20.224 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:19 smithi146 ceph-mon[103278]: pgmap v10270: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:19 smithi078 ceph-mon[91136]: pgmap v10270: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:19 smithi078 ceph-mon[99959]: pgmap v10270: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:02:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:02:21.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:21.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:21.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:21.586Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:21.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:21.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:21 smithi146 ceph-mon[103278]: pgmap v10271: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:21 smithi078 ceph-mon[99959]: pgmap v10271: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:22.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:21 smithi078 ceph-mon[91136]: pgmap v10271: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:22.472 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:02:23.003 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:02:23.003 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 5h - - 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 5h - - 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 5s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 0s ago 5h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 5s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 0s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 5s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 5s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 0s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 5s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 5s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 0s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 5s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:02:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 5s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 5s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 5s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 0s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 0s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 0s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 0s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 0s ago 5h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 5s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 5s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:02:23.005 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 0s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:02:23.312 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:02:23.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:02:23.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:02:23.314 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:02:23.578 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:02:23.579 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:02:23.579 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:02:23.579 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:02:23.579 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:02:23.579 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:02:23.579 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:02:23.889 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:02:23.890 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:23.890 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:23.890 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:23.890 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:23.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[99959]: from='client.87254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:23.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:23.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:23.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[99959]: from='client.67341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2419617178' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[91136]: from='client.87254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[91136]: from='client.67341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:23.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2419617178' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:23 smithi146 ceph-mon[103278]: from='client.87254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:23 smithi146 ceph-mon[103278]: from='client.67341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2419617178' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:24 smithi146 ceph-mon[103278]: from='client.87266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:24 smithi146 ceph-mon[103278]: from='client.87278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:24 smithi146 ceph-mon[103278]: pgmap v10272: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/298618657' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[99959]: from='client.87266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[99959]: from='client.87278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[99959]: pgmap v10272: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/298618657' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:25.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[91136]: from='client.87266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[91136]: from='client.87278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[91136]: pgmap v10272: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/298618657' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:25 smithi146 ceph-mon[103278]: pgmap v10273: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:25 smithi078 ceph-mon[99959]: pgmap v10273: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:25 smithi078 ceph-mon[91136]: pgmap v10273: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:27.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:26 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:27.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:26 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:27.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:26 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:27 smithi078 ceph-mon[91136]: pgmap v10274: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:27 smithi078 ceph-mon[99959]: pgmap v10274: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:27 smithi146 ceph-mon[103278]: pgmap v10274: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:29 smithi146 ceph-mon[103278]: pgmap v10275: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:29 smithi078 ceph-mon[91136]: pgmap v10275: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:29 smithi078 ceph-mon[99959]: pgmap v10275: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:02:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:02:31.920 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:31.586Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:31.920 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:31.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:31.920 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:31.587Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:31 smithi146 ceph-mon[103278]: pgmap v10276: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:31 smithi078 ceph-mon[91136]: pgmap v10276: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:31 smithi078 ceph-mon[99959]: pgmap v10276: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:34.189 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:33 smithi078 ceph-mon[99959]: pgmap v10277: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:34.189 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:33 smithi078 ceph-mon[91136]: pgmap v10277: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:33 smithi146 ceph-mon[103278]: pgmap v10277: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:35 smithi146 ceph-mon[103278]: pgmap v10278: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:35 smithi078 ceph-mon[99959]: pgmap v10278: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:35 smithi078 ceph-mon[91136]: pgmap v10278: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:37.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:36 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:37.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:36 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:36 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:37 smithi146 ceph-mon[103278]: pgmap v10279: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:37 smithi078 ceph-mon[91136]: pgmap v10279: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:37 smithi078 ceph-mon[99959]: pgmap v10279: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:40 smithi146 ceph-mon[103278]: pgmap v10280: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:40 smithi078 ceph-mon[91136]: pgmap v10280: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:40 smithi078 ceph-mon[99959]: pgmap v10280: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:02:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:02:41.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:41.587Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:41.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:41.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:41.922 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:41.588Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:42.141 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:41 smithi146 ceph-mon[103278]: pgmap v10281: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:41 smithi078 ceph-mon[99959]: pgmap v10281: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:41 smithi078 ceph-mon[91136]: pgmap v10281: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:02:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:44 smithi146 ceph-mon[103278]: pgmap v10282: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:45.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:44 smithi078 ceph-mon[99959]: pgmap v10282: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:44 smithi078 ceph-mon[91136]: pgmap v10282: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:02:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:45 smithi146 ceph-mon[103278]: pgmap v10283: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:02:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:45 smithi078 ceph-mon[99959]: pgmap v10283: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:02:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:45 smithi078 ceph-mon[91136]: pgmap v10283: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:02:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:46 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:47.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:46 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:47.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:46 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:47 smithi146 ceph-mon[103278]: pgmap v10284: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:02:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:47 smithi078 ceph-mon[91136]: pgmap v10284: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:02:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:47 smithi078 ceph-mon[99959]: pgmap v10284: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:02:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:49 smithi146 ceph-mon[103278]: pgmap v10285: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:02:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:49 smithi078 ceph-mon[91136]: pgmap v10285: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:02:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:49 smithi078 ceph-mon[99959]: pgmap v10285: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:02:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:02:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:02:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:51.588Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:51.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:02:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:02:51.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:02:52.210 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:51 smithi078 ceph-mon[99959]: pgmap v10286: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:02:52.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:51 smithi078 ceph-mon[91136]: pgmap v10286: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:02:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:51 smithi146 ceph-mon[103278]: pgmap v10286: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:02:54.185 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:53 smithi078 ceph-mon[99959]: pgmap v10287: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:02:54.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:53 smithi078 ceph-mon[91136]: pgmap v10287: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:02:54.185 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:02:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:53 smithi146 ceph-mon[103278]: pgmap v10287: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:02:54.711 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:02:54.711 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 5h - - 2024-09-21T02:02:54.711 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 5h - - 2024-09-21T02:02:54.711 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 16s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:02:54.711 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 11s ago 5h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 11s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 5h 535M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 11s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 16s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 11s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 11s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:02:54.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 11s ago 5h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 11s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 11s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 11s ago 5h 74.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 16s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 16s ago 5h 90.6M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:02:54.713 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 11s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:02:55.026 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:02:55.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:02:55.028 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:02:55.028 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:02:55.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:02:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:02:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:54 smithi078 ceph-mon[91136]: from='client.87290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:54 smithi078 ceph-mon[91136]: from='client.87296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:54 smithi078 ceph-mon[91136]: from='client.67383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:54 smithi078 ceph-mon[99959]: from='client.87290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.028 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:54 smithi078 ceph-mon[99959]: from='client.87296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.029 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:54 smithi078 ceph-mon[99959]: from='client.67383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:54 smithi146 ceph-mon[103278]: from='client.87290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:54 smithi146 ceph-mon[103278]: from='client.87296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:54 smithi146 ceph-mon[103278]: from='client.67383 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:55.302 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:02:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:02:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:02:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:02:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:02:55.302 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:02:55.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:02:55.612 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:02:55.612 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:55.612 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:02:55.612 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:55.612 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2867643930' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[99959]: from='client.67395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2330337295' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[99959]: pgmap v10288: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2867643930' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[91136]: from='client.67395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2330337295' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:56.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:55 smithi078 ceph-mon[91136]: pgmap v10288: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:02:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2867643930' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:02:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:55 smithi146 ceph-mon[103278]: from='client.67395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:02:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2330337295' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:02:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:55 smithi146 ceph-mon[103278]: pgmap v10288: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:02:57.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:56 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:57.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:56 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:56 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:02:58.169 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:57 smithi078 ceph-mon[99959]: pgmap v10289: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:02:58.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:57 smithi078 ceph-mon[91136]: pgmap v10289: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:02:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:57 smithi146 ceph-mon[103278]: pgmap v10289: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:02:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:02:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:02:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:00 smithi146 ceph-mon[103278]: pgmap v10290: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:03:00.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:00 smithi078 ceph-mon[99959]: pgmap v10290: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:03:00.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:00 smithi078 ceph-mon[91136]: pgmap v10290: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:03:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:03:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:03:01.925 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:01.589Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:01.925 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:01.589Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:01.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:01.590Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:01 smithi146 ceph-mon[103278]: pgmap v10291: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:03:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:01 smithi078 ceph-mon[99959]: pgmap v10291: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:03:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:01 smithi078 ceph-mon[91136]: pgmap v10291: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:03:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:03 smithi146 ceph-mon[103278]: pgmap v10292: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:03:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[99959]: pgmap v10292: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:03:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:03 smithi078 ceph-mon[91136]: pgmap v10292: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:03:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:05 smithi146 ceph-mon[103278]: pgmap v10293: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:03:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:05 smithi078 ceph-mon[99959]: pgmap v10293: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:03:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:05 smithi078 ceph-mon[91136]: pgmap v10293: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:03:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:06 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:07.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:06 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:06 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:07 smithi146 ceph-mon[103278]: pgmap v10294: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:03:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:07 smithi078 ceph-mon[91136]: pgmap v10294: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:03:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:07 smithi078 ceph-mon[99959]: pgmap v10294: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:03:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:09 smithi146 ceph-mon[103278]: pgmap v10295: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:03:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:09 smithi078 ceph-mon[91136]: pgmap v10295: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:03:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:09 smithi078 ceph-mon[99959]: pgmap v10295: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:03:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:03:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:03:11.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:11.590Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:11.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:11.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:11.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:11.591Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:11 smithi146 ceph-mon[103278]: pgmap v10296: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:03:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:11 smithi078 ceph-mon[91136]: pgmap v10296: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:03:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:11 smithi078 ceph-mon[99959]: pgmap v10296: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:03:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:14 smithi146 ceph-mon[103278]: pgmap v10297: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:14 smithi078 ceph-mon[91136]: pgmap v10297: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:14 smithi078 ceph-mon[99959]: pgmap v10297: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:16.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:15 smithi078 ceph-mon[99959]: pgmap v10298: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:16.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:15 smithi078 ceph-mon[91136]: pgmap v10298: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:15 smithi146 ceph-mon[103278]: pgmap v10298: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:17.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:16 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:17.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:16 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:16 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:17 smithi146 ceph-mon[103278]: pgmap v10299: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:18.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:17 smithi078 ceph-mon[91136]: pgmap v10299: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:18.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:17 smithi078 ceph-mon[99959]: pgmap v10299: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:20.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:20 smithi078 ceph-mon[91136]: pgmap v10300: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:20 smithi078 ceph-mon[99959]: pgmap v10300: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:20 smithi146 ceph-mon[103278]: pgmap v10300: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:03:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:03:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:21.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:21.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:21.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:22.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:21 smithi078 ceph-mon[99959]: pgmap v10301: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:22.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:21 smithi078 ceph-mon[91136]: pgmap v10301: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:22.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:21 smithi146 ceph-mon[103278]: pgmap v10301: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:24.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.437 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:24 smithi146 ceph-mon[103278]: pgmap v10302: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[91136]: pgmap v10302: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:24 smithi078 ceph-mon[99959]: pgmap v10302: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:25.907 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:03:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:25 smithi146 ceph-mon[103278]: pgmap v10303: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:25 smithi078 ceph-mon[99959]: pgmap v10303: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:25 smithi078 ceph-mon[91136]: pgmap v10303: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:26.434 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:03:26.434 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 5h - - 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 5h - - 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 8s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 3s ago 5h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 8s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 3s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 8s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 8s ago 5h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 3s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 8s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 8s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 3s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 8s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 8s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 8s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:03:26.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 8s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 3s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 3s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 3s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 3s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 3s ago 5h 75.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 8s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 8s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:03:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 3s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:03:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:03:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:03:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:03:26.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:03:26.749 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:03:27.020 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:03:27.021 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:03:27.022 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[99959]: from='client.67407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[99959]: from='client.67413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[99959]: from='client.87338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/281130609' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[91136]: from='client.67407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[91136]: from='client.67413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[91136]: from='client.87338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/281130609' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:27.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:26 smithi146 ceph-mon[103278]: from='client.67407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:26 smithi146 ceph-mon[103278]: from='client.67413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:26 smithi146 ceph-mon[103278]: from='client.87338 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:27.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/281130609' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:27.329 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:03:27.329 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:27.330 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:27 smithi146 ceph-mon[103278]: from='client.87350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/55981336' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:27 smithi146 ceph-mon[103278]: pgmap v10304: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[91136]: from='client.87350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/55981336' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[91136]: pgmap v10304: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[99959]: from='client.87350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/55981336' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:03:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:27 smithi078 ceph-mon[99959]: pgmap v10304: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:29 smithi146 ceph-mon[103278]: pgmap v10305: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:29 smithi078 ceph-mon[91136]: pgmap v10305: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:29 smithi078 ceph-mon[99959]: pgmap v10305: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:03:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:03:31.929 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:31.591Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:31.929 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:31.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:31.929 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:31.592Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:31 smithi146 ceph-mon[103278]: pgmap v10306: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:31 smithi078 ceph-mon[99959]: pgmap v10306: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:31 smithi078 ceph-mon[91136]: pgmap v10306: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:33 smithi146 ceph-mon[103278]: pgmap v10307: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:33 smithi078 ceph-mon[91136]: pgmap v10307: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:33 smithi078 ceph-mon[99959]: pgmap v10307: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:35 smithi146 ceph-mon[103278]: pgmap v10308: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:35 smithi078 ceph-mon[91136]: pgmap v10308: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:35 smithi078 ceph-mon[99959]: pgmap v10308: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:37 smithi146 ceph-mon[103278]: pgmap v10309: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:38.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:37 smithi078 ceph-mon[99959]: pgmap v10309: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:37 smithi078 ceph-mon[91136]: pgmap v10309: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:40 smithi146 ceph-mon[103278]: pgmap v10310: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:40.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:40 smithi078 ceph-mon[91136]: pgmap v10310: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:40.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:40 smithi078 ceph-mon[99959]: pgmap v10310: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:03:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:03:41.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:41.592Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:41.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:41.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:41.930 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:41.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:41 smithi146 ceph-mon[103278]: pgmap v10311: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:41 smithi078 ceph-mon[91136]: pgmap v10311: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:41 smithi078 ceph-mon[99959]: pgmap v10311: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:44 smithi146 ceph-mon[103278]: pgmap v10312: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[99959]: pgmap v10312: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:44 smithi078 ceph-mon[91136]: pgmap v10312: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:45 smithi146 ceph-mon[103278]: pgmap v10313: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:45 smithi078 ceph-mon[91136]: pgmap v10313: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:45 smithi078 ceph-mon[99959]: pgmap v10313: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:47 smithi146 ceph-mon[103278]: pgmap v10314: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:47 smithi078 ceph-mon[91136]: pgmap v10314: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:47 smithi078 ceph-mon[99959]: pgmap v10314: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:49 smithi146 ceph-mon[103278]: pgmap v10315: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:49 smithi078 ceph-mon[91136]: pgmap v10315: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:49 smithi078 ceph-mon[99959]: pgmap v10315: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:03:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:03:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:51.593Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:51.593Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:03:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:03:51.594Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:03:52.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:51 smithi078 ceph-mon[91136]: pgmap v10316: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:52.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:51 smithi078 ceph-mon[99959]: pgmap v10316: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:51 smithi146 ceph-mon[103278]: pgmap v10316: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:53 smithi146 ceph-mon[103278]: pgmap v10317: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:53 smithi078 ceph-mon[91136]: pgmap v10317: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:53 smithi078 ceph-mon[99959]: pgmap v10317: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:03:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:55 smithi146 ceph-mon[103278]: pgmap v10318: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:55 smithi078 ceph-mon[99959]: pgmap v10318: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:55 smithi078 ceph-mon[91136]: pgmap v10318: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:57.625 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:03:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 5h - - 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 5h - - 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 19s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 14s ago 5h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 14s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 19s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 5h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 14s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 19s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 14s ago 5h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:03:58.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 14s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 14s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 14s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 14s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 14s ago 5h 75.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 19s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 19s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:03:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 14s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:03:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:57 smithi146 ceph-mon[103278]: from='client.67443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:57 smithi146 ceph-mon[103278]: pgmap v10319: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:58.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:58.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:57 smithi078 ceph-mon[99959]: from='client.67443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:58.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:57 smithi078 ceph-mon[99959]: pgmap v10319: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:58.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:03:58.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:57 smithi078 ceph-mon[91136]: from='client.67443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:58.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:57 smithi078 ceph-mon[91136]: pgmap v10319: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:03:58.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:03:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:03:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:03:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:03:58.490 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:03:58.772 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:03:58.773 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='client.67449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='client.67455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1697806953' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[99959]: from='client.67467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.096 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:03:59.096 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:03:59.096 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:03:59.096 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='client.67449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.096 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='client.67455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.096 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1697806953' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:59.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:59.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:58 smithi078 ceph-mon[91136]: from='client.67467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='client.67449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='client.67455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:03:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1697806953' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:03:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:03:59.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:03:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:58 smithi146 ceph-mon[103278]: from='client.67467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2721282861' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:03:59 smithi146 ceph-mon[103278]: pgmap v10320: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2721282861' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:03:59 smithi078 ceph-mon[91136]: pgmap v10320: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2721282861' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:03:59 smithi078 ceph-mon[99959]: pgmap v10320: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:04:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:04:01.934 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:01.594Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:01.934 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:01.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:01.934 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:01.595Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:01 smithi146 ceph-mon[103278]: pgmap v10321: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:02.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:01 smithi078 ceph-mon[91136]: pgmap v10321: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:02.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:01 smithi078 ceph-mon[99959]: pgmap v10321: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:04 smithi146 ceph-mon[103278]: pgmap v10322: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[91136]: pgmap v10322: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:04 smithi078 ceph-mon[99959]: pgmap v10322: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:05 smithi146 ceph-mon[103278]: pgmap v10323: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:05 smithi078 ceph-mon[91136]: pgmap v10323: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:05 smithi078 ceph-mon[99959]: pgmap v10323: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:07 smithi146 ceph-mon[103278]: pgmap v10324: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:07 smithi078 ceph-mon[91136]: pgmap v10324: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:07 smithi078 ceph-mon[99959]: pgmap v10324: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:09 smithi146 ceph-mon[103278]: pgmap v10325: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:09 smithi078 ceph-mon[91136]: pgmap v10325: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:09 smithi078 ceph-mon[99959]: pgmap v10325: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:04:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:04:11.934 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:11.595Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:11.935 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:11.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:11.935 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:11.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:11 smithi146 ceph-mon[103278]: pgmap v10326: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:11 smithi078 ceph-mon[99959]: pgmap v10326: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:11 smithi078 ceph-mon[91136]: pgmap v10326: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:14.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:14 smithi146 ceph-mon[103278]: pgmap v10327: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:15.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:14 smithi078 ceph-mon[99959]: pgmap v10327: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:15.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:14 smithi078 ceph-mon[91136]: pgmap v10327: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:15 smithi146 ceph-mon[103278]: pgmap v10328: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:15 smithi078 ceph-mon[91136]: pgmap v10328: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:15 smithi078 ceph-mon[99959]: pgmap v10328: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:18.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:18.069 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:17 smithi078 ceph-mon[99959]: pgmap v10329: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:04:18.070 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:18.070 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:17 smithi078 ceph-mon[91136]: pgmap v10329: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:04:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:17 smithi146 ceph-mon[103278]: pgmap v10329: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:04:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:20.671 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:04:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:04:20.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:20 smithi078 ceph-mon[99959]: pgmap v10330: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:20.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:20 smithi078 ceph-mon[91136]: pgmap v10330: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:20 smithi146 ceph-mon[103278]: pgmap v10330: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:21.907 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:21.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:21.907 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:21.596Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:21.907 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:21.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:21 smithi146 ceph-mon[103278]: pgmap v10331: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:04:22.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:21 smithi078 ceph-mon[91136]: pgmap v10331: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:04:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:21 smithi078 ceph-mon[99959]: pgmap v10331: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:04:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:24 smithi146 ceph-mon[103278]: pgmap v10332: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[99959]: pgmap v10332: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:24 smithi078 ceph-mon[91136]: pgmap v10332: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:25 smithi146 ceph-mon[103278]: pgmap v10333: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:26.018 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 02:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T02:04:25.847547193Z level=info msg="Completed cleanup jobs" duration=11.753714ms 2024-09-21T02:04:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:25 smithi078 ceph-mon[91136]: pgmap v10333: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:25 smithi078 ceph-mon[99959]: pgmap v10333: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:27 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:27 smithi146 ceph-mon[103278]: pgmap v10334: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:27 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:27 smithi078 ceph-mon[91136]: pgmap v10334: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:27 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:27 smithi078 ceph-mon[99959]: pgmap v10334: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:29.425 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:04:29.964 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 5h - - 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 5h - - 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 11s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 6s ago 5h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 11s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 6s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 11s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 11s ago 5h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 6s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 11s ago 5h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 11s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 6s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 11s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:04:29.965 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 11s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 11s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 11s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 6s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 6s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 6s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 6s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 6s ago 5h 68.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 11s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 11s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:04:29.966 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 6s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:04:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:29 smithi146 ceph-mon[103278]: from='client.87398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:29 smithi146 ceph-mon[103278]: from='client.87404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:29 smithi146 ceph-mon[103278]: pgmap v10335: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:04:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:04:30.282 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:04:30.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:29 smithi078 ceph-mon[99959]: from='client.87398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:30.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:29 smithi078 ceph-mon[99959]: from='client.87404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:30.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:29 smithi078 ceph-mon[99959]: pgmap v10335: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:30.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:29 smithi078 ceph-mon[91136]: from='client.87398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:30.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:29 smithi078 ceph-mon[91136]: from='client.87404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:30.283 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:29 smithi078 ceph-mon[91136]: pgmap v10335: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:04:30.552 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:04:30.553 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:04:30.553 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:04:30.553 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:04:30.553 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:04:30.553 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:04:30.863 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:04:30.863 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:04:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:04:30.863 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:04:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:04:30.863 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:04:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:04:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[91136]: from='client.67491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3298674908' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[91136]: from='client.87422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1031025169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[99959]: from='client.67491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3298674908' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[99959]: from='client.87422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1031025169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:31 smithi146 ceph-mon[103278]: from='client.67491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3298674908' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:04:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:31 smithi146 ceph-mon[103278]: from='client.87422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:04:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1031025169' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:04:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:31.596Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:31.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:31.597Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:32 smithi078 ceph-mon[91136]: pgmap v10336: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:32 smithi078 ceph-mon[99959]: pgmap v10336: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:32 smithi146 ceph-mon[103278]: pgmap v10336: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:33 smithi146 ceph-mon[103278]: pgmap v10337: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:33 smithi078 ceph-mon[91136]: pgmap v10337: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:33 smithi078 ceph-mon[99959]: pgmap v10337: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:35 smithi146 ceph-mon[103278]: pgmap v10338: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:35 smithi078 ceph-mon[91136]: pgmap v10338: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:35 smithi078 ceph-mon[99959]: pgmap v10338: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:37 smithi146 ceph-mon[103278]: pgmap v10339: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:37 smithi078 ceph-mon[91136]: pgmap v10339: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:37 smithi078 ceph-mon[99959]: pgmap v10339: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:40.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.067 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.068 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.068 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.068 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:40.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:40 smithi078 ceph-mon[99959]: pgmap v10340: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:40.988 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:04:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:04:40.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:40 smithi078 ceph-mon[91136]: pgmap v10340: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:40 smithi146 ceph-mon[103278]: pgmap v10340: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:41.939 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:41.597Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:41.939 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:41.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:41.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:41.598Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:41 smithi146 ceph-mon[103278]: pgmap v10341: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:41 smithi078 ceph-mon[91136]: pgmap v10341: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:41 smithi078 ceph-mon[99959]: pgmap v10341: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:44.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:44.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:44.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:44.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:44 smithi146 ceph-mon[103278]: pgmap v10342: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[91136]: pgmap v10342: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:44 smithi078 ceph-mon[99959]: pgmap v10342: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:45 smithi146 ceph-mon[103278]: pgmap v10343: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:45 smithi078 ceph-mon[91136]: pgmap v10343: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:45 smithi078 ceph-mon[99959]: pgmap v10343: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:47 smithi146 ceph-mon[103278]: pgmap v10344: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:47 smithi078 ceph-mon[91136]: pgmap v10344: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:47 smithi078 ceph-mon[99959]: pgmap v10344: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:49 smithi146 ceph-mon[103278]: pgmap v10345: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:49 smithi078 ceph-mon[91136]: pgmap v10345: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:49 smithi078 ceph-mon[99959]: pgmap v10345: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:04:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:04:51.942 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:51.598Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:51.942 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:51.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:04:51.942 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:04:51.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:04:52.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:51 smithi078 ceph-mon[99959]: pgmap v10346: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:52.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:51 smithi078 ceph-mon[91136]: pgmap v10346: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:51 smithi146 ceph-mon[103278]: pgmap v10346: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:53 smithi146 ceph-mon[103278]: pgmap v10347: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:53 smithi078 ceph-mon[99959]: pgmap v10347: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:53 smithi078 ceph-mon[91136]: pgmap v10347: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:04:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:55 smithi146 ceph-mon[103278]: pgmap v10348: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:55 smithi078 ceph-mon[99959]: pgmap v10348: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:55 smithi078 ceph-mon[91136]: pgmap v10348: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:57 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:57 smithi146 ceph-mon[103278]: pgmap v10349: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:57 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:57 smithi078 ceph-mon[99959]: pgmap v10349: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:57 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:04:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:57 smithi078 ceph-mon[91136]: pgmap v10349: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:04:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:04:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:04:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:04:59 smithi078 ceph-mon[91136]: pgmap v10350: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:04:59 smithi078 ceph-mon[99959]: pgmap v10350: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:04:59 smithi146 ceph-mon[103278]: pgmap v10350: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:00.721 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:05:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:05:01.159 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:05:01.701 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:01.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:01.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:01.599Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:01.702 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:01.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 5h - - 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 5h - - 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 2s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 18s ago 5h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 2s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 18s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:05:01.702 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 2s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 5h 536M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 18s ago 5h 367M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 2s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 18s ago 5h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 2s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:05:01.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:05:01.704 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 18s ago 5h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:05:01.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 2s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:05:01.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:05:01.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 2s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:05:01.704 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:02.019 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:05:02.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:05:02.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:02.020 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:05:02.020 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:05:02.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:05:02.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:05:02.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[91136]: from='client.87434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[91136]: from='client.87440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[91136]: from='client.67527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[91136]: pgmap v10351: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[99959]: from='client.87434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[99959]: from='client.87440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[99959]: from='client.67527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:01 smithi078 ceph-mon[99959]: pgmap v10351: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:01 smithi146 ceph-mon[103278]: from='client.87434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:01 smithi146 ceph-mon[103278]: from='client.87440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:01 smithi146 ceph-mon[103278]: from='client.67527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:01 smithi146 ceph-mon[103278]: pgmap v10351: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:05:02.291 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:05:02.292 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:05:02.292 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:05:02.292 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:05:02.292 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:05:02.599 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:05:02.599 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:05:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:05:02.599 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:05:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:05:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3881573174' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:02 smithi146 ceph-mon[103278]: from='client.87458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:03.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2707099629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:03.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3881573174' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:03.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:02 smithi078 ceph-mon[99959]: from='client.87458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:03.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2707099629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3881573174' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:02 smithi078 ceph-mon[91136]: from='client.87458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2707099629' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:04 smithi146 ceph-mon[103278]: pgmap v10352: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[91136]: pgmap v10352: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:04 smithi078 ceph-mon[99959]: pgmap v10352: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:06.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:05 smithi146 ceph-mon[103278]: pgmap v10353: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:05 smithi078 ceph-mon[91136]: pgmap v10353: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:05 smithi078 ceph-mon[99959]: pgmap v10353: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:07 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:07 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:07 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:08 smithi146 ceph-mon[103278]: pgmap v10354: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:08 smithi078 ceph-mon[91136]: pgmap v10354: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:08 smithi078 ceph-mon[99959]: pgmap v10354: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:10.957 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:05:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:05:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:10 smithi146 ceph-mon[103278]: pgmap v10355: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:11.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:10 smithi078 ceph-mon[99959]: pgmap v10355: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:11.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:10 smithi078 ceph-mon[91136]: pgmap v10355: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:11.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:11.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:11.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:13.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:12 smithi146 ceph-mon[103278]: pgmap v10356: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:13.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:12 smithi078 ceph-mon[91136]: pgmap v10356: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:12 smithi078 ceph-mon[99959]: pgmap v10356: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:14 smithi146 ceph-mon[103278]: pgmap v10357: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:14 smithi078 ceph-mon[99959]: pgmap v10357: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:14 smithi078 ceph-mon[91136]: pgmap v10357: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:16 smithi146 ceph-mon[103278]: pgmap v10358: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:17.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:16 smithi078 ceph-mon[99959]: pgmap v10358: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:16 smithi078 ceph-mon[91136]: pgmap v10358: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:17 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:18.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:17 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:17 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:18 smithi146 ceph-mon[103278]: pgmap v10359: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:18 smithi078 ceph-mon[99959]: pgmap v10359: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:18 smithi078 ceph-mon[91136]: pgmap v10359: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:20.376 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.376 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.376 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:20.693 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:05:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:05:21.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:21 smithi078 ceph-mon[99959]: pgmap v10360: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:21.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:21 smithi078 ceph-mon[91136]: pgmap v10360: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:21 smithi146 ceph-mon[103278]: pgmap v10360: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:21.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:21.600Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:21.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:21.600Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:21.748 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:21.601Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:23.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:23 smithi078 ceph-mon[91136]: pgmap v10361: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:23 smithi078 ceph-mon[99959]: pgmap v10361: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:23 smithi146 ceph-mon[103278]: pgmap v10361: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:26.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:25 smithi146 ceph-mon[103278]: pgmap v10362: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:25 smithi078 ceph-mon[99959]: pgmap v10362: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:25 smithi078 ceph-mon[91136]: pgmap v10362: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:27 smithi146 ceph-mon[103278]: pgmap v10363: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:27 smithi078 ceph-mon[99959]: pgmap v10363: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:27 smithi078 ceph-mon[91136]: pgmap v10363: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:29 smithi146 ceph-mon[103278]: pgmap v10364: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:29 smithi078 ceph-mon[91136]: pgmap v10364: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:29 smithi078 ceph-mon[99959]: pgmap v10364: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:05:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:05:31.888 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:31.601Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:31.889 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:31.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:31.889 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:31.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:31 smithi146 ceph-mon[103278]: pgmap v10365: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:31 smithi078 ceph-mon[99959]: pgmap v10365: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:31 smithi078 ceph-mon[91136]: pgmap v10365: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:32.896 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:05:33.425 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:05:33.425 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 5h - - 2024-09-21T02:05:33.425 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 5h - - 2024-09-21T02:05:33.425 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 14s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:05:33.425 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 9s ago 5h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 14s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 9s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 14s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 14s ago 5h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 9s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 14s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 14s ago 5h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 9s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 14s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 14s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 14s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 14s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 9s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:05:33.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 9s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 9s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 9s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 9s ago 5h 72.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 14s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 14s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:05:33.427 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 9s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:05:33.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:05:33.741 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:05:34.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:05:34.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:05:34.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:33 smithi078 ceph-mon[91136]: pgmap v10366: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:33 smithi078 ceph-mon[91136]: from='client.67551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:34.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3191411744' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:34.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:33 smithi078 ceph-mon[99959]: pgmap v10366: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:34.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:33 smithi078 ceph-mon[99959]: from='client.67551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:34.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3191411744' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:33 smithi146 ceph-mon[103278]: pgmap v10366: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:33 smithi146 ceph-mon[103278]: from='client.67551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3191411744' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:05:34.326 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:05:34.327 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:05:34.327 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:05:34.327 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:05:34.327 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:05:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:34 smithi146 ceph-mon[103278]: from='client.67557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:34 smithi146 ceph-mon[103278]: from='client.67563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/406593183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:34 smithi078 ceph-mon[91136]: from='client.67557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:35.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:34 smithi078 ceph-mon[91136]: from='client.67563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/406593183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:34 smithi078 ceph-mon[99959]: from='client.67557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:34 smithi078 ceph-mon[99959]: from='client.67563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/406593183' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:05:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:35 smithi146 ceph-mon[103278]: pgmap v10367: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:35 smithi146 ceph-mon[103278]: from='client.87494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:35 smithi078 ceph-mon[91136]: pgmap v10367: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:35 smithi078 ceph-mon[91136]: from='client.87494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:35 smithi078 ceph-mon[99959]: pgmap v10367: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:35 smithi078 ceph-mon[99959]: from='client.87494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:05:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:36 smithi146 ceph-mon[103278]: pgmap v10368: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:37.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:36 smithi078 ceph-mon[91136]: pgmap v10368: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:36 smithi078 ceph-mon[99959]: pgmap v10368: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:37 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:37 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:37 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:39.203 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:38 smithi078 ceph-mon[99959]: pgmap v10369: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:39.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:38 smithi078 ceph-mon[91136]: pgmap v10369: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:38 smithi146 ceph-mon[103278]: pgmap v10369: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:05:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:05:41.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:41 smithi078 ceph-mon[91136]: pgmap v10370: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:41.475 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:41 smithi078 ceph-mon[99959]: pgmap v10370: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:41 smithi146 ceph-mon[103278]: pgmap v10370: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:41.602Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:41.602Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:41.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:43.466 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:43 smithi078 ceph-mon[91136]: pgmap v10371: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:43.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:43 smithi078 ceph-mon[99959]: pgmap v10371: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:43 smithi146 ceph-mon[103278]: pgmap v10371: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:05:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:45 smithi146 ceph-mon[103278]: pgmap v10372: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:45 smithi078 ceph-mon[91136]: pgmap v10372: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:45.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:45 smithi078 ceph-mon[99959]: pgmap v10372: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:47 smithi146 ceph-mon[103278]: pgmap v10373: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:47.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:47 smithi078 ceph-mon[91136]: pgmap v10373: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:47.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:47 smithi078 ceph-mon[99959]: pgmap v10373: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:49 smithi146 ceph-mon[103278]: pgmap v10374: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:49.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:49 smithi078 ceph-mon[91136]: pgmap v10374: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:49 smithi078 ceph-mon[99959]: pgmap v10374: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:05:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:05:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:51 smithi146 ceph-mon[103278]: pgmap v10375: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:51.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:51 smithi078 ceph-mon[91136]: pgmap v10375: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:51 smithi078 ceph-mon[99959]: pgmap v10375: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:51.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:51.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:05:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:05:51.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:05:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:53 smithi146 ceph-mon[103278]: pgmap v10376: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:53.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:53 smithi078 ceph-mon[91136]: pgmap v10376: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:53 smithi078 ceph-mon[99959]: pgmap v10376: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:55 smithi146 ceph-mon[103278]: pgmap v10377: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:55 smithi078 ceph-mon[91136]: pgmap v10377: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:55 smithi078 ceph-mon[99959]: pgmap v10377: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:05:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:57 smithi146 ceph-mon[103278]: pgmap v10378: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:57 smithi078 ceph-mon[91136]: pgmap v10378: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:57.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:57 smithi078 ceph-mon[99959]: pgmap v10378: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:58.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:05:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:59 smithi146 ceph-mon[103278]: pgmap v10379: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:05:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:59 smithi078 ceph-mon[99959]: pgmap v10379: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:05:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:05:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:59 smithi078 ceph-mon[91136]: pgmap v10379: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:05:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:05:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:00.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:06:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:06:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:01 smithi078 ceph-mon[91136]: pgmap v10380: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:01 smithi078 ceph-mon[99959]: pgmap v10380: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:01 smithi146 ceph-mon[103278]: pgmap v10380: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:01.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:01.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:01.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:01.603Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:01.954 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:01.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:03.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:03 smithi078 ceph-mon[91136]: pgmap v10381: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:03.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:03 smithi078 ceph-mon[99959]: pgmap v10381: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:03 smithi146 ceph-mon[103278]: pgmap v10381: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:04.616 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:06:05.143 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 5h - - 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 5h - - 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 5s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 1s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 5s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 1s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 5s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 5s ago 5h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 1s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 5s ago 5h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 5s ago 5h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:06:05.144 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 1s ago 5h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 5s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 5s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 5s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 5s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 1s ago 5h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 5s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 5s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:06:05.145 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:06:05.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[99959]: pgmap v10382: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:05.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.453 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[99959]: from='client.67587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[99959]: from='client.67593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[91136]: pgmap v10382: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[91136]: from='client.67587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:05.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:05 smithi078 ceph-mon[91136]: from='client.67593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:06:05.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:06:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:06:05.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:06:05.456 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:06:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:05 smithi146 ceph-mon[103278]: pgmap v10382: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:05 smithi146 ceph-mon[103278]: from='client.67587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:05 smithi146 ceph-mon[103278]: from='client.67593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:05.721 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:06:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:06:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:06:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:06:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:06:05.721 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:06:05.722 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:06:06.030 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:06:06.030 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:06.030 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:06.030 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:06.031 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[99959]: from='client.87518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1116343729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[99959]: from='client.67611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/506816835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[91136]: from='client.87518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1116343729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[91136]: from='client.67611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/506816835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:06.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:06 smithi146 ceph-mon[103278]: from='client.87518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:06.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1116343729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:06.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:06 smithi146 ceph-mon[103278]: from='client.67611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:06.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/506816835' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:07.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:07 smithi078 ceph-mon[91136]: pgmap v10383: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:07 smithi078 ceph-mon[99959]: pgmap v10383: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:07.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:07 smithi146 ceph-mon[103278]: pgmap v10383: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:08.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:09 smithi146 ceph-mon[103278]: pgmap v10384: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:09.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:09 smithi078 ceph-mon[99959]: pgmap v10384: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:09.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:09 smithi078 ceph-mon[91136]: pgmap v10384: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:06:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:06:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:11 smithi146 ceph-mon[103278]: pgmap v10385: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:11.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:11 smithi078 ceph-mon[91136]: pgmap v10385: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:11 smithi078 ceph-mon[99959]: pgmap v10385: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:11.603Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:11.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:11.604Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:13 smithi146 ceph-mon[103278]: pgmap v10386: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:13 smithi078 ceph-mon[99959]: pgmap v10386: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:13 smithi078 ceph-mon[91136]: pgmap v10386: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:15 smithi146 ceph-mon[103278]: pgmap v10387: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:15 smithi078 ceph-mon[91136]: pgmap v10387: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:15 smithi078 ceph-mon[99959]: pgmap v10387: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:17.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:17 smithi078 ceph-mon[99959]: pgmap v10388: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:17 smithi078 ceph-mon[91136]: pgmap v10388: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:17.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:17 smithi146 ceph-mon[103278]: pgmap v10388: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:19.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:19 smithi078 ceph-mon[91136]: pgmap v10389: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:19.459 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:19 smithi078 ceph-mon[99959]: pgmap v10389: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:19 smithi146 ceph-mon[103278]: pgmap v10389: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:06:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:06:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:21 smithi146 ceph-mon[103278]: pgmap v10390: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:21.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:21 smithi078 ceph-mon[91136]: pgmap v10390: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:21.604Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:21.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:21.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:21 smithi078 ceph-mon[99959]: pgmap v10390: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:23.701 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:23 smithi078 ceph-mon[99959]: pgmap v10391: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:23.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:23 smithi078 ceph-mon[91136]: pgmap v10391: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:23 smithi146 ceph-mon[103278]: pgmap v10391: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:25 smithi146 ceph-mon[103278]: pgmap v10392: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[91136]: pgmap v10392: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[99959]: pgmap v10392: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:27.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:27 smithi078 ceph-mon[91136]: pgmap v10393: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:27 smithi078 ceph-mon[99959]: pgmap v10393: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:27 smithi146 ceph-mon[103278]: pgmap v10393: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:28.504 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:29.330 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:29 smithi146 ceph-mon[103278]: pgmap v10394: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:29.331 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:29 smithi078 ceph-mon[91136]: pgmap v10394: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:29.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:29 smithi078 ceph-mon[99959]: pgmap v10394: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:06:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:06:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:31 smithi146 ceph-mon[103278]: pgmap v10395: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:31.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:31 smithi078 ceph-mon[91136]: pgmap v10395: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:31 smithi078 ceph-mon[99959]: pgmap v10395: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:31.605Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:31.605Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:31.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:33 smithi146 ceph-mon[103278]: pgmap v10396: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:33 smithi078 ceph-mon[91136]: pgmap v10396: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:33 smithi078 ceph-mon[99959]: pgmap v10396: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:35 smithi078 ceph-mon[99959]: pgmap v10397: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:35 smithi078 ceph-mon[91136]: pgmap v10397: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:35 smithi146 ceph-mon[103278]: pgmap v10397: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:36.328 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:06:36.856 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:06:36.856 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 5h - - 2024-09-21T02:06:36.856 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 5h - - 2024-09-21T02:06:36.856 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 17s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 12s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 17s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 12s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 17s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 17s ago 5h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 5h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 17s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 17s ago 5h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 12s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 17s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 17s ago 5h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 17s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:06:36.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 17s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 12s ago 5h 74.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 17s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 17s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:06:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:06:37.167 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:06:37.167 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:06:37.168 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:06:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[91136]: pgmap v10398: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[91136]: from='client.67623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[91136]: from='client.67629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[91136]: from='client.87554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/126419492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[99959]: pgmap v10398: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[99959]: from='client.67623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[99959]: from='client.67629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[99959]: from='client.87554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.436 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/126419492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:37.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:06:37.436 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:06:37.436 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:06:37.437 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:06:37.741 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:06:37.742 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:37.742 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:06:37.742 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:37.742 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:37 smithi146 ceph-mon[103278]: pgmap v10398: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:37 smithi146 ceph-mon[103278]: from='client.67623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:37 smithi146 ceph-mon[103278]: from='client.67629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:37 smithi146 ceph-mon[103278]: from='client.87554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/126419492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:06:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:38 smithi078 ceph-mon[91136]: from='client.87566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1178436494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:38 smithi078 ceph-mon[99959]: from='client.87566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1178436494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:38 smithi146 ceph-mon[103278]: from='client.87566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:06:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1178436494' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:06:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:39 smithi078 ceph-mon[91136]: pgmap v10399: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:39.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:39 smithi078 ceph-mon[99959]: pgmap v10399: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:39 smithi146 ceph-mon[103278]: pgmap v10399: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:06:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:06:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:41 smithi146 ceph-mon[103278]: pgmap v10400: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:42.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:41 smithi078 ceph-mon[91136]: pgmap v10400: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:41 smithi078 ceph-mon[99959]: pgmap v10400: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:41.606Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:41.606Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:41.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:43 smithi146 ceph-mon[103278]: pgmap v10401: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:43 smithi078 ceph-mon[99959]: pgmap v10401: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:44.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:44.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:43 smithi078 ceph-mon[91136]: pgmap v10401: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:44.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[91136]: pgmap v10402: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[99959]: pgmap v10402: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:45 smithi146 ceph-mon[103278]: pgmap v10402: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:06:47.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:47 smithi146 ceph-mon[103278]: pgmap v10403: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:47 smithi078 ceph-mon[99959]: pgmap v10403: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:47.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:47 smithi078 ceph-mon[91136]: pgmap v10403: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:48.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:49 smithi146 ceph-mon[103278]: pgmap v10404: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:49 smithi078 ceph-mon[99959]: pgmap v10404: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:49 smithi078 ceph-mon[91136]: pgmap v10404: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:06:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T02:06:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:51 smithi146 ceph-mon[103278]: pgmap v10405: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:51 smithi078 ceph-mon[91136]: pgmap v10405: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:51 smithi078 ceph-mon[99959]: pgmap v10405: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:51.607Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:51.607Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:06:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:06:51.608Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:06:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:53 smithi146 ceph-mon[103278]: pgmap v10406: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:53 smithi078 ceph-mon[91136]: pgmap v10406: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:53 smithi078 ceph-mon[99959]: pgmap v10406: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:55 smithi146 ceph-mon[103278]: pgmap v10407: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:55 smithi078 ceph-mon[91136]: pgmap v10407: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:55 smithi078 ceph-mon[99959]: pgmap v10407: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:06:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:57 smithi146 ceph-mon[103278]: pgmap v10408: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:57 smithi078 ceph-mon[91136]: pgmap v10408: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:57 smithi078 ceph-mon[99959]: pgmap v10408: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:58.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:06:59.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:59 smithi078 ceph-mon[99959]: pgmap v10409: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:59.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:06:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:59.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:59 smithi078 ceph-mon[91136]: pgmap v10409: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:59.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:06:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:06:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:59 smithi146 ceph-mon[103278]: pgmap v10409: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:06:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:06:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:07:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:07:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:01 smithi146 ceph-mon[103278]: pgmap v10410: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:01 smithi078 ceph-mon[99959]: pgmap v10410: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:01 smithi078 ceph-mon[91136]: pgmap v10410: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:01.608Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:01.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:01.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:03 smithi146 ceph-mon[103278]: pgmap v10411: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:03 smithi078 ceph-mon[91136]: pgmap v10411: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:03 smithi078 ceph-mon[99959]: pgmap v10411: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[99959]: pgmap v10412: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[91136]: pgmap v10412: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:05 smithi146 ceph-mon[103278]: pgmap v10412: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:07 smithi146 ceph-mon[103278]: pgmap v10413: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:07 smithi078 ceph-mon[91136]: pgmap v10413: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:07 smithi078 ceph-mon[99959]: pgmap v10413: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:08.031 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:07:08.558 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:07:08.558 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 5h - - 2024-09-21T02:07:08.558 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 5h - - 2024-09-21T02:07:08.558 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 8s ago 5h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:07:08.558 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 4s ago 5h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:07:08.558 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 8s ago 5h 62.5M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 4s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 8s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 8s ago 5h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 4s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 8s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 8s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 4s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 8s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 8s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 8s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 8s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 4s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:07:08.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:07:08.560 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:07:08.560 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 4s ago 5h 74.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:07:08.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 8s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:07:08.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:07:08.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 8s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:07:08.560 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:07:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:07:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:07:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:07:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:07:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:07:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:07:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:07:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:07:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:07:09.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:07:09.447 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:07:09.447 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:09.447 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:09.447 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:09 smithi146 ceph-mon[103278]: pgmap v10414: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:09 smithi146 ceph-mon[103278]: from='client.67659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:09 smithi146 ceph-mon[103278]: from='client.67665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:09 smithi146 ceph-mon[103278]: from='client.67671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3612423838' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/127107504' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[91136]: pgmap v10414: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[91136]: from='client.67659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[91136]: from='client.67665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[91136]: from='client.67671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3612423838' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/127107504' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[99959]: pgmap v10414: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[99959]: from='client.67659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[99959]: from='client.67665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[99959]: from='client.67671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3612423838' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/127107504' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:10.633 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:10 smithi146 ceph-mon[103278]: from='client.67683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:10 smithi078 ceph-mon[99959]: from='client.67683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:10 smithi078 ceph-mon[91136]: from='client.67683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:07:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:07:11.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:11 smithi078 ceph-mon[99959]: pgmap v10415: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:11.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:11 smithi078 ceph-mon[91136]: pgmap v10415: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:11 smithi146 ceph-mon[103278]: pgmap v10415: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:11.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:11.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:11.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:13 smithi078 ceph-mon[91136]: pgmap v10416: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:13 smithi078 ceph-mon[99959]: pgmap v10416: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:13 smithi146 ceph-mon[103278]: pgmap v10416: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:15 smithi078 ceph-mon[91136]: pgmap v10417: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:15 smithi078 ceph-mon[99959]: pgmap v10417: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:15 smithi146 ceph-mon[103278]: pgmap v10417: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:17 smithi146 ceph-mon[103278]: pgmap v10418: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:17 smithi078 ceph-mon[99959]: pgmap v10418: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:17 smithi078 ceph-mon[91136]: pgmap v10418: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:19.836 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:19 smithi078 ceph-mon[99959]: pgmap v10419: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:19.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:19 smithi078 ceph-mon[91136]: pgmap v10419: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:19 smithi146 ceph-mon[103278]: pgmap v10419: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:07:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:07:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:21.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:21.609Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:21.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:22.223 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:21 smithi078 ceph-mon[99959]: pgmap v10420: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:22.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:21 smithi078 ceph-mon[91136]: pgmap v10420: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:21 smithi146 ceph-mon[103278]: pgmap v10420: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:23 smithi146 ceph-mon[103278]: pgmap v10421: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:23 smithi078 ceph-mon[91136]: pgmap v10421: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:23 smithi078 ceph-mon[99959]: pgmap v10421: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:25.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[91136]: pgmap v10422: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:25.643 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:25 smithi146 ceph-mon[103278]: pgmap v10422: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[99959]: pgmap v10422: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:27 smithi146 ceph-mon[103278]: pgmap v10423: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:27 smithi078 ceph-mon[91136]: pgmap v10423: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:27 smithi078 ceph-mon[99959]: pgmap v10423: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:28.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:28.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:29.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:29.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:29 smithi146 ceph-mon[103278]: pgmap v10424: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:29 smithi078 ceph-mon[91136]: pgmap v10424: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:29 smithi078 ceph-mon[99959]: pgmap v10424: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:07:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:07:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:31 smithi146 ceph-mon[103278]: pgmap v10425: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:31 smithi078 ceph-mon[91136]: pgmap v10425: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:31 smithi078 ceph-mon[99959]: pgmap v10425: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:31.609Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:31.610Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:31.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:34.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:33 smithi146 ceph-mon[103278]: pgmap v10426: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:33 smithi078 ceph-mon[91136]: pgmap v10426: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:33 smithi078 ceph-mon[99959]: pgmap v10426: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:35 smithi146 ceph-mon[103278]: pgmap v10427: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:35 smithi078 ceph-mon[91136]: pgmap v10427: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:35 smithi078 ceph-mon[99959]: pgmap v10427: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:37 smithi146 ceph-mon[103278]: pgmap v10428: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:37 smithi078 ceph-mon[91136]: pgmap v10428: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:37 smithi078 ceph-mon[99959]: pgmap v10428: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:39.746 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:07:39.956 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:39 smithi078 ceph-mon[91136]: pgmap v10429: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:39.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:39 smithi078 ceph-mon[99959]: pgmap v10429: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:39 smithi146 ceph-mon[103278]: pgmap v10429: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:40.283 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:07:40.283 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 5h - - 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 5h - - 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 0s ago 5h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 15s ago 5h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 0s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 15s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 0s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 0s ago 5h 537M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 15s ago 5h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 0s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 0s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 15s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 0s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:07:40.284 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 0s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 0s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 0s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 15s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 15s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 15s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 15s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 15s ago 5h 74.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 0s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 0s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:07:40.285 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 15s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:40.596 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:07:40.597 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:07:40.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:07:40.868 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:07:40.868 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:07:40.868 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:07:40.868 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:07:40.868 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:07:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:07:40.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[91136]: from='client.67695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:40.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:40.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:40.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:40.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3242751812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:40.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[99959]: from='client.67695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:40.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:40.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:40.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:40.869 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3242751812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:40 smithi146 ceph-mon[103278]: from='client.67695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:41.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3242751812' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:07:41.180 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:07:41.180 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:41.180 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:07:41.180 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:41.180 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:07:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:41 smithi146 ceph-mon[103278]: pgmap v10430: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:41 smithi146 ceph-mon[103278]: from='client.67701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:41 smithi146 ceph-mon[103278]: from='client.67707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:41 smithi146 ceph-mon[103278]: from='client.87638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1738909242' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:42.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[91136]: pgmap v10430: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:42.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[91136]: from='client.67701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[91136]: from='client.67707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[91136]: from='client.87638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1738909242' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[99959]: pgmap v10430: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[99959]: from='client.67701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[99959]: from='client.67707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[99959]: from='client.87638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:07:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1738909242' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:07:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:41.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:41.611Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:41.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:43 smithi146 ceph-mon[103278]: pgmap v10431: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:43 smithi078 ceph-mon[91136]: pgmap v10431: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:43 smithi078 ceph-mon[99959]: pgmap v10431: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[91136]: pgmap v10432: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[99959]: pgmap v10432: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:45 smithi146 ceph-mon[103278]: pgmap v10432: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:07:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:47 smithi078 ceph-mon[91136]: pgmap v10433: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:47 smithi078 ceph-mon[99959]: pgmap v10433: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:47 smithi146 ceph-mon[103278]: pgmap v10433: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:49.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:49.228 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:49 smithi146 ceph-mon[103278]: pgmap v10434: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:49 smithi078 ceph-mon[99959]: pgmap v10434: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:49 smithi078 ceph-mon[91136]: pgmap v10434: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:07:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:07:51.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:51 smithi078 ceph-mon[99959]: pgmap v10435: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:51.863 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:51 smithi078 ceph-mon[91136]: pgmap v10435: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:51.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:51.611Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:51.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:51.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:07:51.863 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:07:51.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:07:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:51 smithi146 ceph-mon[103278]: pgmap v10435: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:53 smithi146 ceph-mon[103278]: pgmap v10436: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:53 smithi078 ceph-mon[99959]: pgmap v10436: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:53 smithi078 ceph-mon[91136]: pgmap v10436: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:55 smithi146 ceph-mon[103278]: pgmap v10437: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:55 smithi078 ceph-mon[99959]: pgmap v10437: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:55 smithi078 ceph-mon[91136]: pgmap v10437: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:07:58.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:57 smithi146 ceph-mon[103278]: pgmap v10438: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:57 smithi078 ceph-mon[99959]: pgmap v10438: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:57 smithi078 ceph-mon[91136]: pgmap v10438: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:07:59.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:59.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:07:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:07:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:00.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:07:59 smithi078 ceph-mon[91136]: pgmap v10439: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:00.087 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:07:59 smithi078 ceph-mon[99959]: pgmap v10439: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:07:59 smithi146 ceph-mon[103278]: pgmap v10439: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:08:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:08:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:01 smithi146 ceph-mon[103278]: pgmap v10440: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[91136]: pgmap v10440: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[99959]: pgmap v10440: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:01.612Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:01.612Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:01.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:03 smithi146 ceph-mon[103278]: pgmap v10441: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:03 smithi078 ceph-mon[99959]: pgmap v10441: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:03 smithi078 ceph-mon[91136]: pgmap v10441: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:05 smithi146 ceph-mon[103278]: pgmap v10442: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[99959]: pgmap v10442: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[91136]: pgmap v10442: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:07 smithi146 ceph-mon[103278]: pgmap v10443: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:07 smithi078 ceph-mon[91136]: pgmap v10443: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:07 smithi078 ceph-mon[99959]: pgmap v10443: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:10.020 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:09 smithi146 ceph-mon[103278]: pgmap v10444: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:09 smithi078 ceph-mon[91136]: pgmap v10444: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:09 smithi078 ceph-mon[99959]: pgmap v10444: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:08:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:08:11.473 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:08:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:11.613Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:11.613Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:11.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:11.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:12.002 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 5h - - 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 5h - - 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 11s ago 5h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 7s ago 5h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 11s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 7s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 11s ago 5h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 11s ago 5h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 7s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 11s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 11s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:08:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 7s ago 5h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 11s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 11s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 11s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 11s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 7s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 7s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 7s ago 5h 74.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 11s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 11s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:08:12.004 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:08:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:11 smithi146 ceph-mon[103278]: pgmap v10445: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:11 smithi078 ceph-mon[91136]: pgmap v10445: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:11 smithi078 ceph-mon[99959]: pgmap v10445: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:08:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:08:12.353 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:08:12.620 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:08:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:08:12.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:08:12.932 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:08:12.933 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:12.933 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:12.933 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:12.933 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:12 smithi146 ceph-mon[103278]: from='client.67731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:12 smithi146 ceph-mon[103278]: from='client.67737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1483419092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:12 smithi078 ceph-mon[99959]: from='client.67731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:13.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:12 smithi078 ceph-mon[99959]: from='client.67737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1483419092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:12 smithi078 ceph-mon[91136]: from='client.67731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:12 smithi078 ceph-mon[91136]: from='client.67737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1483419092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:13 smithi146 ceph-mon[103278]: pgmap v10446: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:13 smithi146 ceph-mon[103278]: from='client.87662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:13 smithi146 ceph-mon[103278]: from='client.67755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1340163275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[91136]: pgmap v10446: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[91136]: from='client.87662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[91136]: from='client.67755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1340163275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[99959]: pgmap v10446: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[99959]: from='client.87662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[99959]: from='client.67755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1340163275' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:15.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:14 smithi146 ceph-mon[103278]: pgmap v10447: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:15.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:14 smithi078 ceph-mon[91136]: pgmap v10447: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:15.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:14 smithi078 ceph-mon[99959]: pgmap v10447: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:17 smithi146 ceph-mon[103278]: pgmap v10448: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:17.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:17 smithi078 ceph-mon[91136]: pgmap v10448: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:17 smithi078 ceph-mon[99959]: pgmap v10448: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:19 smithi078 ceph-mon[91136]: pgmap v10449: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:19 smithi078 ceph-mon[99959]: pgmap v10449: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:19 smithi146 ceph-mon[103278]: pgmap v10449: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:08:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:08:21.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[91136]: pgmap v10450: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:21.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.500 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[99959]: pgmap v10450: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:21.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.501 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:21 smithi146 ceph-mon[103278]: pgmap v10450: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:21.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:21.614Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:21.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:23 smithi146 ceph-mon[103278]: pgmap v10451: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:23 smithi078 ceph-mon[99959]: pgmap v10451: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:23 smithi078 ceph-mon[91136]: pgmap v10451: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:25 smithi146 ceph-mon[103278]: pgmap v10452: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[99959]: pgmap v10452: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[91136]: pgmap v10452: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:27.433 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:27 smithi078 ceph-mon[99959]: pgmap v10453: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:27.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:27 smithi078 ceph-mon[91136]: pgmap v10453: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:27 smithi146 ceph-mon[103278]: pgmap v10453: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:28.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:29.509 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:29 smithi078 ceph-mon[99959]: pgmap v10454: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:29.509 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:29.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:29 smithi078 ceph-mon[91136]: pgmap v10454: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:29.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:29 smithi146 ceph-mon[103278]: pgmap v10454: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:08:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:08:31.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:31 smithi146 ceph-mon[103278]: pgmap v10455: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:31 smithi078 ceph-mon[99959]: pgmap v10455: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:31 smithi078 ceph-mon[91136]: pgmap v10455: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:31.614Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:31.615Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:31.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:33 smithi146 ceph-mon[103278]: pgmap v10456: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:33 smithi078 ceph-mon[91136]: pgmap v10456: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:33.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:33 smithi078 ceph-mon[99959]: pgmap v10456: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:35 smithi146 ceph-mon[103278]: pgmap v10457: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:35 smithi078 ceph-mon[91136]: pgmap v10457: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:35 smithi078 ceph-mon[99959]: pgmap v10457: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:37 smithi146 ceph-mon[103278]: pgmap v10458: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:37 smithi078 ceph-mon[91136]: pgmap v10458: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:37.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:37 smithi078 ceph-mon[99959]: pgmap v10458: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:38.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:39 smithi146 ceph-mon[103278]: pgmap v10459: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:39 smithi078 ceph-mon[91136]: pgmap v10459: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:39.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:39 smithi078 ceph-mon[99959]: pgmap v10459: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:08:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[91136]: pgmap v10460: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[99959]: pgmap v10460: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.616 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:41 smithi146 ceph-mon[103278]: pgmap v10460: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:41.615Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:41.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:41.616Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:43.234 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:08:43.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:43 smithi078 ceph-mon[99959]: pgmap v10461: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:43 smithi078 ceph-mon[91136]: pgmap v10461: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 5h - - 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 5h - - 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 18s ago 5h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 3s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 18s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 3s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 3s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:08:43.763 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 18s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 3s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 18s ago 5h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 3s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 3s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 3s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 3s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 18s ago 5h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 3s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:08:43.764 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:08:43.765 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 3s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:08:43.765 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:08:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:43 smithi146 ceph-mon[103278]: pgmap v10461: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:08:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:08:44.077 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:08:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:08:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:08:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:08:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:08:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:08:44.346 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:08:44.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[99959]: from='client.87686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[99959]: from='client.67773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:44.658 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[99959]: from='client.87698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1362520777' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:44.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[91136]: from='client.87686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[91136]: from='client.67773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:44.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[91136]: from='client.87698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1362520777' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:44 smithi146 ceph-mon[103278]: from='client.87686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:44 smithi146 ceph-mon[103278]: from='client.67773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:44 smithi146 ceph-mon[103278]: from='client.87698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1362520777' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:08:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:45 smithi146 ceph-mon[103278]: pgmap v10462: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:45 smithi146 ceph-mon[103278]: from='client.87710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1151068873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[99959]: pgmap v10462: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[99959]: from='client.87710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1151068873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[91136]: pgmap v10462: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[91136]: from='client.87710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1151068873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:08:47.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:47 smithi078 ceph-mon[91136]: pgmap v10463: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:47 smithi078 ceph-mon[99959]: pgmap v10463: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:47 smithi146 ceph-mon[103278]: pgmap v10463: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:48.649 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:48.649 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:49 smithi146 ceph-mon[103278]: pgmap v10464: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:49 smithi078 ceph-mon[91136]: pgmap v10464: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:49 smithi078 ceph-mon[99959]: pgmap v10464: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:08:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:08:51.651 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:51 smithi146 ceph-mon[103278]: pgmap v10465: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:51 smithi078 ceph-mon[91136]: pgmap v10465: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:51 smithi078 ceph-mon[99959]: pgmap v10465: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:51.616Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:51.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:08:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:08:51.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:08:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:53 smithi146 ceph-mon[103278]: pgmap v10466: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:53 smithi078 ceph-mon[91136]: pgmap v10466: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:53 smithi078 ceph-mon[99959]: pgmap v10466: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:55 smithi146 ceph-mon[103278]: pgmap v10467: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:55 smithi078 ceph-mon[91136]: pgmap v10467: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:55 smithi078 ceph-mon[99959]: pgmap v10467: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:08:57.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:57 smithi078 ceph-mon[91136]: pgmap v10468: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:57 smithi078 ceph-mon[99959]: pgmap v10468: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:57 smithi146 ceph-mon[103278]: pgmap v10468: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:58.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:59 smithi146 ceph-mon[103278]: pgmap v10469: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:08:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:59 smithi078 ceph-mon[99959]: pgmap v10469: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:59.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:08:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:08:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:59 smithi078 ceph-mon[91136]: pgmap v10469: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:08:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:08:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:09:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:09:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:01 smithi146 ceph-mon[103278]: pgmap v10470: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[91136]: pgmap v10470: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[99959]: pgmap v10470: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:01.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:01.616Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:01.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:01.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:01.787 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:01.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:03 smithi146 ceph-mon[103278]: pgmap v10471: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:03 smithi078 ceph-mon[91136]: pgmap v10471: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:03 smithi078 ceph-mon[99959]: pgmap v10471: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:05 smithi146 ceph-mon[103278]: pgmap v10472: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[99959]: pgmap v10472: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[91136]: pgmap v10472: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:07 smithi146 ceph-mon[103278]: pgmap v10473: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:07 smithi078 ceph-mon[91136]: pgmap v10473: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:07 smithi078 ceph-mon[99959]: pgmap v10473: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:09 smithi146 ceph-mon[103278]: pgmap v10474: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:09 smithi078 ceph-mon[91136]: pgmap v10474: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:09 smithi078 ceph-mon[99959]: pgmap v10474: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:09:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:09:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:11 smithi146 ceph-mon[103278]: pgmap v10475: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:11.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:11 smithi078 ceph-mon[99959]: pgmap v10475: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:11 smithi078 ceph-mon[91136]: pgmap v10475: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:11.617Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:11.617Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:11.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:13.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:13 smithi146 ceph-mon[103278]: pgmap v10476: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:13 smithi078 ceph-mon[91136]: pgmap v10476: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:13 smithi078 ceph-mon[99959]: pgmap v10476: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:14.953 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:09:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 5h - - 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 5h - - 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 14s ago 5h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 14s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 10s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 14s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 14s ago 6h 538M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:09:15.481 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 10s ago 5h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 14s ago 5h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 14s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 14s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 14s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 14s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 14s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 10s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 10s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 10s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 10s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 10s ago 5h 70.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:09:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 14s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:09:15.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:09:15.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 14s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:09:15.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:09:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:15 smithi078 ceph-mon[91136]: pgmap v10477: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:15 smithi078 ceph-mon[99959]: pgmap v10477: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:15.791 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:09:15.791 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:09:15.791 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:09:15.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:15.791 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:09:15.791 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:09:15.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:09:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:15 smithi146 ceph-mon[103278]: pgmap v10477: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:16.059 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:09:16.059 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:09:16.059 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:09:16.059 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:09:16.060 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:09:16.370 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:09:16.370 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:16.370 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:16.371 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:16.371 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[99959]: from='client.87722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[99959]: from='client.67809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[99959]: from='client.67815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/816115960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1400995132' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[91136]: from='client.87722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[91136]: from='client.67809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[91136]: from='client.67815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/816115960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1400995132' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:16 smithi146 ceph-mon[103278]: from='client.87722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:16 smithi146 ceph-mon[103278]: from='client.67809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:17.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:16 smithi146 ceph-mon[103278]: from='client.67815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/816115960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1400995132' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:17.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:17 smithi078 ceph-mon[99959]: pgmap v10478: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:17 smithi078 ceph-mon[99959]: from='client.67827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:17 smithi078 ceph-mon[91136]: pgmap v10478: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:17 smithi078 ceph-mon[91136]: from='client.67827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:17 smithi146 ceph-mon[103278]: pgmap v10478: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:17 smithi146 ceph-mon[103278]: from='client.67827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:19 smithi146 ceph-mon[103278]: pgmap v10479: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:19 smithi078 ceph-mon[99959]: pgmap v10479: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:19 smithi078 ceph-mon[91136]: pgmap v10479: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:09:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:09:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[99959]: pgmap v10480: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:21.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[91136]: pgmap v10480: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:21.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:21.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:21.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:21.618Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:21.618Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:21.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:21.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:21 smithi146 ceph-mon[103278]: pgmap v10480: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:23 smithi146 ceph-mon[103278]: pgmap v10481: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:24.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:23 smithi078 ceph-mon[99959]: pgmap v10481: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:23 smithi078 ceph-mon[91136]: pgmap v10481: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:25 smithi146 ceph-mon[103278]: pgmap v10482: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[99959]: pgmap v10482: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[91136]: pgmap v10482: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:26.088 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:27 smithi146 ceph-mon[103278]: pgmap v10483: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:27 smithi078 ceph-mon[99959]: pgmap v10483: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:27 smithi078 ceph-mon[91136]: pgmap v10483: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:29.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:29 smithi146 ceph-mon[103278]: pgmap v10484: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:29.876 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:29 smithi078 ceph-mon[91136]: pgmap v10484: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:29.876 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:29 smithi078 ceph-mon[99959]: pgmap v10484: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:30.918 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:09:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:09:31.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:31 smithi078 ceph-mon[91136]: pgmap v10485: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:31.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:31 smithi078 ceph-mon[99959]: pgmap v10485: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:31.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:31.618Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:31.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:31.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:31.747 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:31.619Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:31 smithi146 ceph-mon[103278]: pgmap v10485: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:33 smithi146 ceph-mon[103278]: pgmap v10486: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:33 smithi078 ceph-mon[91136]: pgmap v10486: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:33 smithi078 ceph-mon[99959]: pgmap v10486: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:35 smithi146 ceph-mon[103278]: pgmap v10487: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:35 smithi078 ceph-mon[91136]: pgmap v10487: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:35 smithi078 ceph-mon[99959]: pgmap v10487: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:37 smithi146 ceph-mon[103278]: pgmap v10488: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:37 smithi078 ceph-mon[91136]: pgmap v10488: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:37 smithi078 ceph-mon[99959]: pgmap v10488: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:39 smithi146 ceph-mon[103278]: pgmap v10489: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:39 smithi078 ceph-mon[91136]: pgmap v10489: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:40.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:39 smithi078 ceph-mon[99959]: pgmap v10489: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:09:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:09:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:41 smithi146 ceph-mon[103278]: pgmap v10490: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[91136]: pgmap v10490: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[99959]: pgmap v10490: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:41.619Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:41.620Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:41.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:43 smithi146 ceph-mon[103278]: pgmap v10491: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:43 smithi078 ceph-mon[91136]: pgmap v10491: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:44.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:43 smithi078 ceph-mon[99959]: pgmap v10491: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:45 smithi146 ceph-mon[103278]: pgmap v10492: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[91136]: pgmap v10492: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[99959]: pgmap v10492: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:09:46.665 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:09:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:09:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 6h - - 2024-09-21T02:09:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 6h - - 2024-09-21T02:09:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 6s ago 5h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:09:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 1s ago 5h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:09:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 6s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 1s ago 5h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 6s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 6s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 1s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 6s ago 5h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 6s ago 5h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 1s ago 5h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 6s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 6s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 6s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 6s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:09:47.200 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 1s ago 5h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 6s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 6s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:09:47.201 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:09:47.506 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:09:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:09:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:09:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:09:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:09:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:09:47.508 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:09:47.778 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:09:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:09:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:09:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:09:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:09:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:09:47.780 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:09:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:47 smithi146 ceph-mon[103278]: pgmap v10493: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:47 smithi146 ceph-mon[103278]: from='client.67839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:47 smithi146 ceph-mon[103278]: from='client.87764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1463845907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[91136]: pgmap v10493: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[91136]: from='client.67839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[91136]: from='client.87764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1463845907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[99959]: pgmap v10493: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[99959]: from='client.67839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[99959]: from='client.87764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1463845907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:09:48.087 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:09:48.088 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:09:48.088 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:09:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[91136]: from='client.87770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[91136]: from='client.67863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3897120029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[99959]: from='client.87770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[99959]: from='client.67863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3897120029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:48 smithi146 ceph-mon[103278]: from='client.87770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:48 smithi146 ceph-mon[103278]: from='client.67863 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:09:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3897120029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:09:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:49 smithi078 ceph-mon[99959]: pgmap v10494: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:49 smithi078 ceph-mon[91136]: pgmap v10494: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:49 smithi146 ceph-mon[103278]: pgmap v10494: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:09:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:09:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:51 smithi078 ceph-mon[91136]: pgmap v10495: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:51 smithi078 ceph-mon[99959]: pgmap v10495: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:51.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:51.621Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:09:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:09:51.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:09:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:51 smithi146 ceph-mon[103278]: pgmap v10495: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:53.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:53 smithi146 ceph-mon[103278]: pgmap v10496: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:53 smithi078 ceph-mon[91136]: pgmap v10496: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:53 smithi078 ceph-mon[99959]: pgmap v10496: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:55 smithi078 ceph-mon[99959]: pgmap v10497: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:55 smithi078 ceph-mon[91136]: pgmap v10497: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:55 smithi146 ceph-mon[103278]: pgmap v10497: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:09:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:57 smithi078 ceph-mon[91136]: pgmap v10498: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:57 smithi078 ceph-mon[99959]: pgmap v10498: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:58.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:57 smithi146 ceph-mon[103278]: pgmap v10498: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:09:59.087 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:59.088 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:09:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:09:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:09:59 smithi146 ceph-mon[103278]: pgmap v10499: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:09:59 smithi078 ceph-mon[99959]: pgmap v10499: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:09:59 smithi078 ceph-mon[91136]: pgmap v10499: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T02:09:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:00.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:00.894 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:10:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:10:00.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:01.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:01.884 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:01.621Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:01.884 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:01.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:01.884 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:01.622Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:01 smithi146 ceph-mon[103278]: pgmap v10500: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[99959]: pgmap v10500: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[91136]: pgmap v10500: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:03 smithi146 ceph-mon[103278]: pgmap v10501: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:03 smithi078 ceph-mon[91136]: pgmap v10501: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:03 smithi078 ceph-mon[99959]: pgmap v10501: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:05 smithi146 ceph-mon[103278]: pgmap v10502: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[99959]: pgmap v10502: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[91136]: pgmap v10502: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:07 smithi146 ceph-mon[103278]: pgmap v10503: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:07 smithi078 ceph-mon[91136]: pgmap v10503: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:07 smithi078 ceph-mon[99959]: pgmap v10503: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:09.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:09 smithi146 ceph-mon[103278]: pgmap v10504: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:09 smithi078 ceph-mon[91136]: pgmap v10504: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:09 smithi078 ceph-mon[99959]: pgmap v10504: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:10.993 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:10:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:10:11.909 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:11.622Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:11.910 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:11.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:11.910 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:11.623Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:12.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:11 smithi078 ceph-mon[91136]: pgmap v10505: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:12.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:11 smithi078 ceph-mon[99959]: pgmap v10505: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:11 smithi146 ceph-mon[103278]: pgmap v10505: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:13 smithi078 ceph-mon[91136]: pgmap v10506: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:13 smithi078 ceph-mon[99959]: pgmap v10506: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:14.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:13 smithi146 ceph-mon[103278]: pgmap v10506: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:14.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:16.236 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:15 smithi146 ceph-mon[103278]: pgmap v10507: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:15 smithi078 ceph-mon[99959]: pgmap v10507: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:15 smithi078 ceph-mon[91136]: pgmap v10507: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:17 smithi146 ceph-mon[103278]: pgmap v10508: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:18.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:17 smithi078 ceph-mon[91136]: pgmap v10508: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:17 smithi078 ceph-mon[99959]: pgmap v10508: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:18.383 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 18s ago 5h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 13s ago 5h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 18s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 18s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 18s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 13s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 18s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 18s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:10:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 13s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 18s ago 5h 91.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 18s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 18s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 18s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 13s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 13s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 13s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 13s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 13s ago 5h 70.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 18s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 18s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:10:18.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:10:19.224 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:10:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:10:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:10:19.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:10:19.225 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:10:19.225 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:19.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:10:19.494 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:10:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:10:19.496 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:10:19.496 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:10:19.496 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:10:19.801 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:19.802 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:19.802 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:19.802 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:19.802 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:19 smithi146 ceph-mon[103278]: pgmap v10509: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:19 smithi146 ceph-mon[103278]: from='client.67875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:19 smithi146 ceph-mon[103278]: from='client.87800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:19 smithi146 ceph-mon[103278]: from='client.67887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/575473975' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4186327330' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[91136]: pgmap v10509: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[91136]: from='client.67875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[91136]: from='client.87800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[91136]: from='client.67887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/575473975' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4186327330' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[99959]: pgmap v10509: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[99959]: from='client.67875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[99959]: from='client.87800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[99959]: from='client.67887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/575473975' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4186327330' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:20.930 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:10:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:10:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:20 smithi146 ceph-mon[103278]: from='client.87818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:21.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:20 smithi078 ceph-mon[91136]: from='client.87818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:20 smithi078 ceph-mon[99959]: from='client.87818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:21.932 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:21.623Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:21.932 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:21.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:21.932 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:21.624Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:21 smithi146 ceph-mon[103278]: pgmap v10510: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[99959]: pgmap v10510: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[91136]: pgmap v10510: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:23 smithi146 ceph-mon[103278]: pgmap v10511: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:23 smithi078 ceph-mon[91136]: pgmap v10511: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:23 smithi078 ceph-mon[99959]: pgmap v10511: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:25 smithi146 ceph-mon[103278]: pgmap v10512: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[91136]: pgmap v10512: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[99959]: pgmap v10512: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:27 smithi146 ceph-mon[103278]: pgmap v10513: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:27 smithi078 ceph-mon[91136]: pgmap v10513: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:27 smithi078 ceph-mon[99959]: pgmap v10513: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:29 smithi146 ceph-mon[103278]: pgmap v10514: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:29 smithi078 ceph-mon[91136]: pgmap v10514: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:29 smithi078 ceph-mon[99959]: pgmap v10514: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:30.956 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:10:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:10:31.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:30 smithi146 ceph-mon[103278]: pgmap v10515: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:31.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:30 smithi078 ceph-mon[91136]: pgmap v10515: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:31.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:30 smithi078 ceph-mon[99959]: pgmap v10515: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:31.885 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:31.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:31.885 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:31.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:31.885 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:31.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:33.248 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:33 smithi078 ceph-mon[99959]: pgmap v10516: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:33.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:33 smithi078 ceph-mon[91136]: pgmap v10516: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:33 smithi146 ceph-mon[103278]: pgmap v10516: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:35.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:35 smithi146 ceph-mon[103278]: pgmap v10517: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:35 smithi078 ceph-mon[91136]: pgmap v10517: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:35 smithi078 ceph-mon[99959]: pgmap v10517: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:37 smithi146 ceph-mon[103278]: pgmap v10518: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:37 smithi078 ceph-mon[91136]: pgmap v10518: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:37 smithi078 ceph-mon[99959]: pgmap v10518: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:39 smithi146 ceph-mon[103278]: pgmap v10519: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:39 smithi078 ceph-mon[91136]: pgmap v10519: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:39 smithi078 ceph-mon[99959]: pgmap v10519: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:10:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:10:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:41 smithi146 ceph-mon[103278]: pgmap v10520: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:41 smithi078 ceph-mon[91136]: pgmap v10520: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:41 smithi078 ceph-mon[99959]: pgmap v10520: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:41.624Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:41.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:41.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:43 smithi146 ceph-mon[103278]: pgmap v10521: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:43.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:43 smithi078 ceph-mon[91136]: pgmap v10521: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:43 smithi078 ceph-mon[99959]: pgmap v10521: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:45.452 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:45 smithi078 ceph-mon[99959]: pgmap v10522: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:45.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:45 smithi078 ceph-mon[91136]: pgmap v10522: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:45 smithi146 ceph-mon[103278]: pgmap v10522: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[91136]: pgmap v10523: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:46 smithi078 ceph-mon[99959]: pgmap v10523: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:10:47.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:46 smithi146 ceph-mon[103278]: pgmap v10523: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:47 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:47 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:47 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:48 smithi146 ceph-mon[103278]: pgmap v10524: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:49.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:48 smithi078 ceph-mon[99959]: pgmap v10524: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:48 smithi078 ceph-mon[91136]: pgmap v10524: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:50.097 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:10:50.626 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:10:50.626 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 6h - - 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 9s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 4s ago 5h 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 9s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 4s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 9s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 9s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 4s ago 6h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 9s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 9s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 4s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 9s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 9s ago 5h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:10:50.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 9s ago 5h 81.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 9s ago 5h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 4s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 5h 88.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 5h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 5h 89.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 4s ago 5h 70.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 9s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 9s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:10:50.628 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:50.941 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:10:50.942 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:10:50.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:10:50.942 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:10:50.942 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:10:50.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:10:50.942 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:10:50.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:10:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:10:51.213 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:10:51.213 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:10:51.213 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:10:51.213 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:10:51.213 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:10:51.214 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:10:51.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[91136]: pgmap v10525: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:51.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[91136]: from='client.87830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[91136]: from='client.87836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[91136]: from='client.87842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2419776119' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[99959]: pgmap v10525: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[99959]: from='client.87830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[99959]: from='client.87836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[99959]: from='client.87842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.215 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2419776119' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:51 smithi146 ceph-mon[103278]: pgmap v10525: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:10:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:51 smithi146 ceph-mon[103278]: from='client.87830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:51 smithi146 ceph-mon[103278]: from='client.87836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:51 smithi146 ceph-mon[103278]: from='client.87842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2419776119' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:10:51.534 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:10:51.534 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:10:51.534 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:10:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:51.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:51.625Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:10:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:10:51.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:10:52.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:52 smithi078 ceph-mon[99959]: from='client.87854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:52.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/186659369' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:52.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:52 smithi078 ceph-mon[91136]: from='client.87854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:52.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/186659369' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:52 smithi146 ceph-mon[103278]: from='client.87854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:10:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/186659369' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:10:53.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:53 smithi078 ceph-mon[91136]: pgmap v10526: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:53.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:53 smithi078 ceph-mon[99959]: pgmap v10526: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:53 smithi146 ceph-mon[103278]: pgmap v10526: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:10:55.350 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:55 smithi146 ceph-mon[103278]: pgmap v10527: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:10:55.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:55 smithi078 ceph-mon[91136]: pgmap v10527: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:10:55.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:55 smithi078 ceph-mon[99959]: pgmap v10527: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-21T02:10:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:57 smithi146 ceph-mon[103278]: pgmap v10528: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:10:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:57 smithi078 ceph-mon[91136]: pgmap v10528: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:10:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:57 smithi078 ceph-mon[99959]: pgmap v10528: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:10:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:10:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:59 smithi146 ceph-mon[103278]: pgmap v10529: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:10:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:59 smithi078 ceph-mon[91136]: pgmap v10529: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:10:59.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:59 smithi078 ceph-mon[99959]: pgmap v10529: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T02:10:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:11:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:11:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:01 smithi146 ceph-mon[103278]: pgmap v10530: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:11:01.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:01 smithi078 ceph-mon[91136]: pgmap v10530: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:11:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:01 smithi078 ceph-mon[99959]: pgmap v10530: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:11:01.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:01.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:01.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:01.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:01.901 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:01.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:02.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:03 smithi146 ceph-mon[103278]: pgmap v10531: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:11:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:03 smithi078 ceph-mon[91136]: pgmap v10531: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:11:03.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:03 smithi078 ceph-mon[99959]: pgmap v10531: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:11:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:05 smithi146 ceph-mon[103278]: pgmap v10532: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:11:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:05 smithi078 ceph-mon[91136]: pgmap v10532: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:11:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:05 smithi078 ceph-mon[99959]: pgmap v10532: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-09-21T02:11:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:07 smithi146 ceph-mon[103278]: pgmap v10533: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T02:11:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[91136]: pgmap v10533: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T02:11:07.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[99959]: pgmap v10533: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T02:11:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:08.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:09 smithi146 ceph-mon[103278]: pgmap v10534: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-21T02:11:09.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:09 smithi078 ceph-mon[99959]: pgmap v10534: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-21T02:11:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:09 smithi078 ceph-mon[91136]: pgmap v10534: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-21T02:11:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:11:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:11:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:11 smithi146 ceph-mon[103278]: pgmap v10535: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T02:11:11.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:11 smithi078 ceph-mon[91136]: pgmap v10535: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T02:11:11.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:11 smithi078 ceph-mon[99959]: pgmap v10535: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T02:11:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:11.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:11.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:11.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:13 smithi146 ceph-mon[103278]: pgmap v10536: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:11:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:13 smithi078 ceph-mon[99959]: pgmap v10536: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:11:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:13 smithi078 ceph-mon[91136]: pgmap v10536: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:11:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:15 smithi078 ceph-mon[91136]: pgmap v10537: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:11:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:15 smithi078 ceph-mon[99959]: pgmap v10537: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:11:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:15 smithi146 ceph-mon[103278]: pgmap v10537: 161 pgs: 161 active+clean; 585 KiB data, 370 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:11:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:17 smithi146 ceph-mon[103278]: pgmap v10538: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-21T02:11:17.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:17 smithi078 ceph-mon[99959]: pgmap v10538: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-21T02:11:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:17 smithi078 ceph-mon[91136]: pgmap v10538: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-09-21T02:11:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:19 smithi146 ceph-mon[103278]: pgmap v10539: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:11:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:19 smithi078 ceph-mon[91136]: pgmap v10539: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:11:19.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:19 smithi078 ceph-mon[99959]: pgmap v10539: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:11:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:11:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:11:21.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:21 smithi078 ceph-mon[91136]: pgmap v10540: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T02:11:21.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:21 smithi078 ceph-mon[99959]: pgmap v10540: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T02:11:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:21 smithi146 ceph-mon[103278]: pgmap v10540: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T02:11:21.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:21.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:21.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:21.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:21.828 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:11:22.355 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:11:22.355 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 6h - - 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 0s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 16s ago 5h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 0s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 0s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 0s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 16s ago 6h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 0s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 0s ago 5h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 16s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 0s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 0s ago 6h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 0s ago 6h 81.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:11:22.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 0s ago 5h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 16s ago 5h 111M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 16s ago 5h 88.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 16s ago 5h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 16s ago 5h 89.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 5h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 0s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 0s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:11:22.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:11:22.667 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:11:22.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.668 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[91136]: from='client.67947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:22.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.668 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:22 smithi078 ceph-mon[99959]: from='client.67947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:22 smithi146 ceph-mon[103278]: from='client.67947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:22.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:11:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:11:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:11:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:11:22.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:11:23.245 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:11:23.246 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:23.246 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:23.246 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:23.246 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[99959]: pgmap v10541: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[99959]: from='client.87872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[99959]: from='client.67959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1482717058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[99959]: from='client.67971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/623961649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[91136]: pgmap v10541: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[91136]: from='client.87872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[91136]: from='client.67959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1482717058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[91136]: from='client.67971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/623961649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:23 smithi146 ceph-mon[103278]: pgmap v10541: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-09-21T02:11:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:23 smithi146 ceph-mon[103278]: from='client.87872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:23 smithi146 ceph-mon[103278]: from='client.67959 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1482717058' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:23 smithi146 ceph-mon[103278]: from='client.67971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/623961649' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:25 smithi146 ceph-mon[103278]: pgmap v10542: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:11:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:25 smithi078 ceph-mon[99959]: pgmap v10542: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:11:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:25 smithi078 ceph-mon[91136]: pgmap v10542: 161 pgs: 161 active+clean; 585 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:11:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:27 smithi146 ceph-mon[103278]: pgmap v10543: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:11:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[91136]: pgmap v10543: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[99959]: pgmap v10543: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:28.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:29 smithi146 ceph-mon[103278]: pgmap v10544: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:11:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:29 smithi078 ceph-mon[99959]: pgmap v10544: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:11:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:29 smithi078 ceph-mon[91136]: pgmap v10544: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:11:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:11:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:11:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:31 smithi146 ceph-mon[103278]: pgmap v10545: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:11:31.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:31 smithi078 ceph-mon[91136]: pgmap v10545: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:11:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:31 smithi078 ceph-mon[99959]: pgmap v10545: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:11:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:31.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:31.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:31.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:33 smithi146 ceph-mon[103278]: pgmap v10546: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:11:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:33 smithi078 ceph-mon[91136]: pgmap v10546: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:11:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:33 smithi078 ceph-mon[99959]: pgmap v10546: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:11:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:35 smithi146 ceph-mon[103278]: pgmap v10547: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:11:35.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:35 smithi078 ceph-mon[91136]: pgmap v10547: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:11:35.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:35 smithi078 ceph-mon[99959]: pgmap v10547: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:11:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:37 smithi078 ceph-mon[91136]: pgmap v10548: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:11:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:37 smithi078 ceph-mon[99959]: pgmap v10548: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:11:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:37 smithi146 ceph-mon[103278]: pgmap v10548: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:11:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:39 smithi146 ceph-mon[103278]: pgmap v10549: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:39 smithi078 ceph-mon[91136]: pgmap v10549: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:39.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:39 smithi078 ceph-mon[99959]: pgmap v10549: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:11:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:11:41.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:41 smithi146 ceph-mon[103278]: pgmap v10550: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:41 smithi078 ceph-mon[99959]: pgmap v10550: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:41.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:41 smithi078 ceph-mon[91136]: pgmap v10550: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:41.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:41.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:41.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:43 smithi146 ceph-mon[103278]: pgmap v10551: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:43 smithi078 ceph-mon[91136]: pgmap v10551: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:43 smithi078 ceph-mon[99959]: pgmap v10551: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:45 smithi146 ceph-mon[103278]: pgmap v10552: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:45 smithi078 ceph-mon[91136]: pgmap v10552: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:45 smithi078 ceph-mon[99959]: pgmap v10552: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:47 smithi146 ceph-mon[103278]: pgmap v10553: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[99959]: pgmap v10553: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[91136]: pgmap v10553: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:11:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:49 smithi146 ceph-mon[103278]: pgmap v10554: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:49 smithi078 ceph-mon[99959]: pgmap v10554: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:49 smithi078 ceph-mon[91136]: pgmap v10554: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:11:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:11:51.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:51 smithi078 ceph-mon[91136]: pgmap v10555: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:51.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:51 smithi078 ceph-mon[99959]: pgmap v10555: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:51 smithi146 ceph-mon[103278]: pgmap v10555: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:52.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:51.630Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:51.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:11:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:11:51.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:11:53.560 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:11:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:53 smithi146 ceph-mon[103278]: pgmap v10556: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:53 smithi078 ceph-mon[99959]: pgmap v10556: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:53 smithi078 ceph-mon[91136]: pgmap v10556: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:54.088 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 6h - - 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 6h - - 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 12s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 7s ago 5h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 12s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 7s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 12s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 12s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 7s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 12s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 12s ago 5h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 7s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 12s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:11:54.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 12s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 12s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 12s ago 6h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 7s ago 5h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 7s ago 5h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 5h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 5h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 7s ago 5h 68.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 12s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 12s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:11:54.090 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:11:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:11:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:11:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:11:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:11:54.400 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:11:54.667 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:11:54.668 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:11:54.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:54 smithi078 ceph-mon[99959]: from='client.67983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:54.669 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:54 smithi078 ceph-mon[99959]: from='client.67989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:54.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:54 smithi078 ceph-mon[91136]: from='client.67983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:54.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:54 smithi078 ceph-mon[91136]: from='client.67989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:54 smithi146 ceph-mon[103278]: from='client.67983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:54.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:54 smithi146 ceph-mon[103278]: from='client.67989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:54.978 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:11:54.978 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:54.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:11:54.979 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:54.979 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:11:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[99959]: pgmap v10557: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[99959]: from='client.67995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:55.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/880469244' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[99959]: from='client.68007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/91462312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[91136]: pgmap v10557: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[91136]: from='client.67995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/880469244' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[91136]: from='client.68007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/91462312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:55 smithi146 ceph-mon[103278]: pgmap v10557: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:11:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:55 smithi146 ceph-mon[103278]: from='client.67995 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/880469244' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:11:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:55 smithi146 ceph-mon[103278]: from='client.68007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:11:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/91462312' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:11:57.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:57 smithi146 ceph-mon[103278]: pgmap v10558: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:57.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:57 smithi078 ceph-mon[99959]: pgmap v10558: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:57.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:57 smithi078 ceph-mon[91136]: pgmap v10558: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:11:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:59 smithi146 ceph-mon[103278]: pgmap v10559: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:11:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:59 smithi078 ceph-mon[91136]: pgmap v10559: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:11:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:11:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:59 smithi078 ceph-mon[99959]: pgmap v10559: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:11:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:11:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:12:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:12:01.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:01 smithi078 ceph-mon[91136]: pgmap v10560: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:01.631 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:01 smithi078 ceph-mon[99959]: pgmap v10560: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:01 smithi146 ceph-mon[103278]: pgmap v10560: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:01.631Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:01.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:01.631Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:03 smithi146 ceph-mon[103278]: pgmap v10561: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:04.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:03 smithi078 ceph-mon[99959]: pgmap v10561: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:03 smithi078 ceph-mon[91136]: pgmap v10561: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:05 smithi146 ceph-mon[103278]: pgmap v10562: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:05 smithi078 ceph-mon[99959]: pgmap v10562: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:05 smithi078 ceph-mon[91136]: pgmap v10562: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:07 smithi146 ceph-mon[103278]: pgmap v10563: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[91136]: pgmap v10563: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[99959]: pgmap v10563: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:09 smithi146 ceph-mon[103278]: pgmap v10564: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:09 smithi078 ceph-mon[91136]: pgmap v10564: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:09.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:09 smithi078 ceph-mon[99959]: pgmap v10564: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:12:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:12:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:11 smithi146 ceph-mon[103278]: pgmap v10565: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:11.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:11 smithi078 ceph-mon[91136]: pgmap v10565: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:11 smithi078 ceph-mon[99959]: pgmap v10565: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:11.632Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:11.632Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:11.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:13 smithi146 ceph-mon[103278]: pgmap v10566: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:13 smithi078 ceph-mon[91136]: pgmap v10566: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:13 smithi078 ceph-mon[99959]: pgmap v10566: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:15 smithi146 ceph-mon[103278]: pgmap v10567: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:15 smithi078 ceph-mon[91136]: pgmap v10567: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:15.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:15 smithi078 ceph-mon[99959]: pgmap v10567: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:17 smithi146 ceph-mon[103278]: pgmap v10568: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:17.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:17 smithi078 ceph-mon[91136]: pgmap v10568: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:17 smithi078 ceph-mon[99959]: pgmap v10568: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:19 smithi146 ceph-mon[103278]: pgmap v10569: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:19 smithi078 ceph-mon[91136]: pgmap v10569: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:19 smithi078 ceph-mon[99959]: pgmap v10569: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:12:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:12:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:21 smithi146 ceph-mon[103278]: pgmap v10570: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:21 smithi078 ceph-mon[91136]: pgmap v10570: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:21 smithi078 ceph-mon[99959]: pgmap v10570: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:21.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:21.633Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:21.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:23.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:23 smithi146 ceph-mon[103278]: pgmap v10571: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:23 smithi078 ceph-mon[91136]: pgmap v10571: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:23 smithi078 ceph-mon[99959]: pgmap v10571: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:12:25.800 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:12:25.800 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 6h - - 2024-09-21T02:12:25.800 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 6h - - 2024-09-21T02:12:25.800 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 3s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 19s ago 5h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 3s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 19s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 3s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 3s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 19s ago 6h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 3s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 3s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 19s ago 5h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 3s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:12:25.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 3s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 3s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 3s ago 6h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 19s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 19s ago 5h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 19s ago 5h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 19s ago 5h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 19s ago 5h 68.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 3s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 3s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:12:25.802 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 19s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:12:26.112 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:25 smithi078 ceph-mon[91136]: pgmap v10572: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:26.113 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:25 smithi078 ceph-mon[99959]: pgmap v10572: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:12:26.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:12:26.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:12:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:25 smithi146 ceph-mon[103278]: pgmap v10572: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:12:26.380 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:12:26.381 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:12:26.381 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:12:26.381 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:12:26.700 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:12:26.700 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:26.700 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:26.700 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:26.700 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='client.68019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='client.87944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='client.87950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/245963057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2807592785' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:27.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='client.68019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='client.87944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='client.87950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/245963057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2807592785' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='client.68019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='client.87944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='client.87950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/245963057' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2807592785' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:27 smithi146 ceph-mon[103278]: pgmap v10573: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:12:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:27 smithi146 ceph-mon[103278]: from='client.68043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:27 smithi078 ceph-mon[91136]: pgmap v10573: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:12:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:27 smithi078 ceph-mon[91136]: from='client.68043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:27 smithi078 ceph-mon[99959]: pgmap v10573: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:12:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:27 smithi078 ceph-mon[99959]: from='client.68043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:29.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:29.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:29 smithi146 ceph-mon[103278]: pgmap v10574: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:29 smithi078 ceph-mon[91136]: pgmap v10574: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:29 smithi078 ceph-mon[99959]: pgmap v10574: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:12:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:12:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:31 smithi078 ceph-mon[91136]: pgmap v10575: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:31 smithi078 ceph-mon[99959]: pgmap v10575: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:31.633Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:31.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:31.634Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:31 smithi146 ceph-mon[103278]: pgmap v10575: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:33 smithi146 ceph-mon[103278]: pgmap v10576: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:12:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:33 smithi078 ceph-mon[91136]: pgmap v10576: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:12:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:33 smithi078 ceph-mon[99959]: pgmap v10576: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:12:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:35 smithi146 ceph-mon[103278]: pgmap v10577: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:35 smithi078 ceph-mon[91136]: pgmap v10577: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:35 smithi078 ceph-mon[99959]: pgmap v10577: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:37.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:37 smithi146 ceph-mon[103278]: pgmap v10578: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:37.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:37 smithi078 ceph-mon[91136]: pgmap v10578: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:37.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:37 smithi078 ceph-mon[99959]: pgmap v10578: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:39.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:39 smithi146 ceph-mon[103278]: pgmap v10579: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:39 smithi078 ceph-mon[99959]: pgmap v10579: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:39 smithi078 ceph-mon[91136]: pgmap v10579: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:12:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:12:41.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:41 smithi078 ceph-mon[91136]: pgmap v10580: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:41.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:41 smithi078 ceph-mon[99959]: pgmap v10580: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:41.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:41.634Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:41.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:41.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:41.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:41.635Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:41 smithi146 ceph-mon[103278]: pgmap v10580: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:43 smithi146 ceph-mon[103278]: pgmap v10581: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:43 smithi078 ceph-mon[99959]: pgmap v10581: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:43 smithi078 ceph-mon[91136]: pgmap v10581: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:45 smithi146 ceph-mon[103278]: pgmap v10582: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:45 smithi078 ceph-mon[99959]: pgmap v10582: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:45 smithi078 ceph-mon[91136]: pgmap v10582: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:47 smithi146 ceph-mon[103278]: pgmap v10583: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[99959]: pgmap v10583: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[91136]: pgmap v10583: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:12:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:49 smithi146 ceph-mon[103278]: pgmap v10584: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:49 smithi078 ceph-mon[99959]: pgmap v10584: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:49 smithi078 ceph-mon[91136]: pgmap v10584: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:12:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:12:51.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:51 smithi078 ceph-mon[99959]: pgmap v10585: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:51.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:51 smithi078 ceph-mon[91136]: pgmap v10585: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:51.635Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:51.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:12:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:12:51.636Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:12:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:51 smithi146 ceph-mon[103278]: pgmap v10585: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:53 smithi146 ceph-mon[103278]: pgmap v10586: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:54.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:53 smithi078 ceph-mon[99959]: pgmap v10586: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:53 smithi078 ceph-mon[91136]: pgmap v10586: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:55 smithi146 ceph-mon[103278]: pgmap v10587: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:55 smithi078 ceph-mon[99959]: pgmap v10587: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:55 smithi078 ceph-mon[91136]: pgmap v10587: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:56.995 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:12:57.533 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 6h - - 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 6h - - 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 15s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 10s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 15s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 10s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 15s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 15s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 10s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 15s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:12:57.534 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 15s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 10s ago 5h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 15s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 15s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 15s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 15s ago 6h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 10s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 10s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 10s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 10s ago 5h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 10s ago 5h 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 15s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 15s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:12:57.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 10s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:12:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:57 smithi078 ceph-mon[91136]: pgmap v10588: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:57 smithi078 ceph-mon[99959]: pgmap v10588: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:12:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:12:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:12:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:12:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:12:57.846 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:12:58.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:57 smithi146 ceph-mon[103278]: pgmap v10588: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:12:58.157 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:12:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:12:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:12:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:12:58.158 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:12:58.470 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:12:58.471 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:12:58.471 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:12:59.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='client.87974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='client.68061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='client.87986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/850694645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/573016873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='client.87974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='client.68061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='client.87986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/850694645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/573016873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:59.004 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='client.87974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='client.68061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='client.87986 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/850694645' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/573016873' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:12:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:12:59.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:59 smithi146 ceph-mon[103278]: pgmap v10589: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:12:59.862 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:12:59 smithi146 ceph-mon[103278]: from='client.87998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:59 smithi078 ceph-mon[91136]: pgmap v10589: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:12:59 smithi078 ceph-mon[91136]: from='client.87998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:59 smithi078 ceph-mon[99959]: pgmap v10589: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:12:59 smithi078 ceph-mon[99959]: from='client.87998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:13:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:13:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:01 smithi146 ceph-mon[103278]: pgmap v10590: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:01 smithi078 ceph-mon[99959]: pgmap v10590: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:02.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:01 smithi078 ceph-mon[91136]: pgmap v10590: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:01.636Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:01.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:01.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:03 smithi146 ceph-mon[103278]: pgmap v10591: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[99959]: pgmap v10591: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[91136]: pgmap v10591: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:05 smithi146 ceph-mon[103278]: pgmap v10592: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:05 smithi078 ceph-mon[91136]: pgmap v10592: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:05 smithi078 ceph-mon[99959]: pgmap v10592: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:07 smithi146 ceph-mon[103278]: pgmap v10593: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[91136]: pgmap v10593: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[99959]: pgmap v10593: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:09.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:09 smithi146 ceph-mon[103278]: pgmap v10594: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:09 smithi078 ceph-mon[91136]: pgmap v10594: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:10.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:09 smithi078 ceph-mon[99959]: pgmap v10594: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:13:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:13:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:11 smithi078 ceph-mon[91136]: pgmap v10595: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:11 smithi078 ceph-mon[99959]: pgmap v10595: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:11.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:11.637Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:11.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:12.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:11 smithi146 ceph-mon[103278]: pgmap v10595: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:13 smithi146 ceph-mon[103278]: pgmap v10596: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:13 smithi078 ceph-mon[99959]: pgmap v10596: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:13 smithi078 ceph-mon[91136]: pgmap v10596: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:15 smithi146 ceph-mon[103278]: pgmap v10597: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:15 smithi078 ceph-mon[91136]: pgmap v10597: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:15 smithi078 ceph-mon[99959]: pgmap v10597: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:18.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:17 smithi078 ceph-mon[91136]: pgmap v10598: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:18.171 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:17 smithi078 ceph-mon[99959]: pgmap v10598: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:17 smithi146 ceph-mon[103278]: pgmap v10598: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:19.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:19 smithi146 ceph-mon[103278]: pgmap v10599: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:19 smithi078 ceph-mon[91136]: pgmap v10599: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:19 smithi078 ceph-mon[99959]: pgmap v10599: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:20.704 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:13:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:13:21.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:21.637Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:21.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:21.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:21.895 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:21.638Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:22.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:21 smithi078 ceph-mon[99959]: pgmap v10600: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:22.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:21 smithi078 ceph-mon[91136]: pgmap v10600: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:21 smithi146 ceph-mon[103278]: pgmap v10600: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:23 smithi146 ceph-mon[103278]: pgmap v10601: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[91136]: pgmap v10601: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[99959]: pgmap v10601: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:25 smithi146 ceph-mon[103278]: pgmap v10602: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:25 smithi078 ceph-mon[91136]: pgmap v10602: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:25 smithi078 ceph-mon[99959]: pgmap v10602: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:27 smithi146 ceph-mon[103278]: pgmap v10603: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[91136]: pgmap v10603: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[99959]: pgmap v10603: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:28.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:28.765 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 6h - - 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 7s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 2s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 7s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:13:29.292 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 7s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 7s ago 6h 539M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 6h 368M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 7s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 7s ago 5h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 2s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 7s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 7s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 7s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 7s ago 6h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 2s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 2s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 2s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 2s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 5h 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:13:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 7s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:13:29.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:13:29.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 7s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:13:29.294 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 2s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:29 smithi146 ceph-mon[103278]: pgmap v10604: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:29 smithi146 ceph-mon[103278]: from='client.68091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:29 smithi078 ceph-mon[99959]: pgmap v10604: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:29 smithi078 ceph-mon[99959]: from='client.68091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:29 smithi078 ceph-mon[91136]: pgmap v10604: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:29 smithi078 ceph-mon[91136]: from='client.68091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:13:29.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:13:29.601 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:13:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:13:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:13:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:13:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:13:29.871 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:13:30.179 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:13:30.179 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:13:30.179 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:13:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:13:30.180 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:13:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:30 smithi146 ceph-mon[103278]: from='client.68097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:30 smithi146 ceph-mon[103278]: from='client.88022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1437960759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:30 smithi146 ceph-mon[103278]: from='client.88034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1085912370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[91136]: from='client.68097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[91136]: from='client.88022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1437960759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[91136]: from='client.88034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1085912370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[99959]: from='client.68097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[99959]: from='client.88022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1437960759' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[99959]: from='client.88034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:13:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1085912370' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:13:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:13:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:13:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:31 smithi146 ceph-mon[103278]: pgmap v10605: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:31 smithi078 ceph-mon[91136]: pgmap v10605: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:31 smithi078 ceph-mon[99959]: pgmap v10605: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:31.638Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:31.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:31.639Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:33 smithi146 ceph-mon[103278]: pgmap v10606: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:33 smithi078 ceph-mon[91136]: pgmap v10606: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:33.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:33 smithi078 ceph-mon[99959]: pgmap v10606: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:35 smithi146 ceph-mon[103278]: pgmap v10607: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:35.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:35 smithi078 ceph-mon[91136]: pgmap v10607: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:35 smithi078 ceph-mon[99959]: pgmap v10607: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:37.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:37 smithi146 ceph-mon[103278]: pgmap v10608: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:37.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:37 smithi078 ceph-mon[99959]: pgmap v10608: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:37 smithi078 ceph-mon[91136]: pgmap v10608: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:39.431 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:39 smithi078 ceph-mon[99959]: pgmap v10609: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:39.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:39 smithi078 ceph-mon[91136]: pgmap v10609: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:39.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:39 smithi146 ceph-mon[103278]: pgmap v10609: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:13:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:13:41.360 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:41 smithi078 ceph-mon[99959]: pgmap v10610: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:41.361 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:41 smithi078 ceph-mon[91136]: pgmap v10610: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:41.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:41 smithi146 ceph-mon[103278]: pgmap v10610: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:41.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:41.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:41.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:43 smithi146 ceph-mon[103278]: pgmap v10611: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[91136]: pgmap v10611: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[99959]: pgmap v10611: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:44.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:45 smithi146 ceph-mon[103278]: pgmap v10612: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:45 smithi078 ceph-mon[91136]: pgmap v10612: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:45 smithi078 ceph-mon[99959]: pgmap v10612: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:47 smithi146 ceph-mon[103278]: pgmap v10613: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[91136]: pgmap v10613: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[99959]: pgmap v10613: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:13:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:48.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:49 smithi146 ceph-mon[103278]: pgmap v10614: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:49 smithi078 ceph-mon[91136]: pgmap v10614: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:49 smithi078 ceph-mon[99959]: pgmap v10614: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:13:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:13:51.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:51 smithi078 ceph-mon[99959]: pgmap v10615: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:51.640 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:51 smithi078 ceph-mon[91136]: pgmap v10615: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:51 smithi146 ceph-mon[103278]: pgmap v10615: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:51.890 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:51.639Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:51.890 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:51.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:13:51.890 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:13:51.640Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:13:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:53 smithi146 ceph-mon[103278]: pgmap v10616: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:53.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:53 smithi078 ceph-mon[91136]: pgmap v10616: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:53 smithi078 ceph-mon[99959]: pgmap v10616: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:55 smithi146 ceph-mon[103278]: pgmap v10617: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:55 smithi078 ceph-mon[91136]: pgmap v10617: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:55 smithi078 ceph-mon[99959]: pgmap v10617: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:57 smithi146 ceph-mon[103278]: pgmap v10618: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:57 smithi078 ceph-mon[99959]: pgmap v10618: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:57 smithi078 ceph-mon[91136]: pgmap v10618: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:13:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:13:59.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:59 smithi078 ceph-mon[99959]: pgmap v10619: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:59.659 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:13:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:59.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:59 smithi078 ceph-mon[91136]: pgmap v10619: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:59.659 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:13:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:13:59.666 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:59 smithi146 ceph-mon[103278]: pgmap v10619: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:13:59.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:13:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:00.475 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:14:00.736 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:14:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 18s ago 5h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (5h) 13s ago 5h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 18s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 18s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 18s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:14:01.043 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 13s ago 6h 369M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 18s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 18s ago 5h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 13s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 18s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 18s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 18s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 18s ago 6h 97.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 13s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 13s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 13s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 13s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 13s ago 6h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 18s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 18s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:14:01.044 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 13s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:14:01.366 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:14:01.366 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:14:01.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:14:01.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:01.366 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:14:01.367 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:14:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:01 smithi146 ceph-mon[103278]: pgmap v10620: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:01 smithi146 ceph-mon[103278]: from='client.68127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:01.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:01 smithi146 ceph-mon[103278]: from='client.88052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:01.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-mon[99959]: pgmap v10620: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:01.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-mon[99959]: from='client.68127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:01.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-mon[99959]: from='client.88052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:01.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-mon[91136]: pgmap v10620: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:01.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-mon[91136]: from='client.68127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:01.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-mon[91136]: from='client.88052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:14:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:14:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:14:01.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:01.640Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:01.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:01.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:01.926 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:01.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:01.960 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:14:01.960 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:14:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:14:01.960 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:14:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:14:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[91136]: from='client.68139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2755086186' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[91136]: from='client.68151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3725414829' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[99959]: from='client.68139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2755086186' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[99959]: from='client.68151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3725414829' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:02.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:02 smithi146 ceph-mon[103278]: from='client.68139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:02.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2755086186' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:02.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:02 smithi146 ceph-mon[103278]: from='client.68151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:02.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3725414829' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:03 smithi146 ceph-mon[103278]: pgmap v10621: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[91136]: pgmap v10621: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[99959]: pgmap v10621: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:05 smithi146 ceph-mon[103278]: pgmap v10622: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:05 smithi078 ceph-mon[99959]: pgmap v10622: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:05 smithi078 ceph-mon[91136]: pgmap v10622: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:07 smithi146 ceph-mon[103278]: pgmap v10623: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[91136]: pgmap v10623: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[99959]: pgmap v10623: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:09 smithi146 ceph-mon[103278]: pgmap v10624: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:09.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:09 smithi078 ceph-mon[99959]: pgmap v10624: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:09 smithi078 ceph-mon[91136]: pgmap v10624: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:14:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:14:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:11 smithi146 ceph-mon[103278]: pgmap v10625: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:11 smithi078 ceph-mon[91136]: pgmap v10625: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:11 smithi078 ceph-mon[99959]: pgmap v10625: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:11.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:11.641Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:11.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:13 smithi146 ceph-mon[103278]: pgmap v10626: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:13 smithi078 ceph-mon[91136]: pgmap v10626: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:13 smithi078 ceph-mon[99959]: pgmap v10626: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:15 smithi146 ceph-mon[103278]: pgmap v10627: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:15 smithi078 ceph-mon[99959]: pgmap v10627: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:15 smithi078 ceph-mon[91136]: pgmap v10627: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:17 smithi146 ceph-mon[103278]: pgmap v10628: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:17.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:17 smithi078 ceph-mon[99959]: pgmap v10628: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:17.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:17 smithi078 ceph-mon[91136]: pgmap v10628: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:19 smithi146 ceph-mon[103278]: pgmap v10629: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:19 smithi078 ceph-mon[99959]: pgmap v10629: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:19 smithi078 ceph-mon[91136]: pgmap v10629: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:20.656 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:14:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:14:21.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:21 smithi078 ceph-mon[91136]: pgmap v10630: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:21.733 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:21 smithi078 ceph-mon[99959]: pgmap v10630: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:21.734 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:21.641Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:21.734 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:21.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:21.734 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:21.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:21 smithi146 ceph-mon[103278]: pgmap v10630: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:23.933 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:23 smithi146 ceph-mon[103278]: pgmap v10631: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:23.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:23.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:23.934 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[99959]: pgmap v10631: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[91136]: pgmap v10631: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:25.849 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:25 smithi146 ceph-mon[103278]: pgmap v10632: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:25 smithi078 ceph-mon[91136]: pgmap v10632: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:25 smithi078 ceph-mon[99959]: pgmap v10632: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 02:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T02:14:25.849465089Z level=info msg="Completed cleanup jobs" duration=14.339713ms 2024-09-21T02:14:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:27 smithi146 ceph-mon[103278]: pgmap v10633: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[91136]: pgmap v10633: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[99959]: pgmap v10633: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:29.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:29 smithi146 ceph-mon[103278]: pgmap v10634: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:30.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:29 smithi078 ceph-mon[99959]: pgmap v10634: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:29 smithi078 ceph-mon[91136]: pgmap v10634: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:14:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:14:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:31 smithi146 ceph-mon[103278]: pgmap v10635: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:32.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:31 smithi078 ceph-mon[99959]: pgmap v10635: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:31 smithi078 ceph-mon[91136]: pgmap v10635: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:31.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:31.642Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:31.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:32.255 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:14:32.781 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:14:32.781 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 6h - - 2024-09-21T02:14:32.781 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 6h - - 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 10s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 5s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 10s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 5s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 10s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 10s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 5s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 10s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (5h) 10s ago 5h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (5h) 5s ago 5h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 10s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 10s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 10s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 10s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:14:32.782 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 5s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 5s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 5s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 5s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 5s ago 6h 69.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 10s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 10s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:14:32.783 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 5s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:14:33.092 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:14:33.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:14:33.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:14:33.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:14:33.363 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:14:33.364 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:14:33.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[99959]: pgmap v10636: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:33.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[99959]: from='client.68163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:33.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[99959]: from='client.68169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[99959]: from='client.68175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4137825315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[91136]: pgmap v10636: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[91136]: from='client.68163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[91136]: from='client.68169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[91136]: from='client.68175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:33.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4137825315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:33.673 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:14:33.674 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:14:33.674 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:14:33.674 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:14:33.674 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:14:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:33 smithi146 ceph-mon[103278]: pgmap v10636: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:33 smithi146 ceph-mon[103278]: from='client.68163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:33 smithi146 ceph-mon[103278]: from='client.68169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:33 smithi146 ceph-mon[103278]: from='client.68175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4137825315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:14:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:34 smithi146 ceph-mon[103278]: from='client.68187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:35.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1787578625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:34 smithi078 ceph-mon[99959]: from='client.68187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:35.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1787578625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:34 smithi078 ceph-mon[91136]: from='client.68187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:14:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1787578625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:14:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:35 smithi146 ceph-mon[103278]: pgmap v10637: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:35 smithi078 ceph-mon[99959]: pgmap v10637: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:35 smithi078 ceph-mon[91136]: pgmap v10637: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:37 smithi078 ceph-mon[99959]: pgmap v10638: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:37 smithi078 ceph-mon[91136]: pgmap v10638: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:37 smithi146 ceph-mon[103278]: pgmap v10638: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:39.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:39 smithi146 ceph-mon[103278]: pgmap v10639: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:40.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:39 smithi078 ceph-mon[99959]: pgmap v10639: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:39 smithi078 ceph-mon[91136]: pgmap v10639: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:14:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:14:41.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:41 smithi078 ceph-mon[91136]: pgmap v10640: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:41.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:41 smithi078 ceph-mon[99959]: pgmap v10640: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:41.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:41.642Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:41.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:41.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:41.720 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:41.643Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:41.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:41 smithi146 ceph-mon[103278]: pgmap v10640: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:43.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:43 smithi146 ceph-mon[103278]: pgmap v10641: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:43.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:43.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:43.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:43.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:44.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[91136]: pgmap v10641: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[99959]: pgmap v10641: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:45 smithi146 ceph-mon[103278]: pgmap v10642: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:46.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:45 smithi078 ceph-mon[91136]: pgmap v10642: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:45 smithi078 ceph-mon[99959]: pgmap v10642: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[99959]: pgmap v10643: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[91136]: pgmap v10643: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:47 smithi146 ceph-mon[103278]: pgmap v10643: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:14:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:49 smithi146 ceph-mon[103278]: pgmap v10644: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:49 smithi078 ceph-mon[91136]: pgmap v10644: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:49 smithi078 ceph-mon[99959]: pgmap v10644: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:14:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:14:51.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:51 smithi078 ceph-mon[91136]: pgmap v10645: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:51.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:51 smithi078 ceph-mon[99959]: pgmap v10645: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:51.643Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:51.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:14:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:14:51.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:14:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:51 smithi146 ceph-mon[103278]: pgmap v10645: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:53 smithi146 ceph-mon[103278]: pgmap v10646: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:54.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:53 smithi078 ceph-mon[91136]: pgmap v10646: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:53 smithi078 ceph-mon[99959]: pgmap v10646: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:55 smithi146 ceph-mon[103278]: pgmap v10647: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:55 smithi078 ceph-mon[91136]: pgmap v10647: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:55 smithi078 ceph-mon[99959]: pgmap v10647: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:14:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:57 smithi078 ceph-mon[91136]: pgmap v10648: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:57 smithi078 ceph-mon[99959]: pgmap v10648: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:57 smithi146 ceph-mon[103278]: pgmap v10648: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:14:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:14:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:14:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:14:59 smithi146 ceph-mon[103278]: pgmap v10649: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:14:59 smithi078 ceph-mon[91136]: pgmap v10649: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:14:59 smithi078 ceph-mon[99959]: pgmap v10649: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:15:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:15:01.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:01 smithi078 ceph-mon[91136]: pgmap v10650: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:01.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:01 smithi078 ceph-mon[99959]: pgmap v10650: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:01.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:01.644Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:01.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:01.644Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:01.993 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:01.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:15:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:01 smithi146 ceph-mon[103278]: pgmap v10650: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:03.967 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:15:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[91136]: pgmap v10651: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[99959]: pgmap v10651: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:03 smithi146 ceph-mon[103278]: pgmap v10651: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:04.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:04.500 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:15:04.500 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 6h - - 2024-09-21T02:15:04.500 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T02:15:04.500 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (5h) 1s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:15:04.500 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 1s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 1s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 1s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 16s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 1s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 1s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 1s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 1s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 1s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 1s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:15:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 16s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 16s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 16s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 16s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 16s ago 6h 69.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 1s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 1s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:15:04.502 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 16s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:04.815 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:15:04.816 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:15:05.085 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:15:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:15:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:15:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:15:05.086 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:15:05.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:04 smithi078 ceph-mon[99959]: from='client.68199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:05.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:04 smithi078 ceph-mon[91136]: from='client.68199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:05.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:04 smithi146 ceph-mon[103278]: from='client.68199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:15:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:05.395 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:05.395 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:06.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:05 smithi146 ceph-mon[103278]: pgmap v10652: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:06.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:05 smithi146 ceph-mon[103278]: from='client.88124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:06.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:05 smithi146 ceph-mon[103278]: from='client.68211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:06.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1859859770' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:06.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4155380828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[91136]: pgmap v10652: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[91136]: from='client.88124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[91136]: from='client.68211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1859859770' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4155380828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[99959]: pgmap v10652: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[99959]: from='client.88124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[99959]: from='client.68211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1859859770' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4155380828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:06 smithi146 ceph-mon[103278]: from='client.68223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:07.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:06 smithi078 ceph-mon[99959]: from='client.68223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:06 smithi078 ceph-mon[91136]: from='client.68223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:07 smithi146 ceph-mon[103278]: pgmap v10653: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[99959]: pgmap v10653: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[91136]: pgmap v10653: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:09 smithi146 ceph-mon[103278]: pgmap v10654: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:10.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:09 smithi078 ceph-mon[91136]: pgmap v10654: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:09 smithi078 ceph-mon[99959]: pgmap v10654: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:15:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:15:12.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:11 smithi078 ceph-mon[91136]: pgmap v10655: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:11 smithi078 ceph-mon[99959]: pgmap v10655: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:11.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:11.645Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:11.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:15:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:11 smithi146 ceph-mon[103278]: pgmap v10655: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:13 smithi146 ceph-mon[103278]: pgmap v10656: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:14.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:13 smithi078 ceph-mon[91136]: pgmap v10656: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:13 smithi078 ceph-mon[99959]: pgmap v10656: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:15 smithi146 ceph-mon[103278]: pgmap v10657: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:16.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:15 smithi078 ceph-mon[99959]: pgmap v10657: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:15 smithi078 ceph-mon[91136]: pgmap v10657: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:17 smithi146 ceph-mon[103278]: pgmap v10658: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:17 smithi078 ceph-mon[91136]: pgmap v10658: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:17 smithi078 ceph-mon[99959]: pgmap v10658: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:19 smithi146 ceph-mon[103278]: pgmap v10659: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:19 smithi078 ceph-mon[91136]: pgmap v10659: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:19 smithi078 ceph-mon[99959]: pgmap v10659: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:15:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:15:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:21.645Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:21.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:21.646Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:15:22.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:21 smithi078 ceph-mon[99959]: pgmap v10660: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:22.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:21 smithi078 ceph-mon[91136]: pgmap v10660: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:21 smithi146 ceph-mon[103278]: pgmap v10660: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[91136]: pgmap v10661: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[99959]: pgmap v10661: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:23 smithi146 ceph-mon[103278]: pgmap v10661: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:25 smithi146 ceph-mon[103278]: pgmap v10662: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:25 smithi078 ceph-mon[91136]: pgmap v10662: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:25 smithi078 ceph-mon[99959]: pgmap v10662: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:27 smithi146 ceph-mon[103278]: pgmap v10663: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[91136]: pgmap v10663: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[99959]: pgmap v10663: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:29 smithi146 ceph-mon[103278]: pgmap v10664: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:29 smithi078 ceph-mon[91136]: pgmap v10664: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:29 smithi078 ceph-mon[99959]: pgmap v10664: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:15:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:15:31.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:31.646Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:31.938 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:31.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:31.939 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:31.647Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:15:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:31 smithi146 ceph-mon[103278]: pgmap v10665: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:32.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:31 smithi078 ceph-mon[91136]: pgmap v10665: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:32.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:31 smithi078 ceph-mon[99959]: pgmap v10665: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:33 smithi146 ceph-mon[103278]: pgmap v10666: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:33 smithi078 ceph-mon[91136]: pgmap v10666: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:33 smithi078 ceph-mon[99959]: pgmap v10666: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:35.692 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:15:36.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:35 smithi078 ceph-mon[91136]: pgmap v10667: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:36.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:35 smithi078 ceph-mon[99959]: pgmap v10667: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:36.216 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:15:36.216 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 6h - - 2024-09-21T02:15:36.216 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 6h - - 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 13s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 8s ago 6h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 13s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 8s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 13s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 13s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 8s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 13s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 13s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 8s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 13s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 13s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:15:36.217 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 13s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 13s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 8s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 8s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 8s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 8s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 8s ago 6h 73.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 13s ago 5h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 13s ago 5h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:15:36.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 8s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:15:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:35 smithi146 ceph-mon[103278]: pgmap v10667: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:36.526 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:15:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:15:36.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:15:36.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:15:36.528 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:15:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:15:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:15:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:15:36.798 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:15:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:15:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:15:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:15:37.109 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:15:37.109 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:36 smithi078 ceph-mon[99959]: from='client.88154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:37.109 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:36 smithi078 ceph-mon[99959]: from='client.68241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:37.109 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3818676087' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:37.109 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:36 smithi078 ceph-mon[91136]: from='client.88154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:37.109 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:36 smithi078 ceph-mon[91136]: from='client.68241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:37.109 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3818676087' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:36 smithi146 ceph-mon[103278]: from='client.88154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:36 smithi146 ceph-mon[103278]: from='client.68241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3818676087' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:15:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:37 smithi146 ceph-mon[103278]: pgmap v10668: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:37 smithi146 ceph-mon[103278]: from='client.88166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:37 smithi146 ceph-mon[103278]: from='client.68259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/880839089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[91136]: pgmap v10668: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[91136]: from='client.88166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[91136]: from='client.68259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/880839089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[99959]: pgmap v10668: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[99959]: from='client.88166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[99959]: from='client.68259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:15:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/880839089' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:15:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:39 smithi146 ceph-mon[103278]: pgmap v10669: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:39 smithi078 ceph-mon[91136]: pgmap v10669: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:39 smithi078 ceph-mon[99959]: pgmap v10669: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:15:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:15:41.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:41.647Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:41.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:41.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:41.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:41.648Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:15:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:41 smithi146 ceph-mon[103278]: pgmap v10670: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:41 smithi078 ceph-mon[91136]: pgmap v10670: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:41 smithi078 ceph-mon[99959]: pgmap v10670: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:44.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:43 smithi146 ceph-mon[103278]: pgmap v10671: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:44.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[91136]: pgmap v10671: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[99959]: pgmap v10671: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:45 smithi146 ceph-mon[103278]: pgmap v10672: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:45 smithi078 ceph-mon[99959]: pgmap v10672: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:45 smithi078 ceph-mon[91136]: pgmap v10672: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:47 smithi146 ceph-mon[103278]: pgmap v10673: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[99959]: pgmap v10673: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[91136]: pgmap v10673: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:15:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:49.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:49 smithi146 ceph-mon[103278]: pgmap v10674: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:50.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:49 smithi078 ceph-mon[91136]: pgmap v10674: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:49 smithi078 ceph-mon[99959]: pgmap v10674: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:15:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:15:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:51.648Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:51.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:15:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:15:51.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:15:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:51 smithi146 ceph-mon[103278]: pgmap v10675: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:51 smithi078 ceph-mon[91136]: pgmap v10675: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:51 smithi078 ceph-mon[99959]: pgmap v10675: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:53 smithi146 ceph-mon[103278]: pgmap v10676: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:53 smithi078 ceph-mon[91136]: pgmap v10676: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:53 smithi078 ceph-mon[99959]: pgmap v10676: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:55 smithi146 ceph-mon[103278]: pgmap v10677: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:55 smithi078 ceph-mon[91136]: pgmap v10677: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:55 smithi078 ceph-mon[99959]: pgmap v10677: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:15:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:57 smithi146 ceph-mon[103278]: pgmap v10678: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:57 smithi078 ceph-mon[99959]: pgmap v10678: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:57 smithi078 ceph-mon[91136]: pgmap v10678: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:15:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:15:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:15:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:00 smithi146 ceph-mon[103278]: pgmap v10679: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:00 smithi078 ceph-mon[99959]: pgmap v10679: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:00 smithi078 ceph-mon[91136]: pgmap v10679: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:01.016 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:16:00] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T02:16:01.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:01 smithi078 ceph-mon[91136]: pgmap v10680: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:01 smithi078 ceph-mon[99959]: pgmap v10680: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:01 smithi146 ceph-mon[103278]: pgmap v10680: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:01.649Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:01.649Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:01.650Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:03 smithi146 ceph-mon[103278]: pgmap v10681: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:03 smithi078 ceph-mon[91136]: pgmap v10681: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:03 smithi078 ceph-mon[99959]: pgmap v10681: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:04.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:05 smithi146 ceph-mon[103278]: pgmap v10682: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:05.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:05 smithi078 ceph-mon[99959]: pgmap v10682: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:05.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:05 smithi078 ceph-mon[91136]: pgmap v10682: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:07.403 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:16:07.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:07 smithi146 ceph-mon[103278]: pgmap v10683: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:07.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:07 smithi078 ceph-mon[91136]: pgmap v10683: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:07.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:07 smithi078 ceph-mon[99959]: pgmap v10683: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 6h - - 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 6h - - 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 4s ago 6h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 20s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 4s ago 5h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:16:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 20s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 4s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 4s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 20s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 4s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 4s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 20s ago 6h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 4s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 4s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 4s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 4s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 20s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 20s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:16:07.929 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 20s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:16:07.930 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 20s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:16:07.930 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 20s ago 6h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:16:07.930 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 4s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:16:07.930 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 20s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:16:07.930 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 4s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:16:07.930 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 20s ago 5h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:16:08.244 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:16:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:16:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:16:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:16:08.245 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:16:08.512 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:16:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:16:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:16:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:16:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:16:08.513 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:16:08.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='client.68271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.513 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='client.68277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='client.68283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='client.68271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='client.68277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='client.68283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='client.68271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='client.68277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='client.68283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:08.821 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:16:08.821 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:08.821 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:09 smithi146 ceph-mon[103278]: pgmap v10684: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1798317839' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:09 smithi146 ceph-mon[103278]: from='client.88214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3129389706' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:09.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[91136]: pgmap v10684: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1798317839' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[91136]: from='client.88214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3129389706' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[99959]: pgmap v10684: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1798317839' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[99959]: from='client.88214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3129389706' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:16:10] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T02:16:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:11 smithi146 ceph-mon[103278]: pgmap v10685: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:11.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:11 smithi078 ceph-mon[91136]: pgmap v10685: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:11 smithi078 ceph-mon[99959]: pgmap v10685: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:11.650Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:11.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:11.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:13 smithi146 ceph-mon[103278]: pgmap v10686: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:13 smithi078 ceph-mon[91136]: pgmap v10686: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:13 smithi078 ceph-mon[99959]: pgmap v10686: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:14.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:15 smithi146 ceph-mon[103278]: pgmap v10687: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:15 smithi078 ceph-mon[91136]: pgmap v10687: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:15 smithi078 ceph-mon[99959]: pgmap v10687: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:17.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:17 smithi146 ceph-mon[103278]: pgmap v10688: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:17.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:17 smithi078 ceph-mon[99959]: pgmap v10688: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:17 smithi078 ceph-mon[91136]: pgmap v10688: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:18.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:19 smithi146 ceph-mon[103278]: pgmap v10689: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:19.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:19 smithi078 ceph-mon[91136]: pgmap v10689: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:19.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:19 smithi078 ceph-mon[99959]: pgmap v10689: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:16:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:16:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:21 smithi146 ceph-mon[103278]: pgmap v10690: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:21.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:21 smithi078 ceph-mon[91136]: pgmap v10690: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:21 smithi078 ceph-mon[99959]: pgmap v10690: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:21.651Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:21.651Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:21.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:23 smithi146 ceph-mon[103278]: pgmap v10691: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:23.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:23 smithi078 ceph-mon[99959]: pgmap v10691: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:23 smithi078 ceph-mon[91136]: pgmap v10691: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:24.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.496 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.669 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.670 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.670 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:24.670 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:25.487 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:25 smithi146 ceph-mon[103278]: pgmap v10692: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:25.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:25 smithi078 ceph-mon[99959]: pgmap v10692: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:25.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:25 smithi078 ceph-mon[91136]: pgmap v10692: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:27 smithi146 ceph-mon[103278]: pgmap v10693: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:27.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:27 smithi078 ceph-mon[91136]: pgmap v10693: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:27.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:27 smithi078 ceph-mon[99959]: pgmap v10693: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:29 smithi146 ceph-mon[103278]: pgmap v10694: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:29 smithi078 ceph-mon[91136]: pgmap v10694: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:29 smithi078 ceph-mon[99959]: pgmap v10694: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:16:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:16:31.652 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:31 smithi078 ceph-mon[99959]: pgmap v10695: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:31.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:31 smithi078 ceph-mon[91136]: pgmap v10695: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:31 smithi146 ceph-mon[103278]: pgmap v10695: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:31.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:31.652Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:31.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:33 smithi146 ceph-mon[103278]: pgmap v10696: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:33.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:33 smithi078 ceph-mon[99959]: pgmap v10696: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:33 smithi078 ceph-mon[91136]: pgmap v10696: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:35 smithi146 ceph-mon[103278]: pgmap v10697: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:35 smithi078 ceph-mon[99959]: pgmap v10697: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:35 smithi078 ceph-mon[91136]: pgmap v10697: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:37 smithi146 ceph-mon[103278]: pgmap v10698: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:16:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:37 smithi078 ceph-mon[91136]: pgmap v10698: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:16:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:37 smithi078 ceph-mon[99959]: pgmap v10698: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:16:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:38.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:39.116 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:16:39.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:39 smithi078 ceph-mon[99959]: pgmap v10699: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:39 smithi078 ceph-mon[91136]: pgmap v10699: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 6h - - 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 6h - - 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 16s ago 6h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 11s ago 6h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 11s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:16:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 6h 540M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 11s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 16s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 11s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 11s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 11s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 11s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 11s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 11s ago 6h 74.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 16s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:16:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 16s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:16:39.646 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:16:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:39 smithi146 ceph-mon[103278]: pgmap v10699: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:39.956 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:16:39.956 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:16:39.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:16:39.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:39.956 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:16:39.957 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:16:40.226 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:16:40.227 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:16:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[99959]: from='client.68307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[99959]: from='client.68313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[99959]: from='client.68319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/462313605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[91136]: from='client.68307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[91136]: from='client.68313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[91136]: from='client.68319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/462313605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:40.539 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:16:40.539 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:40.539 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:16:40.539 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:40.539 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:16:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:40 smithi146 ceph-mon[103278]: from='client.68307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:40 smithi146 ceph-mon[103278]: from='client.68313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:40 smithi146 ceph-mon[103278]: from='client.68319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:40.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/462313605' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:16:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:16:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:16:41.653 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-mon[99959]: pgmap v10700: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:41.653 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-mon[99959]: from='client.88250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:41.653 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3109049874' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:41.653 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-mon[91136]: pgmap v10700: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:41.653 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-mon[91136]: from='client.88250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:41.653 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3109049874' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:41 smithi146 ceph-mon[103278]: pgmap v10700: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:41 smithi146 ceph-mon[103278]: from='client.88250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:16:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3109049874' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:16:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:41.652Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:41.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:41.653Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:43 smithi146 ceph-mon[103278]: pgmap v10701: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:43.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:43 smithi078 ceph-mon[99959]: pgmap v10701: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:43 smithi078 ceph-mon[91136]: pgmap v10701: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:44.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:45.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:45 smithi146 ceph-mon[103278]: pgmap v10702: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:45 smithi078 ceph-mon[99959]: pgmap v10702: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:45 smithi078 ceph-mon[91136]: pgmap v10702: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:47.597 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:47 smithi078 ceph-mon[91136]: pgmap v10703: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:47.597 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:47 smithi078 ceph-mon[99959]: pgmap v10703: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:47 smithi146 ceph-mon[103278]: pgmap v10703: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:48.683 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.684 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:16:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:49 smithi146 ceph-mon[103278]: pgmap v10704: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:49 smithi078 ceph-mon[91136]: pgmap v10704: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:49 smithi078 ceph-mon[99959]: pgmap v10704: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:16:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:16:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:51 smithi146 ceph-mon[103278]: pgmap v10705: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:51 smithi078 ceph-mon[91136]: pgmap v10705: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:51 smithi078 ceph-mon[99959]: pgmap v10705: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:51.653Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:51.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:16:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:16:51.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:16:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:53 smithi146 ceph-mon[103278]: pgmap v10706: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:53.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:53 smithi078 ceph-mon[99959]: pgmap v10706: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:53 smithi078 ceph-mon[91136]: pgmap v10706: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:55 smithi146 ceph-mon[103278]: pgmap v10707: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:55 smithi078 ceph-mon[91136]: pgmap v10707: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:55 smithi078 ceph-mon[99959]: pgmap v10707: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:57 smithi146 ceph-mon[103278]: pgmap v10708: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:57.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:57 smithi078 ceph-mon[91136]: pgmap v10708: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:57 smithi078 ceph-mon[99959]: pgmap v10708: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:16:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:16:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:59 smithi146 ceph-mon[103278]: pgmap v10709: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:59 smithi078 ceph-mon[91136]: pgmap v10709: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:16:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:59 smithi078 ceph-mon[99959]: pgmap v10709: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:16:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:17:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:17:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:01 smithi146 ceph-mon[103278]: pgmap v10710: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:01.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:01 smithi078 ceph-mon[91136]: pgmap v10710: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:01 smithi078 ceph-mon[99959]: pgmap v10710: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:01.654Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:01.654Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:01.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:01.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:03 smithi146 ceph-mon[103278]: pgmap v10711: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:03.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:03 smithi078 ceph-mon[91136]: pgmap v10711: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:03 smithi078 ceph-mon[99959]: pgmap v10711: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:05.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:05 smithi146 ceph-mon[103278]: pgmap v10712: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:05 smithi078 ceph-mon[91136]: pgmap v10712: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:05 smithi078 ceph-mon[99959]: pgmap v10712: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:07.716 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:07 smithi146 ceph-mon[103278]: pgmap v10713: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:07 smithi078 ceph-mon[91136]: pgmap v10713: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:07 smithi078 ceph-mon[99959]: pgmap v10713: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:09 smithi078 ceph-mon[91136]: pgmap v10714: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:09 smithi078 ceph-mon[99959]: pgmap v10714: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:09 smithi146 ceph-mon[103278]: pgmap v10714: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:10.834 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:17:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:17:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:17:11.358 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:17:11.358 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 6h - - 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 7s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 2s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 7s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 7s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 7s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 2s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 7s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 7s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 2s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 7s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 7s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 7s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:17:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 7s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 2s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 2s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 2s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 2s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 2s ago 6h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 7s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 7s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:17:11.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:17:11.655 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-mon[99959]: pgmap v10715: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:11.655 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-mon[99959]: from='client.88262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:11.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-mon[91136]: pgmap v10715: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:11.655 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-mon[91136]: from='client.88262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:17:11.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:17:11.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:11.670 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:17:11.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:17:11.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:17:11.670 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:17:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:17:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:17:11.940 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:17:11.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:11.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:11.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:11.655Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:11.940 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:11.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:11 smithi146 ceph-mon[103278]: pgmap v10715: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:11 smithi146 ceph-mon[103278]: from='client.88262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:12.248 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:17:12.249 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:12.249 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:12.249 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:12.249 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:12 smithi146 ceph-mon[103278]: from='client.88268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:12 smithi146 ceph-mon[103278]: from='client.68355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/504909999' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:12 smithi146 ceph-mon[103278]: from='client.68367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4276301863' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[91136]: from='client.88268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[91136]: from='client.68355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/504909999' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[91136]: from='client.68367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4276301863' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[99959]: from='client.88268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[99959]: from='client.68355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/504909999' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[99959]: from='client.68367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4276301863' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:13 smithi146 ceph-mon[103278]: pgmap v10716: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:13 smithi078 ceph-mon[91136]: pgmap v10716: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:13 smithi078 ceph-mon[99959]: pgmap v10716: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:15 smithi146 ceph-mon[103278]: pgmap v10717: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:15 smithi078 ceph-mon[91136]: pgmap v10717: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:15 smithi078 ceph-mon[99959]: pgmap v10717: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:17 smithi146 ceph-mon[103278]: pgmap v10718: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:17 smithi078 ceph-mon[99959]: pgmap v10718: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:17 smithi078 ceph-mon[91136]: pgmap v10718: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:19.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:19 smithi146 ceph-mon[103278]: pgmap v10719: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:19 smithi078 ceph-mon[99959]: pgmap v10719: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:19 smithi078 ceph-mon[91136]: pgmap v10719: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:17:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:17:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:21 smithi078 ceph-mon[99959]: pgmap v10720: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:21.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:21 smithi078 ceph-mon[91136]: pgmap v10720: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:21.655Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:21.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:21.656Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:21 smithi146 ceph-mon[103278]: pgmap v10720: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:23 smithi146 ceph-mon[103278]: pgmap v10721: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:24.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:23 smithi078 ceph-mon[91136]: pgmap v10721: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:23 smithi078 ceph-mon[99959]: pgmap v10721: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:25 smithi146 ceph-mon[103278]: pgmap v10722: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:25 smithi078 ceph-mon[91136]: pgmap v10722: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:25 smithi078 ceph-mon[99959]: pgmap v10722: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:27 smithi078 ceph-mon[91136]: pgmap v10723: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:27 smithi078 ceph-mon[99959]: pgmap v10723: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:27 smithi146 ceph-mon[103278]: pgmap v10723: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:28.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:28.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:29 smithi146 ceph-mon[103278]: pgmap v10724: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:29 smithi078 ceph-mon[91136]: pgmap v10724: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:29 smithi078 ceph-mon[99959]: pgmap v10724: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:17:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:17:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:31 smithi146 ceph-mon[103278]: pgmap v10725: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:31 smithi078 ceph-mon[91136]: pgmap v10725: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:31 smithi078 ceph-mon[99959]: pgmap v10725: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:31.656Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:31.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:31.657Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:33 smithi146 ceph-mon[103278]: pgmap v10726: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:33 smithi078 ceph-mon[91136]: pgmap v10726: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:33 smithi078 ceph-mon[99959]: pgmap v10726: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:35 smithi146 ceph-mon[103278]: pgmap v10727: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:36.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:35 smithi078 ceph-mon[99959]: pgmap v10727: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:35 smithi078 ceph-mon[91136]: pgmap v10727: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:37 smithi146 ceph-mon[103278]: pgmap v10728: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:38.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:37 smithi078 ceph-mon[91136]: pgmap v10728: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:37 smithi078 ceph-mon[99959]: pgmap v10728: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:39.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:39 smithi146 ceph-mon[103278]: pgmap v10729: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:40.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:39 smithi078 ceph-mon[99959]: pgmap v10729: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:40.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:39 smithi078 ceph-mon[91136]: pgmap v10729: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:17:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:17:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:41 smithi146 ceph-mon[103278]: pgmap v10730: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:41 smithi078 ceph-mon[99959]: pgmap v10730: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:41 smithi078 ceph-mon[91136]: pgmap v10730: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:41.657Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:41.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:41.658Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:42.543 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 6h - - 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 14s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 14s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 19s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 14s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:17:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 14s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 14s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 14s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 14s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 14s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 14s ago 6h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 19s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:17:43.069 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 19s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:17:43.070 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:17:43.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:17:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:17:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:17:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:17:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:17:43.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:17:43.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:17:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:17:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:17:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:17:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:17:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:17:43.651 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:17:43.993 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:17:43.993 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:17:43.993 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:17:43.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[99959]: pgmap v10731: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:43.993 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[99959]: from='client.68379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[99959]: from='client.68385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/203039465' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[91136]: pgmap v10731: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[91136]: from='client.68379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[91136]: from='client.68385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/203039465' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:43.994 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:43 smithi146 ceph-mon[103278]: pgmap v10731: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:43 smithi146 ceph-mon[103278]: from='client.68379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:43 smithi146 ceph-mon[103278]: from='client.68385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/203039465' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:17:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:45.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[91136]: from='client.88310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[91136]: from='client.68403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1160165214' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[99959]: from='client.88310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[99959]: from='client.68403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1160165214' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:44 smithi146 ceph-mon[103278]: from='client.88310 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:44 smithi146 ceph-mon[103278]: from='client.68403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:17:45.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1160165214' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:17:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:45 smithi146 ceph-mon[103278]: pgmap v10732: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:45 smithi078 ceph-mon[91136]: pgmap v10732: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:45 smithi078 ceph-mon[99959]: pgmap v10732: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:47 smithi078 ceph-mon[91136]: pgmap v10733: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:47 smithi078 ceph-mon[99959]: pgmap v10733: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:48.098 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:47 smithi146 ceph-mon[103278]: pgmap v10733: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:49.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:49.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:17:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:49 smithi146 ceph-mon[103278]: pgmap v10734: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:49 smithi078 ceph-mon[91136]: pgmap v10734: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:49 smithi078 ceph-mon[99959]: pgmap v10734: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:50.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:17:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:17:52.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:51 smithi078 ceph-mon[91136]: pgmap v10735: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:51 smithi078 ceph-mon[99959]: pgmap v10735: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:51.658Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:51.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:17:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:17:51.659Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:17:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:51 smithi146 ceph-mon[103278]: pgmap v10735: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:53 smithi146 ceph-mon[103278]: pgmap v10736: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:53 smithi078 ceph-mon[91136]: pgmap v10736: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:53 smithi078 ceph-mon[99959]: pgmap v10736: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:55 smithi146 ceph-mon[103278]: pgmap v10737: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:56.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:55 smithi078 ceph-mon[91136]: pgmap v10737: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:55 smithi078 ceph-mon[99959]: pgmap v10737: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:17:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:57 smithi078 ceph-mon[99959]: pgmap v10738: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:57 smithi078 ceph-mon[91136]: pgmap v10738: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:57 smithi146 ceph-mon[103278]: pgmap v10738: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:17:59.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:59.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:17:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:17:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:17:59 smithi146 ceph-mon[103278]: pgmap v10739: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:17:59 smithi078 ceph-mon[91136]: pgmap v10739: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:17:59 smithi078 ceph-mon[99959]: pgmap v10739: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:18:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:18:02.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:01 smithi078 ceph-mon[91136]: pgmap v10740: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:18:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:01 smithi078 ceph-mon[99959]: pgmap v10740: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:18:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:01.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:01.660Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:01.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:01 smithi146 ceph-mon[103278]: pgmap v10740: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:18:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:03 smithi146 ceph-mon[103278]: pgmap v10741: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:18:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:03 smithi078 ceph-mon[91136]: pgmap v10741: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:18:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:03 smithi078 ceph-mon[99959]: pgmap v10741: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:18:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:05.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:05 smithi146 ceph-mon[103278]: pgmap v10742: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:06.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:05 smithi078 ceph-mon[91136]: pgmap v10742: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:05 smithi078 ceph-mon[99959]: pgmap v10742: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:07 smithi146 ceph-mon[103278]: pgmap v10743: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:07 smithi078 ceph-mon[91136]: pgmap v10743: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:07 smithi078 ceph-mon[99959]: pgmap v10743: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:09.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:09.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.251 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.252 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:09.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:10.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:09 smithi078 ceph-mon[99959]: pgmap v10744: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:10.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:09 smithi078 ceph-mon[91136]: pgmap v10744: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:09 smithi146 ceph-mon[103278]: pgmap v10744: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:18:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:18:11.713 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:11.660Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:11.714 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:11.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:11.714 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:11.661Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:11 smithi146 ceph-mon[103278]: pgmap v10745: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:11 smithi078 ceph-mon[91136]: pgmap v10745: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:11 smithi078 ceph-mon[99959]: pgmap v10745: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:13 smithi146 ceph-mon[103278]: pgmap v10746: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:14.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:13 smithi078 ceph-mon[99959]: pgmap v10746: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:14.278 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:14.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:13 smithi078 ceph-mon[91136]: pgmap v10746: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:14.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:14.278 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 6h - - 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 6h - - 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 10s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 10s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 6s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 10s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 10s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 6s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 10s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 10s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:18:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 10s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 10s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 10s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 10s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 6s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 6s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 6s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 6s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 6s ago 6h 75.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 10s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 10s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:18:14.806 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:18:15.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:18:15.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:18:15.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:18:15.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:15.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:18:15.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:18:15.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:18:15.386 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:18:15.387 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:18:15.697 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:18:15.697 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:15.697 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:15 smithi146 ceph-mon[103278]: pgmap v10747: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:15 smithi146 ceph-mon[103278]: from='client.88334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:15 smithi146 ceph-mon[103278]: from='client.68421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:15 smithi146 ceph-mon[103278]: from='client.68427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2157754455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3860063704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[91136]: pgmap v10747: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[91136]: from='client.88334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[91136]: from='client.68421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[91136]: from='client.68427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2157754455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3860063704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[99959]: pgmap v10747: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[99959]: from='client.88334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[99959]: from='client.68421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[99959]: from='client.68427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2157754455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3860063704' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:17.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:17 smithi146 ceph-mon[103278]: from='client.88358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:17 smithi078 ceph-mon[99959]: from='client.88358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:17 smithi078 ceph-mon[91136]: from='client.88358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:18 smithi146 ceph-mon[103278]: pgmap v10748: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:18 smithi078 ceph-mon[91136]: pgmap v10748: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:18 smithi078 ceph-mon[99959]: pgmap v10748: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:19.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:20 smithi146 ceph-mon[103278]: pgmap v10749: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:20.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:20 smithi078 ceph-mon[99959]: pgmap v10749: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:20 smithi078 ceph-mon[91136]: pgmap v10749: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:18:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:18:22.014 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:21.661Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:22.014 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:21.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:22.014 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:21.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:22 smithi146 ceph-mon[103278]: pgmap v10750: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:22 smithi078 ceph-mon[99959]: pgmap v10750: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:22 smithi078 ceph-mon[91136]: pgmap v10750: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:24 smithi078 ceph-mon[99959]: pgmap v10751: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:24 smithi078 ceph-mon[91136]: pgmap v10751: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:24 smithi146 ceph-mon[103278]: pgmap v10751: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:25 smithi146 ceph-mon[103278]: pgmap v10752: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[91136]: pgmap v10752: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:25 smithi078 ceph-mon[99959]: pgmap v10752: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:27 smithi146 ceph-mon[103278]: pgmap v10753: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:27.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:27 smithi078 ceph-mon[91136]: pgmap v10753: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:27 smithi078 ceph-mon[99959]: pgmap v10753: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:28.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:29 smithi146 ceph-mon[103278]: pgmap v10754: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[99959]: pgmap v10754: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:29.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[91136]: pgmap v10754: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:30.913 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:18:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:18:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:31 smithi146 ceph-mon[103278]: pgmap v10755: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:31.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:31 smithi078 ceph-mon[99959]: pgmap v10755: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:31 smithi078 ceph-mon[91136]: pgmap v10755: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:31.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:31.662Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:31.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:33 smithi146 ceph-mon[103278]: pgmap v10756: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:33 smithi078 ceph-mon[91136]: pgmap v10756: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:33 smithi078 ceph-mon[99959]: pgmap v10756: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:35.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:35 smithi146 ceph-mon[103278]: pgmap v10757: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:35.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:35 smithi078 ceph-mon[99959]: pgmap v10757: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:35 smithi078 ceph-mon[91136]: pgmap v10757: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:37.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:37 smithi078 ceph-mon[91136]: pgmap v10758: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:37 smithi078 ceph-mon[99959]: pgmap v10758: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:37 smithi146 ceph-mon[103278]: pgmap v10758: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:38.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:39 smithi078 ceph-mon[91136]: pgmap v10759: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:39 smithi078 ceph-mon[99959]: pgmap v10759: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:39 smithi146 ceph-mon[103278]: pgmap v10759: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:18:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:18:41.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:41 smithi078 ceph-mon[91136]: pgmap v10760: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:41 smithi078 ceph-mon[99959]: pgmap v10760: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:41.598 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:41 smithi146 ceph-mon[103278]: pgmap v10760: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:41.662Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:41.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:41.663Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:43 smithi078 ceph-mon[91136]: pgmap v10761: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:43 smithi078 ceph-mon[99959]: pgmap v10761: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:43 smithi146 ceph-mon[103278]: pgmap v10761: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:45 smithi146 ceph-mon[103278]: pgmap v10762: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:45 smithi078 ceph-mon[91136]: pgmap v10762: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:45 smithi078 ceph-mon[99959]: pgmap v10762: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:45.982 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 6h - - 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 17s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:18:46.513 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 17s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 2s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 17s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 17s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 2s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 17s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 17s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 17s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 17s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 17s ago 6h 75.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:18:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 2s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:18:46.515 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:18:46.515 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 2s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:18:46.515 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:18:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:46 smithi146 ceph-mon[103278]: from='client.68451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:46.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:46 smithi078 ceph-mon[99959]: from='client.68451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:46 smithi078 ceph-mon[91136]: from='client.68451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:18:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:18:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:18:47.093 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:18:47.093 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:18:47.407 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:18:47.407 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:18:47.407 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[99959]: pgmap v10763: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[99959]: from='client.68457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[99959]: from='client.88382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1487593067' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[91136]: pgmap v10763: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[91136]: from='client.68457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[91136]: from='client.88382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:47.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1487593067' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:47 smithi146 ceph-mon[103278]: pgmap v10763: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:47 smithi146 ceph-mon[103278]: from='client.68457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:47 smithi146 ceph-mon[103278]: from='client.88382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1487593067' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:18:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:48 smithi146 ceph-mon[103278]: from='client.68475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2238956707' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:48 smithi078 ceph-mon[99959]: from='client.68475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:48.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2238956707' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:48 smithi078 ceph-mon[91136]: from='client.68475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:18:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2238956707' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:18:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:49.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:49 smithi146 ceph-mon[103278]: pgmap v10764: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:49.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.436 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.436 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[99959]: pgmap v10764: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[91136]: pgmap v10764: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:18:50.765 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:18:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:18:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:51 smithi146 ceph-mon[103278]: pgmap v10765: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:51.612 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:51 smithi078 ceph-mon[99959]: pgmap v10765: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:51.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:51 smithi078 ceph-mon[91136]: pgmap v10765: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:51.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:51.663Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:51.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:51.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:18:51.912 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:18:51.664Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:18:53.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:53 smithi078 ceph-mon[91136]: pgmap v10766: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:53.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:53 smithi078 ceph-mon[99959]: pgmap v10766: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:53 smithi146 ceph-mon[103278]: pgmap v10766: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:55 smithi146 ceph-mon[103278]: pgmap v10767: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:55 smithi078 ceph-mon[91136]: pgmap v10767: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:55 smithi078 ceph-mon[99959]: pgmap v10767: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:57.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:57 smithi078 ceph-mon[91136]: pgmap v10768: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:57.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:57 smithi078 ceph-mon[99959]: pgmap v10768: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:57 smithi146 ceph-mon[103278]: pgmap v10768: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:18:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:58.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:18:59.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:59 smithi146 ceph-mon[103278]: pgmap v10769: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:59.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:18:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:59 smithi078 ceph-mon[91136]: pgmap v10769: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:18:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:18:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:59 smithi078 ceph-mon[99959]: pgmap v10769: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:18:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:18:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:19:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:19:01.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:01 smithi078 ceph-mon[91136]: pgmap v10770: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:01.665 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:01 smithi078 ceph-mon[99959]: pgmap v10770: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:01 smithi146 ceph-mon[103278]: pgmap v10770: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:01.664Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:01.665Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:01.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:03 smithi078 ceph-mon[91136]: pgmap v10771: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:03 smithi078 ceph-mon[99959]: pgmap v10771: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:03 smithi146 ceph-mon[103278]: pgmap v10771: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[99959]: pgmap v10772: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[91136]: pgmap v10772: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:05 smithi146 ceph-mon[103278]: pgmap v10772: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:07.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:07 smithi078 ceph-mon[91136]: pgmap v10773: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:07.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:07 smithi078 ceph-mon[99959]: pgmap v10773: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:07 smithi146 ceph-mon[103278]: pgmap v10773: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:09.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:09 smithi146 ceph-mon[103278]: pgmap v10774: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:09.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.497 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[91136]: pgmap v10774: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[99959]: pgmap v10774: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:19:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:19:11.666 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:11 smithi078 ceph-mon[99959]: pgmap v10775: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:11.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:11 smithi078 ceph-mon[91136]: pgmap v10775: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:11 smithi146 ceph-mon[103278]: pgmap v10775: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:11.665Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:11.666Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:11.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:13.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:13 smithi078 ceph-mon[99959]: pgmap v10776: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:13 smithi078 ceph-mon[91136]: pgmap v10776: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:13 smithi146 ceph-mon[103278]: pgmap v10776: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:15 smithi146 ceph-mon[103278]: pgmap v10777: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:15 smithi078 ceph-mon[91136]: pgmap v10777: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:15 smithi078 ceph-mon[99959]: pgmap v10777: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:17.702 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:17 smithi078 ceph-mon[99959]: pgmap v10778: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:17.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:17 smithi078 ceph-mon[91136]: pgmap v10778: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:17.702 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:19:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:17 smithi146 ceph-mon[103278]: pgmap v10778: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:18.226 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 6h - - 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 6h - - 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 13s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 9s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 13s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 9s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 13s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 13s ago 6h 541M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 9s ago 6h 370M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 13s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 13s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 9s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 13s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:19:18.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 13s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 13s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 13s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 9s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 9s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 9s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 9s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 9s ago 6h 75.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 13s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 13s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:19:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:19:18.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:18 smithi078 ceph-mon[99959]: from='client.88406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:18 smithi078 ceph-mon[99959]: from='client.68493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:18 smithi078 ceph-mon[91136]: from='client.88406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:18 smithi078 ceph-mon[91136]: from='client.68493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:18.541 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:19:18.542 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:19:18.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:19:18.542 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:19:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:18 smithi146 ceph-mon[103278]: from='client.88406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:18 smithi146 ceph-mon[103278]: from='client.68493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:18.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:19:18.808 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:19:18.808 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:19:18.809 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:19:19.120 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:19:19.121 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:19.121 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:19.121 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:19.121 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:19 smithi146 ceph-mon[103278]: pgmap v10779: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:19 smithi146 ceph-mon[103278]: from='client.68499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2633217220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:19 smithi146 ceph-mon[103278]: from='client.88430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2009302918' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[91136]: pgmap v10779: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[91136]: from='client.68499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2633217220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[91136]: from='client.88430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2009302918' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[99959]: pgmap v10779: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[99959]: from='client.68499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2633217220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[99959]: from='client.88430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2009302918' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:19:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:19:21.667 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:21 smithi078 ceph-mon[99959]: pgmap v10780: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:21.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:21 smithi078 ceph-mon[91136]: pgmap v10780: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:21.702 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:21 smithi146 ceph-mon[103278]: pgmap v10780: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:21.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:21.667Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:21.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:21.667Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:21.919 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:21.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:23 smithi146 ceph-mon[103278]: pgmap v10781: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:23.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:23 smithi078 ceph-mon[99959]: pgmap v10781: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:23 smithi078 ceph-mon[91136]: pgmap v10781: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:25 smithi146 ceph-mon[103278]: pgmap v10782: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[91136]: pgmap v10782: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[99959]: pgmap v10782: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:27.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:27 smithi078 ceph-mon[91136]: pgmap v10783: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:27.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:27 smithi078 ceph-mon[99959]: pgmap v10783: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:27 smithi146 ceph-mon[103278]: pgmap v10783: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:28.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:29.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:29 smithi146 ceph-mon[103278]: pgmap v10784: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:29.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:29.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[91136]: pgmap v10784: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[99959]: pgmap v10784: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:19:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:19:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:31 smithi146 ceph-mon[103278]: pgmap v10785: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:31.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:31 smithi078 ceph-mon[99959]: pgmap v10785: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:31 smithi078 ceph-mon[91136]: pgmap v10785: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:31.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:31.668Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:31.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:33 smithi146 ceph-mon[103278]: pgmap v10786: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:33.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:33 smithi078 ceph-mon[99959]: pgmap v10786: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:33 smithi078 ceph-mon[91136]: pgmap v10786: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:35 smithi146 ceph-mon[103278]: pgmap v10787: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:35 smithi078 ceph-mon[91136]: pgmap v10787: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:35 smithi078 ceph-mon[99959]: pgmap v10787: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:37 smithi146 ceph-mon[103278]: pgmap v10788: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:37 smithi078 ceph-mon[91136]: pgmap v10788: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:37 smithi078 ceph-mon[99959]: pgmap v10788: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:38.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:39 smithi078 ceph-mon[91136]: pgmap v10789: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:39 smithi078 ceph-mon[99959]: pgmap v10789: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:39 smithi146 ceph-mon[103278]: pgmap v10789: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:19:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:19:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:41 smithi078 ceph-mon[91136]: pgmap v10790: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:41 smithi078 ceph-mon[99959]: pgmap v10790: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:41.668Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:41.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:41.669Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:41 smithi146 ceph-mon[103278]: pgmap v10790: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:43 smithi078 ceph-mon[91136]: pgmap v10791: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:43 smithi078 ceph-mon[99959]: pgmap v10791: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:43 smithi146 ceph-mon[103278]: pgmap v10791: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:45 smithi146 ceph-mon[103278]: pgmap v10792: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[91136]: pgmap v10792: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[99959]: pgmap v10792: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:47 smithi146 ceph-mon[103278]: pgmap v10793: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:47 smithi078 ceph-mon[91136]: pgmap v10793: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:47 smithi078 ceph-mon[99959]: pgmap v10793: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:49.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:49.414 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:19:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:49 smithi146 ceph-mon[103278]: pgmap v10794: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[99959]: pgmap v10794: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[91136]: pgmap v10794: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:19:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 6h - - 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 6h - - 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 5s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 0s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 5s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 0s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 5s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 5s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 0s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 5s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 5s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:19:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 0s ago 6h 16.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 5s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 5s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 5s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 5s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 0s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 0s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 0s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 0s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 0s ago 6h 75.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 5s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 5s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:19:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:19:50.260 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:19:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:19:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:19:50.262 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:19:50.533 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:19:50.534 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:19:50.534 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:19:50.534 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:19:50.534 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:19:50.534 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:19:50.534 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:19:50.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:19:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[91136]: from='client.68523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[91136]: from='client.68529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[91136]: from='client.88454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2864580991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[99959]: from='client.68523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[99959]: from='client.68529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[99959]: from='client.88454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2864580991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:50.843 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:19:50.844 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:50.844 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:19:50.844 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:50.844 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:19:50.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:50 smithi146 ceph-mon[103278]: from='client.68523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:50 smithi146 ceph-mon[103278]: from='client.68529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:50 smithi146 ceph-mon[103278]: from='client.88454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:50.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2864580991' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:19:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:51 smithi146 ceph-mon[103278]: pgmap v10795: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:51 smithi146 ceph-mon[103278]: from='client.88466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1495243961' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-mon[91136]: pgmap v10795: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-mon[91136]: from='client.88466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1495243961' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-mon[99959]: pgmap v10795: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-mon[99959]: from='client.88466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:19:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1495243961' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:19:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:51.669Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:51.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:19:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:19:51.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:19:53.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:53 smithi078 ceph-mon[99959]: pgmap v10796: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:53.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:53 smithi078 ceph-mon[91136]: pgmap v10796: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:53 smithi146 ceph-mon[103278]: pgmap v10796: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:55 smithi146 ceph-mon[103278]: pgmap v10797: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:55 smithi078 ceph-mon[99959]: pgmap v10797: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:55 smithi078 ceph-mon[91136]: pgmap v10797: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:19:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:57 smithi146 ceph-mon[103278]: pgmap v10798: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:57 smithi078 ceph-mon[91136]: pgmap v10798: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:57 smithi078 ceph-mon[99959]: pgmap v10798: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:19:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:19:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:19:59.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:19:59 smithi146 ceph-mon[103278]: pgmap v10799: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:00.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:19:59 smithi078 ceph-mon[91136]: pgmap v10799: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:00.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:19:59 smithi078 ceph-mon[99959]: pgmap v10799: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T02:19:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:20:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:20:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:01.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:01 smithi078 ceph-mon[99959]: pgmap v10800: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:01.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:01 smithi078 ceph-mon[91136]: pgmap v10800: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:01.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:01.669Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:01.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:01.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:01.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:01.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:01 smithi146 ceph-mon[103278]: pgmap v10800: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:03 smithi146 ceph-mon[103278]: pgmap v10801: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:03 smithi078 ceph-mon[91136]: pgmap v10801: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:03 smithi078 ceph-mon[99959]: pgmap v10801: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:05 smithi146 ceph-mon[103278]: pgmap v10802: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[91136]: pgmap v10802: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[99959]: pgmap v10802: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:07 smithi146 ceph-mon[103278]: pgmap v10803: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:07 smithi078 ceph-mon[91136]: pgmap v10803: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:07 smithi078 ceph-mon[99959]: pgmap v10803: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:09.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:09 smithi146 ceph-mon[103278]: pgmap v10804: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[91136]: pgmap v10804: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[99959]: pgmap v10804: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:20:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:20:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:11 smithi146 ceph-mon[103278]: pgmap v10805: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:12.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:11 smithi078 ceph-mon[91136]: pgmap v10805: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:11 smithi078 ceph-mon[99959]: pgmap v10805: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:11.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:11.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:11.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:13 smithi078 ceph-mon[91136]: pgmap v10806: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:13 smithi078 ceph-mon[99959]: pgmap v10806: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:13 smithi146 ceph-mon[103278]: pgmap v10806: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:15 smithi078 ceph-mon[91136]: pgmap v10807: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:15 smithi078 ceph-mon[99959]: pgmap v10807: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:15 smithi146 ceph-mon[103278]: pgmap v10807: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:17 smithi078 ceph-mon[91136]: pgmap v10808: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:17 smithi078 ceph-mon[99959]: pgmap v10808: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:17 smithi146 ceph-mon[103278]: pgmap v10808: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:19.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:19 smithi146 ceph-mon[103278]: pgmap v10809: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:19 smithi078 ceph-mon[91136]: pgmap v10809: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:19 smithi078 ceph-mon[99959]: pgmap v10809: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:20:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:20:21.137 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 6h - - 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 6h - - 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 16s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 12s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 16s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 12s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:20:21.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 16s ago 6h 583M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 16s ago 6h 542M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 16s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 16s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 12s ago 6h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 16s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 16s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 16s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 16s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 12s ago 6h 75.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:20:21.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 16s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:20:21.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:20:21.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 16s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:20:21.664 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:20:21.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:21 smithi078 ceph-mon[91136]: pgmap v10810: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:21.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:21 smithi078 ceph-mon[99959]: pgmap v10810: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:21.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:21.670Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:21.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:21.670Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:21.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:21.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:20:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:20:21.973 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:20:22.239 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:20:22.240 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:20:22.240 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:20:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:21 smithi146 ceph-mon[103278]: pgmap v10810: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:22.548 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:22.548 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:22.548 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:23.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[91136]: from='client.68559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[91136]: from='client.68565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[91136]: from='client.68571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2403665213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2846301251' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[99959]: from='client.68559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[99959]: from='client.68565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[99959]: from='client.68571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2403665213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2846301251' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:22 smithi146 ceph-mon[103278]: from='client.68559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:22 smithi146 ceph-mon[103278]: from='client.68565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:22 smithi146 ceph-mon[103278]: from='client.68571 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:23.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2403665213' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2846301251' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:23 smithi146 ceph-mon[103278]: pgmap v10811: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:23 smithi146 ceph-mon[103278]: from='client.88502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:23 smithi078 ceph-mon[91136]: pgmap v10811: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:24.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:23 smithi078 ceph-mon[91136]: from='client.88502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:23 smithi078 ceph-mon[99959]: pgmap v10811: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:23 smithi078 ceph-mon[99959]: from='client.88502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[99959]: pgmap v10812: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[91136]: pgmap v10812: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:25 smithi146 ceph-mon[103278]: pgmap v10812: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:27 smithi146 ceph-mon[103278]: pgmap v10813: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:28.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:27 smithi078 ceph-mon[91136]: pgmap v10813: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:27 smithi078 ceph-mon[99959]: pgmap v10813: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:29.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:29.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:29.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:29 smithi146 ceph-mon[103278]: pgmap v10814: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[91136]: pgmap v10814: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[99959]: pgmap v10814: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:20:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:20:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:31 smithi078 ceph-mon[91136]: pgmap v10815: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:31 smithi078 ceph-mon[99959]: pgmap v10815: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:31.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:31.671Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:31.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:32.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:31 smithi146 ceph-mon[103278]: pgmap v10815: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:33 smithi146 ceph-mon[103278]: pgmap v10816: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:34.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:33 smithi078 ceph-mon[99959]: pgmap v10816: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:34.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:33 smithi078 ceph-mon[91136]: pgmap v10816: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:35 smithi146 ceph-mon[103278]: pgmap v10817: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:35 smithi078 ceph-mon[99959]: pgmap v10817: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:35 smithi078 ceph-mon[91136]: pgmap v10817: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:37 smithi146 ceph-mon[103278]: pgmap v10818: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:37 smithi078 ceph-mon[91136]: pgmap v10818: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:37 smithi078 ceph-mon[99959]: pgmap v10818: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:39 smithi146 ceph-mon[103278]: pgmap v10819: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:39 smithi078 ceph-mon[91136]: pgmap v10819: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:39 smithi078 ceph-mon[99959]: pgmap v10819: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:20:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:20:41.929 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:41.671Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:41.929 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:41.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:41.929 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:41.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:41 smithi146 ceph-mon[103278]: pgmap v10820: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:42.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:41 smithi078 ceph-mon[91136]: pgmap v10820: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:41 smithi078 ceph-mon[99959]: pgmap v10820: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:43 smithi146 ceph-mon[103278]: pgmap v10821: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:43 smithi078 ceph-mon[91136]: pgmap v10821: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:43 smithi078 ceph-mon[99959]: pgmap v10821: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[99959]: pgmap v10822: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[91136]: pgmap v10822: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:46 smithi146 ceph-mon[103278]: pgmap v10822: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:48.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:48 smithi078 ceph-mon[91136]: pgmap v10823: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:48 smithi078 ceph-mon[99959]: pgmap v10823: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:48 smithi146 ceph-mon[103278]: pgmap v10823: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:49.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:50 smithi146 ceph-mon[103278]: pgmap v10824: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[91136]: pgmap v10824: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[99959]: pgmap v10824: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:20:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:20:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:20:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:51.672Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:51.672Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:20:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:20:51.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:20:52.461 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:52 smithi146 ceph-mon[103278]: pgmap v10825: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:52.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:52 smithi078 ceph-mon[99959]: pgmap v10825: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:52.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:52 smithi078 ceph-mon[91136]: pgmap v10825: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:52.840 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 6h - - 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 6h - - 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 8s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 3s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 8s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 3s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (5h) 8s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 8s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 3s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 8s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:20:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 8s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 3s ago 6h 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 8s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 8s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 8s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 8s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 3s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 3s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 3s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 3s ago 6h 90.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 3s ago 6h 75.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 8s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 8s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:20:53.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:20:53.710 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:20:53.710 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:20:53.711 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:20:53.980 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:20:53.980 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:20:53.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:20:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:54 smithi078 ceph-mon[99959]: pgmap v10826: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:54 smithi078 ceph-mon[99959]: from='client.68595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/979880472' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:54 smithi078 ceph-mon[91136]: pgmap v10826: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:54 smithi078 ceph-mon[91136]: from='client.68595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/979880472' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:54.291 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:20:54.291 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:20:54.291 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:54 smithi146 ceph-mon[103278]: pgmap v10826: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:54 smithi146 ceph-mon[103278]: from='client.68595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/979880472' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:20:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:55 smithi146 ceph-mon[103278]: from='client.68601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:55 smithi146 ceph-mon[103278]: from='client.68607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:55 smithi146 ceph-mon[103278]: from='client.88538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1129698671' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:55.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[99959]: from='client.68601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[99959]: from='client.68607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[99959]: from='client.88538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1129698671' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[91136]: from='client.68601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[91136]: from='client.68607 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[91136]: from='client.88538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:20:55.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1129698671' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:20:56.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:56 smithi078 ceph-mon[99959]: pgmap v10827: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:56.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:56 smithi078 ceph-mon[91136]: pgmap v10827: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:56 smithi146 ceph-mon[103278]: pgmap v10827: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:20:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:57 smithi146 ceph-mon[103278]: pgmap v10828: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:20:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:57 smithi078 ceph-mon[91136]: pgmap v10828: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:20:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:57 smithi078 ceph-mon[99959]: pgmap v10828: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:20:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:20:59.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:59 smithi146 ceph-mon[103278]: pgmap v10829: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:20:59.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:59 smithi078 ceph-mon[91136]: pgmap v10829: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:20:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:20:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:59 smithi078 ceph-mon[99959]: pgmap v10829: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:20:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:21:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:21:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:01 smithi078 ceph-mon[91136]: pgmap v10830: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:21:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:01 smithi078 ceph-mon[99959]: pgmap v10830: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:21:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:01 smithi146 ceph-mon[103278]: pgmap v10830: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T02:21:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:01.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:01.673Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:01.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:03.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:03 smithi078 ceph-mon[99959]: pgmap v10831: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:21:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:03 smithi078 ceph-mon[91136]: pgmap v10831: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:21:03.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:03 smithi146 ceph-mon[103278]: pgmap v10831: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:21:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[99959]: pgmap v10832: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:21:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[91136]: pgmap v10832: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:05 smithi146 ceph-mon[103278]: pgmap v10832: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:21:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:07 smithi078 ceph-mon[99959]: pgmap v10833: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:21:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:07 smithi078 ceph-mon[91136]: pgmap v10833: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:21:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:07 smithi146 ceph-mon[103278]: pgmap v10833: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:21:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:09 smithi146 ceph-mon[103278]: pgmap v10834: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:21:09.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:09 smithi078 ceph-mon[99959]: pgmap v10834: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:21:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:09 smithi078 ceph-mon[91136]: pgmap v10834: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:21:10.963 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:21:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:21:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:11.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:11.673Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:11.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:11.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:11.976 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:11.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:11 smithi146 ceph-mon[103278]: pgmap v10835: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T02:21:12.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:11 smithi078 ceph-mon[91136]: pgmap v10835: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T02:21:12.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:11 smithi078 ceph-mon[99959]: pgmap v10835: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-21T02:21:14.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:13 smithi146 ceph-mon[103278]: pgmap v10836: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:21:14.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:13 smithi078 ceph-mon[99959]: pgmap v10836: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:21:14.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:13 smithi078 ceph-mon[91136]: pgmap v10836: 161 pgs: 161 active+clean; 585 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:21:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:15 smithi146 ceph-mon[103278]: pgmap v10837: 161 pgs: 161 active+clean; 585 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:21:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:15 smithi078 ceph-mon[99959]: pgmap v10837: 161 pgs: 161 active+clean; 585 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:21:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:15 smithi078 ceph-mon[91136]: pgmap v10837: 161 pgs: 161 active+clean; 585 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:21:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:17 smithi146 ceph-mon[103278]: pgmap v10838: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T02:21:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:17 smithi078 ceph-mon[91136]: pgmap v10838: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T02:21:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:17 smithi078 ceph-mon[99959]: pgmap v10838: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 0 B/s wr, 155 op/s 2024-09-21T02:21:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:19.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:19 smithi146 ceph-mon[103278]: pgmap v10839: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T02:21:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:19 smithi078 ceph-mon[91136]: pgmap v10839: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T02:21:20.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:19 smithi078 ceph-mon[99959]: pgmap v10839: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T02:21:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:21:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:21:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:21.674Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:21.674Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:21.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:21 smithi146 ceph-mon[103278]: pgmap v10840: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:21:22.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:21 smithi078 ceph-mon[99959]: pgmap v10840: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:21:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:21 smithi078 ceph-mon[91136]: pgmap v10840: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:21:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:23 smithi146 ceph-mon[103278]: pgmap v10841: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T02:21:24.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:23 smithi078 ceph-mon[99959]: pgmap v10841: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T02:21:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:23 smithi078 ceph-mon[91136]: pgmap v10841: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T02:21:24.583 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:21:25.111 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:21:25.111 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T02:21:25.111 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 6h - - 2024-09-21T02:21:25.111 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 15s ago 6h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (5h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 15s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 19s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 15s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 19s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 15s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 19s ago 6h 92.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 19s ago 6h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 19s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 19s ago 6h 97.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:21:25.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 15s ago 6h 112M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 15s ago 6h 89.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 15s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 15s ago 6h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (5h) 15s ago 6h 75.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 19s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 19s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:21:25.113 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:21:25.456 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:21:25.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:21:25.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:21:25.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:25.456 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:21:25.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:21:25.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:21:25.726 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:21:25.727 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:21:25.727 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:21:25.727 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:21:26.034 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:21:26.035 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:26.035 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:26.035 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:26.035 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: pgmap v10842: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T02:21:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: from='client.88550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: from='client.68637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1007792537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:26.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: pgmap v10842: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: from='client.88550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: from='client.68637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1007792537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: pgmap v10842: 161 pgs: 161 active+clean; 585 KiB data, 378 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: from='client.88550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: from='client.68637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1007792537' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:27 smithi146 ceph-mon[103278]: from='client.68643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:27 smithi146 ceph-mon[103278]: from='client.68655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:27.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1594954923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:27 smithi078 ceph-mon[91136]: from='client.68643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:27 smithi078 ceph-mon[91136]: from='client.68655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1594954923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:27 smithi078 ceph-mon[99959]: from='client.68643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:27 smithi078 ceph-mon[99959]: from='client.68655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:27.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1594954923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:28 smithi146 ceph-mon[103278]: pgmap v10843: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:21:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:28 smithi078 ceph-mon[99959]: pgmap v10843: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:21:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:28 smithi078 ceph-mon[91136]: pgmap v10843: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:21:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:30 smithi146 ceph-mon[103278]: pgmap v10844: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:21:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:30 smithi078 ceph-mon[91136]: pgmap v10844: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:21:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:30 smithi078 ceph-mon[99959]: pgmap v10844: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:21:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:21:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:21:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:31 smithi146 ceph-mon[103278]: pgmap v10845: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[99959]: pgmap v10845: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-mon[91136]: pgmap v10845: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:21:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:31.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:31.675Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:31.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:33.591 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:33 smithi146 ceph-mon[103278]: pgmap v10846: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:21:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:33 smithi078 ceph-mon[91136]: pgmap v10846: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:21:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:33 smithi078 ceph-mon[99959]: pgmap v10846: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:21:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:35 smithi146 ceph-mon[103278]: pgmap v10847: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T02:21:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:35 smithi078 ceph-mon[91136]: pgmap v10847: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T02:21:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:35 smithi078 ceph-mon[99959]: pgmap v10847: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T02:21:37.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:37 smithi078 ceph-mon[99959]: pgmap v10848: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T02:21:37.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:37 smithi078 ceph-mon[91136]: pgmap v10848: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T02:21:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:37 smithi146 ceph-mon[103278]: pgmap v10848: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-21T02:21:38.328 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:38.328 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:39.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:39 smithi078 ceph-mon[91136]: pgmap v10849: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:39 smithi078 ceph-mon[99959]: pgmap v10849: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:39 smithi146 ceph-mon[103278]: pgmap v10849: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:21:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:21:41.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:41 smithi078 ceph-mon[99959]: pgmap v10850: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:41.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:41 smithi078 ceph-mon[91136]: pgmap v10850: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:41 smithi146 ceph-mon[103278]: pgmap v10850: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:41.675Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:41.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:41.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:43 smithi146 ceph-mon[103278]: pgmap v10851: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:43 smithi078 ceph-mon[91136]: pgmap v10851: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:43 smithi078 ceph-mon[99959]: pgmap v10851: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:45 smithi146 ceph-mon[103278]: pgmap v10852: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:45 smithi078 ceph-mon[99959]: pgmap v10852: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:45 smithi078 ceph-mon[91136]: pgmap v10852: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:46.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:47.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:47 smithi078 ceph-mon[99959]: pgmap v10853: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:47.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:47 smithi078 ceph-mon[91136]: pgmap v10853: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:47 smithi146 ceph-mon[103278]: pgmap v10853: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:48.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:48.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:48.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:49.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:49 smithi146 ceph-mon[103278]: pgmap v10854: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:49.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:49 smithi078 ceph-mon[99959]: pgmap v10854: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:49 smithi078 ceph-mon[91136]: pgmap v10854: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:21:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:21:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:51 smithi146 ceph-mon[103278]: pgmap v10855: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[99959]: pgmap v10855: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[91136]: pgmap v10855: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:21:51.987 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:51.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:51.987 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:51.676Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:21:51.987 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:21:51.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:21:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:53 smithi146 ceph-mon[103278]: pgmap v10856: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:53 smithi078 ceph-mon[91136]: pgmap v10856: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:53 smithi078 ceph-mon[99959]: pgmap v10856: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:55.404 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:55 smithi146 ceph-mon[103278]: pgmap v10857: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:55 smithi078 ceph-mon[91136]: pgmap v10857: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:55 smithi078 ceph-mon[99959]: pgmap v10857: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:56.327 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 6h - - 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 6h - - 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 11s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 11s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 6s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 11s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 11s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 6s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 11s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 11s ago 6h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:21:56.857 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 11s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 11s ago 6h 98.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 11s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 11s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 6s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 6s ago 6h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 6s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 6s ago 6h 91.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 6s ago 6h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 11s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 11s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:21:56.858 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:21:57.176 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:21:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:21:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:21:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:21:57.177 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:21:57.448 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:21:57.449 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:21:57.449 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:21:57.449 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:21:57.449 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:21:57.449 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:21:57.449 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:21:57.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[91136]: pgmap v10858: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:57.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[91136]: from='client.88586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[91136]: from='client.68673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[91136]: from='client.88598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/477190157' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[99959]: pgmap v10858: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[99959]: from='client.88586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[99959]: from='client.68673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[99959]: from='client.88598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.450 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/477190157' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:57.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:57 smithi146 ceph-mon[103278]: pgmap v10858: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:21:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:57 smithi146 ceph-mon[103278]: from='client.88586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:57 smithi146 ceph-mon[103278]: from='client.68673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:57 smithi146 ceph-mon[103278]: from='client.88598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/477190157' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:21:57.763 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:21:57.763 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:57.763 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:21:57.763 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:57.763 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:21:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:58 smithi146 ceph-mon[103278]: from='client.88610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1279921221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:58.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:58 smithi078 ceph-mon[91136]: from='client.88610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1279921221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:58 smithi078 ceph-mon[99959]: from='client.88610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:21:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1279921221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:21:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:21:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:59 smithi078 ceph-mon[99959]: pgmap v10859: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:21:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:59 smithi078 ceph-mon[91136]: pgmap v10859: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:21:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:21:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:59 smithi146 ceph-mon[103278]: pgmap v10859: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:21:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:21:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:22:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:22:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:01 smithi146 ceph-mon[103278]: pgmap v10860: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:01 smithi078 ceph-mon[99959]: pgmap v10860: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:01 smithi078 ceph-mon[91136]: pgmap v10860: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:02.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:01.676Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:01.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:01.677Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:03 smithi146 ceph-mon[103278]: pgmap v10861: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:03 smithi078 ceph-mon[91136]: pgmap v10861: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:03 smithi078 ceph-mon[99959]: pgmap v10861: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:05 smithi146 ceph-mon[103278]: pgmap v10862: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:05.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:05 smithi078 ceph-mon[99959]: pgmap v10862: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:05.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:05 smithi078 ceph-mon[91136]: pgmap v10862: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:07.890 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:07 smithi078 ceph-mon[99959]: pgmap v10863: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:07.891 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:07 smithi078 ceph-mon[91136]: pgmap v10863: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:07 smithi146 ceph-mon[103278]: pgmap v10863: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:09 smithi146 ceph-mon[103278]: pgmap v10864: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:10.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:09 smithi078 ceph-mon[91136]: pgmap v10864: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:09 smithi078 ceph-mon[99959]: pgmap v10864: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:22:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:22:11.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[99959]: pgmap v10865: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:11.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.678 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[91136]: pgmap v10865: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:11.678 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:11 smithi146 ceph-mon[103278]: pgmap v10865: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:11.677Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:11.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:11.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:13 smithi146 ceph-mon[103278]: pgmap v10866: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:13 smithi078 ceph-mon[91136]: pgmap v10866: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:13 smithi078 ceph-mon[99959]: pgmap v10866: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:14.735 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:15.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:15 smithi146 ceph-mon[103278]: pgmap v10867: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:15 smithi078 ceph-mon[99959]: pgmap v10867: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:15 smithi078 ceph-mon[91136]: pgmap v10867: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:17.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:17 smithi078 ceph-mon[91136]: pgmap v10868: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:17.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:17 smithi078 ceph-mon[99959]: pgmap v10868: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:17 smithi146 ceph-mon[103278]: pgmap v10868: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:18.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:19 smithi146 ceph-mon[103278]: pgmap v10869: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:19 smithi078 ceph-mon[91136]: pgmap v10869: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:19 smithi078 ceph-mon[99959]: pgmap v10869: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:22:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:22:21.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:21 smithi078 ceph-mon[99959]: pgmap v10870: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:21.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:21 smithi078 ceph-mon[91136]: pgmap v10870: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:21 smithi146 ceph-mon[103278]: pgmap v10870: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:21.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:21.678Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:21.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:23 smithi146 ceph-mon[103278]: pgmap v10871: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:23.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:23 smithi078 ceph-mon[91136]: pgmap v10871: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:23 smithi078 ceph-mon[99959]: pgmap v10871: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:25.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:25 smithi078 ceph-mon[91136]: pgmap v10872: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:25.692 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:25 smithi078 ceph-mon[99959]: pgmap v10872: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:25 smithi146 ceph-mon[103278]: pgmap v10872: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:27 smithi146 ceph-mon[103278]: pgmap v10873: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:27 smithi078 ceph-mon[99959]: pgmap v10873: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:27 smithi078 ceph-mon[91136]: pgmap v10873: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:28.064 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 6h - - 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 18s ago 6h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 18s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 2s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (5h) 2s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 18s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 2s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:22:28.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 18s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 2s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 2s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 2s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 2s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 6h 89.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 18s ago 6h 76.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 2s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 2s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:22:28.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:22:28.971 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:22:28.971 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:22:28.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:22:28.972 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:22:28.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:28.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:28 smithi078 ceph-mon[99959]: from='client.88622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:28.973 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:28.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:28.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:28 smithi078 ceph-mon[91136]: from='client.88622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:28.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:28 smithi146 ceph-mon[103278]: from='client.88622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:22:29.239 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:22:29.240 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:22:29.240 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:22:29.240 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:22:29.240 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:22:29.547 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:22:29.547 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:22:29.547 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:22:29.547 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:22:29.548 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:29 smithi146 ceph-mon[103278]: pgmap v10874: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:29 smithi146 ceph-mon[103278]: from='client.68709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:29 smithi146 ceph-mon[103278]: from='client.88634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2213011587' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2686113150' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:30.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[91136]: pgmap v10874: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[91136]: from='client.68709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[91136]: from='client.88634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2213011587' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2686113150' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[99959]: pgmap v10874: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[99959]: from='client.68709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[99959]: from='client.88634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2213011587' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:22:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2686113150' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:22:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:30 smithi146 ceph-mon[103278]: from='client.68727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:30.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:22:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:22:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[91136]: from='client.68727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[99959]: from='client.68727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:22:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:31 smithi146 ceph-mon[103278]: pgmap v10875: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:31 smithi078 ceph-mon[91136]: pgmap v10875: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:31 smithi078 ceph-mon[99959]: pgmap v10875: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:31.678Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:31.679Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:31.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:33 smithi146 ceph-mon[103278]: pgmap v10876: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:33 smithi078 ceph-mon[91136]: pgmap v10876: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:33 smithi078 ceph-mon[99959]: pgmap v10876: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:35 smithi146 ceph-mon[103278]: pgmap v10877: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:35 smithi078 ceph-mon[91136]: pgmap v10877: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:35 smithi078 ceph-mon[99959]: pgmap v10877: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:37 smithi146 ceph-mon[103278]: pgmap v10878: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:37 smithi078 ceph-mon[91136]: pgmap v10878: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:37 smithi078 ceph-mon[99959]: pgmap v10878: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:39.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:39 smithi078 ceph-mon[99959]: pgmap v10879: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:39 smithi078 ceph-mon[91136]: pgmap v10879: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:39 smithi146 ceph-mon[103278]: pgmap v10879: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:40.953 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:22:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:22:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:41 smithi146 ceph-mon[103278]: pgmap v10880: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:41 smithi078 ceph-mon[99959]: pgmap v10880: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:41 smithi078 ceph-mon[91136]: pgmap v10880: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:41.679Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:41.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:41.680Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:43 smithi146 ceph-mon[103278]: pgmap v10881: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:43 smithi078 ceph-mon[99959]: pgmap v10881: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:43 smithi078 ceph-mon[91136]: pgmap v10881: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:45 smithi146 ceph-mon[103278]: pgmap v10882: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:46.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:45 smithi078 ceph-mon[99959]: pgmap v10882: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:45 smithi078 ceph-mon[91136]: pgmap v10882: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:47.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.065 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:47 smithi146 ceph-mon[103278]: pgmap v10883: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:48.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:47 smithi078 ceph-mon[99959]: pgmap v10883: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:47 smithi078 ceph-mon[91136]: pgmap v10883: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:22:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:49.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:49.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:49 smithi146 ceph-mon[103278]: pgmap v10884: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:50.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:49 smithi078 ceph-mon[99959]: pgmap v10884: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:49 smithi078 ceph-mon[91136]: pgmap v10884: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:22:50] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T02:22:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:51 smithi146 ceph-mon[103278]: pgmap v10885: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[99959]: pgmap v10885: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[91136]: pgmap v10885: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:22:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:51.680Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:51.681Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:22:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:22:51.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:22:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:53 smithi146 ceph-mon[103278]: pgmap v10886: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:53 smithi078 ceph-mon[91136]: pgmap v10886: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:53 smithi078 ceph-mon[99959]: pgmap v10886: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:55.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:55 smithi146 ceph-mon[103278]: pgmap v10887: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:55 smithi078 ceph-mon[91136]: pgmap v10887: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:55 smithi078 ceph-mon[99959]: pgmap v10887: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:57.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:57 smithi078 ceph-mon[91136]: pgmap v10888: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:57.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:57 smithi078 ceph-mon[99959]: pgmap v10888: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:58.000 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:57 smithi146 ceph-mon[103278]: pgmap v10888: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:22:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:22:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:22:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:22:59 smithi146 ceph-mon[103278]: pgmap v10889: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:22:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:23:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:22:59 smithi078 ceph-mon[99959]: pgmap v10889: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:22:59 smithi078 ceph-mon[91136]: pgmap v10889: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:00.400 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 6h - - 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 6h - - 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 14s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 9s ago 6h 88.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 14s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (5h) 9s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 14s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 14s ago 6h 543M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 9s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 14s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:23:00.401 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 14s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 9s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 14s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 14s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 14s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 14s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 9s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 9s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 9s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 9s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 9s ago 6h 76.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 14s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:23:00.402 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 14s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:23:00.403 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:23:00.653 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:23:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:23:00.723 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:23:00.723 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:23:00.723 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:23:00.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:00.723 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:23:00.724 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:23:01.004 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:23:01.005 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:23:01.005 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:23:01.005 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:23:01.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:00 smithi078 ceph-mon[99959]: from='client.68739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:01.005 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:00 smithi078 ceph-mon[91136]: from='client.68739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:00 smithi146 ceph-mon[103278]: from='client.68739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:01.324 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:23:01.325 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:01.325 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:01.325 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:01.325 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:01 smithi146 ceph-mon[103278]: from='client.88664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:01 smithi146 ceph-mon[103278]: pgmap v10890: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:01 smithi146 ceph-mon[103278]: from='client.68751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2795139480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:01 smithi146 ceph-mon[103278]: from='client.88682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4022624658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:01.681Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:01.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:01.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[91136]: from='client.88664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[91136]: pgmap v10890: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[91136]: from='client.68751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2795139480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[91136]: from='client.88682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4022624658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[99959]: from='client.88664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[99959]: pgmap v10890: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[99959]: from='client.68751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2795139480' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[99959]: from='client.88682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4022624658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:03 smithi146 ceph-mon[103278]: pgmap v10891: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:03 smithi078 ceph-mon[99959]: pgmap v10891: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:03 smithi078 ceph-mon[91136]: pgmap v10891: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:05.952 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:05 smithi078 ceph-mon[99959]: pgmap v10892: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:05.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:05 smithi078 ceph-mon[91136]: pgmap v10892: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:05 smithi146 ceph-mon[103278]: pgmap v10892: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:07.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:07 smithi146 ceph-mon[103278]: pgmap v10893: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:07 smithi078 ceph-mon[91136]: pgmap v10893: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:07 smithi078 ceph-mon[99959]: pgmap v10893: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:09 smithi146 ceph-mon[103278]: pgmap v10894: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:09 smithi078 ceph-mon[91136]: pgmap v10894: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:09 smithi078 ceph-mon[99959]: pgmap v10894: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:23:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:23:11.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[99959]: pgmap v10895: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:11.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:11.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:11.942 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:11.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[91136]: pgmap v10895: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:11.942 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:11.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:11.943 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:11.943 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:11.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:11.943 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:11.682Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:11.943 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:11.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:11 smithi146 ceph-mon[103278]: pgmap v10895: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:13 smithi078 ceph-mon[91136]: pgmap v10896: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:13 smithi078 ceph-mon[99959]: pgmap v10896: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:13 smithi146 ceph-mon[103278]: pgmap v10896: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:16.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:15 smithi078 ceph-mon[99959]: pgmap v10897: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:15 smithi078 ceph-mon[91136]: pgmap v10897: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:15 smithi146 ceph-mon[103278]: pgmap v10897: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:17 smithi146 ceph-mon[103278]: pgmap v10898: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:17 smithi078 ceph-mon[99959]: pgmap v10898: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:17 smithi078 ceph-mon[91136]: pgmap v10898: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:19.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:19.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:19 smithi078 ceph-mon[91136]: pgmap v10899: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:19.881 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:19 smithi078 ceph-mon[99959]: pgmap v10899: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:19 smithi146 ceph-mon[103278]: pgmap v10899: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:20.910 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:23:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:23:21.782 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:21.682Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:21.782 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:21.683Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:21.782 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:21.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:21 smithi078 ceph-mon[91136]: pgmap v10900: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:21 smithi078 ceph-mon[99959]: pgmap v10900: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:21 smithi146 ceph-mon[103278]: pgmap v10900: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:23 smithi146 ceph-mon[103278]: pgmap v10901: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:23 smithi078 ceph-mon[91136]: pgmap v10901: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:23 smithi078 ceph-mon[99959]: pgmap v10901: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:26.090 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:25 smithi078 ceph-mon[91136]: pgmap v10902: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:26.090 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:25 smithi078 ceph-mon[99959]: pgmap v10902: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:25 smithi146 ceph-mon[103278]: pgmap v10902: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:27 smithi146 ceph-mon[103278]: pgmap v10903: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[91136]: pgmap v10903: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:27 smithi078 ceph-mon[99959]: pgmap v10903: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:29 smithi146 ceph-mon[103278]: pgmap v10904: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:29 smithi078 ceph-mon[99959]: pgmap v10904: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:29 smithi078 ceph-mon[91136]: pgmap v10904: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:23:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:23:31.620 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:23:31.900 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:31.683Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:31.900 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:31.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:31.900 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:31.684Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 6h - - 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 6h - - 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 6s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 1s ago 6h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 6s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 1s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 6s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 6s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:23:32.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 1s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (5h) 6s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 6s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 1s ago 6h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 6s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 6s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 6s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 6s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 1s ago 6h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 6s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 6s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:23:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[99959]: pgmap v10905: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[91136]: pgmap v10905: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.153 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:31 smithi146 ceph-mon[103278]: pgmap v10905: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:23:32.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:23:32.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:23:32.739 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:23:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:23:32.739 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:23:32.740 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:23:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:32 smithi078 ceph-mon[91136]: from='client.88694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:32 smithi078 ceph-mon[91136]: from='client.68781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3134642860' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:32 smithi078 ceph-mon[99959]: from='client.88694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:32 smithi078 ceph-mon[99959]: from='client.68781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:33.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3134642860' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:33.060 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:23:33.060 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:23:33.060 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:33.060 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:23:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:32 smithi146 ceph-mon[103278]: from='client.88694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:32 smithi146 ceph-mon[103278]: from='client.68781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3134642860' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:23:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:33 smithi146 ceph-mon[103278]: from='client.88706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:33 smithi146 ceph-mon[103278]: pgmap v10906: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:33 smithi146 ceph-mon[103278]: from='client.68799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2405016116' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[91136]: from='client.88706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[91136]: pgmap v10906: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[91136]: from='client.68799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2405016116' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[99959]: from='client.88706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[99959]: pgmap v10906: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[99959]: from='client.68799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:23:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2405016116' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:23:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:35 smithi146 ceph-mon[103278]: pgmap v10907: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:35 smithi078 ceph-mon[99959]: pgmap v10907: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:35 smithi078 ceph-mon[91136]: pgmap v10907: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:37 smithi146 ceph-mon[103278]: pgmap v10908: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:37 smithi078 ceph-mon[99959]: pgmap v10908: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:37 smithi078 ceph-mon[91136]: pgmap v10908: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:38.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:39.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:39.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:39 smithi146 ceph-mon[103278]: pgmap v10909: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:39 smithi078 ceph-mon[99959]: pgmap v10909: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:39 smithi078 ceph-mon[91136]: pgmap v10909: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:40.697 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:23:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:23:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:41.684Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:41.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:41.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:41 smithi146 ceph-mon[103278]: pgmap v10910: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:42.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:41 smithi078 ceph-mon[99959]: pgmap v10910: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:42.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:41 smithi078 ceph-mon[91136]: pgmap v10910: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:43 smithi146 ceph-mon[103278]: pgmap v10911: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:43 smithi078 ceph-mon[91136]: pgmap v10911: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:43 smithi078 ceph-mon[99959]: pgmap v10911: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:46.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:45 smithi078 ceph-mon[91136]: pgmap v10912: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:46.227 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:45 smithi078 ceph-mon[99959]: pgmap v10912: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:45 smithi146 ceph-mon[103278]: pgmap v10912: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:47 smithi146 ceph-mon[103278]: pgmap v10913: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:47.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[91136]: pgmap v10913: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:47 smithi078 ceph-mon[99959]: pgmap v10913: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:49.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:49 smithi146 ceph-mon[103278]: pgmap v10914: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:49.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:49 smithi078 ceph-mon[99959]: pgmap v10914: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:49 smithi078 ceph-mon[91136]: pgmap v10914: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:23:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:23:52.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:51.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:52.031 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:51.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:23:52.031 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:23:51.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[99959]: pgmap v10915: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[91136]: pgmap v10915: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:52 smithi146 ceph-mon[103278]: pgmap v10915: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:23:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:54 smithi078 ceph-mon[99959]: pgmap v10916: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:54 smithi078 ceph-mon[91136]: pgmap v10916: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:54 smithi146 ceph-mon[103278]: pgmap v10916: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:56 smithi146 ceph-mon[103278]: pgmap v10917: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:56 smithi078 ceph-mon[99959]: pgmap v10917: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:56 smithi078 ceph-mon[91136]: pgmap v10917: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:23:58.237 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:58 smithi146 ceph-mon[103278]: pgmap v10918: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:58 smithi078 ceph-mon[91136]: pgmap v10918: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:58 smithi078 ceph-mon[99959]: pgmap v10918: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:23:59.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:59.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:23:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:23:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:23:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:00.415 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:00 smithi146 ceph-mon[103278]: pgmap v10919: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:00 smithi078 ceph-mon[91136]: pgmap v10919: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:00 smithi078 ceph-mon[99959]: pgmap v10919: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:24:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:24:01.950 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:01.685Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:01.950 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:01.685Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:01.950 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:01.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:02 smithi146 ceph-mon[103278]: pgmap v10920: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:02.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:02 smithi078 ceph-mon[91136]: pgmap v10920: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:02 smithi078 ceph-mon[99959]: pgmap v10920: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:03.354 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:24:03.883 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:24:03.883 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 6h - - 2024-09-21T02:24:03.883 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 6h - - 2024-09-21T02:24:03.883 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 17s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:24:03.883 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 12s ago 6h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:24:03.883 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 17s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 12s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 17s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 17s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (5h) 12s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 17s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 17s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 12s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 17s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 17s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 17s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 17s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 12s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 12s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 12s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 12s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 12s ago 6h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:24:03.884 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 17s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:24:03.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:24:03.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 17s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:24:03.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:24:04.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:04 smithi078 ceph-mon[99959]: pgmap v10921: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:04.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:04 smithi078 ceph-mon[91136]: pgmap v10921: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:24:04.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:24:04.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:04.199 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:24:04.199 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:24:04.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:24:04.199 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:24:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:24:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:24:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:24:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:24:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:24:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:24:04.469 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:24:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:04 smithi146 ceph-mon[103278]: pgmap v10921: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:04.778 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:24:04.778 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:24:04.778 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:24:04.778 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:04.779 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:05.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:05 smithi146 ceph-mon[103278]: from='client.68811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:05 smithi146 ceph-mon[103278]: from='client.68817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:05 smithi146 ceph-mon[103278]: from='client.68823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2213824795' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2546652624' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[99959]: from='client.68811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[99959]: from='client.68817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[99959]: from='client.68823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2213824795' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2546652624' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[91136]: from='client.68811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[91136]: from='client.68817 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[91136]: from='client.68823 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2213824795' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:05.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2546652624' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:06.359 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:06 smithi078 ceph-mon[99959]: pgmap v10922: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:06.359 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:06 smithi078 ceph-mon[99959]: from='client.88754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:06.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:06 smithi078 ceph-mon[91136]: pgmap v10922: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:06.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:06 smithi078 ceph-mon[91136]: from='client.88754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:06 smithi146 ceph-mon[103278]: pgmap v10922: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:06 smithi146 ceph-mon[103278]: from='client.88754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:07 smithi146 ceph-mon[103278]: pgmap v10923: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[91136]: pgmap v10923: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[99959]: pgmap v10923: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:09 smithi146 ceph-mon[103278]: pgmap v10924: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:09.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:09 smithi078 ceph-mon[99959]: pgmap v10924: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:09 smithi078 ceph-mon[91136]: pgmap v10924: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:24:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:24:11.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:11 smithi146 ceph-mon[103278]: pgmap v10925: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:11.472 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[99959]: pgmap v10925: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:11.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.686 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[91136]: pgmap v10925: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:11.686 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:11.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:11.686Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:12.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:11.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:12.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:11.686Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:13 smithi146 ceph-mon[103278]: pgmap v10926: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:13 smithi078 ceph-mon[91136]: pgmap v10926: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:13 smithi078 ceph-mon[99959]: pgmap v10926: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:14.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:14.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:15 smithi146 ceph-mon[103278]: pgmap v10927: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:15 smithi078 ceph-mon[91136]: pgmap v10927: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:15 smithi078 ceph-mon[99959]: pgmap v10927: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:17 smithi146 ceph-mon[103278]: pgmap v10928: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:17 smithi078 ceph-mon[91136]: pgmap v10928: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:17 smithi078 ceph-mon[99959]: pgmap v10928: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:18.653 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:18.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:19.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:19 smithi146 ceph-mon[103278]: pgmap v10929: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:19 smithi078 ceph-mon[99959]: pgmap v10929: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:19 smithi078 ceph-mon[91136]: pgmap v10929: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:24:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:24:21.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:21 smithi078 ceph-mon[91136]: pgmap v10930: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:21.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:21 smithi078 ceph-mon[99959]: pgmap v10930: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:21 smithi146 ceph-mon[103278]: pgmap v10930: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:21.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:21.687Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:21.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:21.687Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:21.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:21.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:23 smithi078 ceph-mon[99959]: pgmap v10931: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:23 smithi078 ceph-mon[91136]: pgmap v10931: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:23.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:23 smithi146 ceph-mon[103278]: pgmap v10931: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:25 smithi078 ceph-mon[91136]: pgmap v10932: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:25 smithi078 ceph-mon[99959]: pgmap v10932: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:25 smithi146 ceph-mon[103278]: pgmap v10932: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:26.265 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 02:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T02:24:25.847147383Z level=info msg="Completed cleanup jobs" duration=11.957937ms 2024-09-21T02:24:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:27 smithi146 ceph-mon[103278]: pgmap v10933: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[91136]: pgmap v10933: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[99959]: pgmap v10933: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:28.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:29 smithi146 ceph-mon[103278]: pgmap v10934: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:29 smithi078 ceph-mon[91136]: pgmap v10934: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:29 smithi078 ceph-mon[99959]: pgmap v10934: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:24:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:24:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:31 smithi146 ceph-mon[103278]: pgmap v10935: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[91136]: pgmap v10935: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[99959]: pgmap v10935: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:31.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:31.688Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:31.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:31.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:31.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:31.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:33 smithi146 ceph-mon[103278]: pgmap v10936: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:33.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:33 smithi078 ceph-mon[99959]: pgmap v10936: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:33 smithi078 ceph-mon[91136]: pgmap v10936: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:35.080 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 6h - - 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 4s ago 6h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 4s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:24:35.611 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 4s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 4s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 9s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 9s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 9s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 9s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 4s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 4s ago 6h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 9s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 9s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:24:35.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:24:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:35 smithi146 ceph-mon[103278]: pgmap v10937: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:35 smithi146 ceph-mon[103278]: from='client.68847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:35 smithi078 ceph-mon[91136]: pgmap v10937: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:35 smithi078 ceph-mon[91136]: from='client.68847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:35 smithi078 ceph-mon[99959]: pgmap v10937: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:35 smithi078 ceph-mon[99959]: from='client.68847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:24:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:24:35.923 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:24:35.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:24:35.925 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:24:36.198 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:24:36.199 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:24:36.200 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:24:36.200 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:24:36.513 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:24:36.513 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:24:36.513 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:24:36.513 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:36.513 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:24:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:36 smithi146 ceph-mon[103278]: from='client.68853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:36 smithi146 ceph-mon[103278]: from='client.68859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:36.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/762500964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:36 smithi078 ceph-mon[91136]: from='client.68853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:36 smithi078 ceph-mon[91136]: from='client.68859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:36.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/762500964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:36 smithi078 ceph-mon[99959]: from='client.68853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:36 smithi078 ceph-mon[99959]: from='client.68859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/762500964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:24:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:37 smithi146 ceph-mon[103278]: from='client.88790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:37 smithi146 ceph-mon[103278]: pgmap v10938: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1237317848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:37 smithi078 ceph-mon[99959]: from='client.88790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:37 smithi078 ceph-mon[99959]: pgmap v10938: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1237317848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:37 smithi078 ceph-mon[91136]: from='client.88790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:24:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:37 smithi078 ceph-mon[91136]: pgmap v10938: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1237317848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:24:38.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:39.684 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:39 smithi146 ceph-mon[103278]: pgmap v10939: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:39 smithi078 ceph-mon[99959]: pgmap v10939: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:39 smithi078 ceph-mon[91136]: pgmap v10939: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:24:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T02:24:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:41 smithi146 ceph-mon[103278]: pgmap v10940: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:41.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:41 smithi078 ceph-mon[91136]: pgmap v10940: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:41 smithi078 ceph-mon[99959]: pgmap v10940: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:41.688Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:41.688Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:41.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:43.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:43 smithi078 ceph-mon[91136]: pgmap v10941: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:43.757 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:43 smithi078 ceph-mon[99959]: pgmap v10941: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:43 smithi146 ceph-mon[103278]: pgmap v10941: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:44.729 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:44.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:45 smithi146 ceph-mon[103278]: pgmap v10942: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:45 smithi078 ceph-mon[91136]: pgmap v10942: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:45 smithi078 ceph-mon[99959]: pgmap v10942: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:47 smithi146 ceph-mon[103278]: pgmap v10943: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[91136]: pgmap v10943: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[99959]: pgmap v10943: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:48.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:49 smithi146 ceph-mon[103278]: pgmap v10944: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:49 smithi078 ceph-mon[99959]: pgmap v10944: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:50.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:49 smithi078 ceph-mon[91136]: pgmap v10944: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:24:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:24:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:51 smithi146 ceph-mon[103278]: pgmap v10945: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[99959]: pgmap v10945: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[91136]: pgmap v10945: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:24:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:51.689Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:51.689Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:24:51.786 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:24:51.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:24:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:53 smithi146 ceph-mon[103278]: pgmap v10946: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:54.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:53 smithi078 ceph-mon[91136]: pgmap v10946: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:53 smithi078 ceph-mon[99959]: pgmap v10946: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:55 smithi146 ceph-mon[103278]: pgmap v10947: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:55 smithi078 ceph-mon[91136]: pgmap v10947: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:55 smithi078 ceph-mon[99959]: pgmap v10947: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:24:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:57 smithi146 ceph-mon[103278]: pgmap v10948: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:58.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:57 smithi078 ceph-mon[91136]: pgmap v10948: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:57 smithi078 ceph-mon[99959]: pgmap v10948: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:24:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:58 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:58 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:58 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:24:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:24:59.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:24:59 smithi146 ceph-mon[103278]: pgmap v10949: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:24:59 smithi078 ceph-mon[91136]: pgmap v10949: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:24:59 smithi078 ceph-mon[99959]: pgmap v10949: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:25:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:25:01.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:01 smithi146 ceph-mon[103278]: pgmap v10950: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:01 smithi078 ceph-mon[91136]: pgmap v10950: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:01 smithi078 ceph-mon[99959]: pgmap v10950: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:01.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:01.690Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:01.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:03 smithi078 ceph-mon[91136]: pgmap v10951: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:03 smithi078 ceph-mon[99959]: pgmap v10951: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:03 smithi146 ceph-mon[103278]: pgmap v10951: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:05 smithi078 ceph-mon[91136]: pgmap v10952: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:05 smithi078 ceph-mon[99959]: pgmap v10952: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:05 smithi146 ceph-mon[103278]: pgmap v10952: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:06.809 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:25:07.341 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 6h - - 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 6h - - 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 0s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 15s ago 6h 88.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 0s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 15s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 0s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 0s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 15s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:25:07.342 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 0s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 0s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 15s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5h) 0s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 0s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 0s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 0s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 15s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 15s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 15s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 15s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 15s ago 6h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 0s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 0s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:25:07.343 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:25:07.648 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:25:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:25:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:25:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:25:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:25:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:25:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:25:07.651 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:25:07.921 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:25:07.921 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:25:07.922 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: pgmap v10953: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: from='client.68883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: from='client.68889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2137323297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: pgmap v10953: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:07.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:07.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:07.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:07.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: from='client.68883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:07.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: from='client.68889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:07.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2137323297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: pgmap v10953: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: from='client.68883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: from='client.68889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2137323297' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:08.234 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:25:08.234 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:08.234 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:08 smithi146 ceph-mon[103278]: from='client.68895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:08 smithi146 ceph-mon[103278]: from='client.88826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:08 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:09.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1012693152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[99959]: from='client.68895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:09.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[99959]: from='client.88826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1012693152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[91136]: from='client.68895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[91136]: from='client.88826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1012693152' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:09 smithi146 ceph-mon[103278]: pgmap v10954: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:09 smithi078 ceph-mon[99959]: pgmap v10954: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:09 smithi078 ceph-mon[91136]: pgmap v10954: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:25:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:25:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:11 smithi146 ceph-mon[103278]: pgmap v10955: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[99959]: pgmap v10955: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[91136]: pgmap v10955: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:11.690Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:11.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:11.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:13 smithi146 ceph-mon[103278]: pgmap v10956: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:13 smithi078 ceph-mon[91136]: pgmap v10956: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:13 smithi078 ceph-mon[99959]: pgmap v10956: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:15 smithi146 ceph-mon[103278]: pgmap v10957: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:15 smithi078 ceph-mon[91136]: pgmap v10957: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:15 smithi078 ceph-mon[99959]: pgmap v10957: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:17 smithi146 ceph-mon[103278]: pgmap v10958: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:17 smithi078 ceph-mon[91136]: pgmap v10958: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:18.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:17 smithi078 ceph-mon[99959]: pgmap v10958: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:19.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:18 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:18 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:18 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:19 smithi146 ceph-mon[103278]: pgmap v10959: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:19 smithi078 ceph-mon[99959]: pgmap v10959: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:19 smithi078 ceph-mon[91136]: pgmap v10959: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:25:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:25:21.951 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:21 smithi078 ceph-mon[99959]: pgmap v10960: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:21.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:21 smithi078 ceph-mon[91136]: pgmap v10960: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:21.951 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:21.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:21.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:21.691Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:21.952 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:21.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:21 smithi146 ceph-mon[103278]: pgmap v10960: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:23 smithi146 ceph-mon[103278]: pgmap v10961: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:24.032 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:23 smithi078 ceph-mon[99959]: pgmap v10961: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:24.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:23 smithi078 ceph-mon[91136]: pgmap v10961: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:25 smithi146 ceph-mon[103278]: pgmap v10962: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:25 smithi078 ceph-mon[91136]: pgmap v10962: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:25 smithi078 ceph-mon[99959]: pgmap v10962: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:27 smithi146 ceph-mon[103278]: pgmap v10963: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[99959]: pgmap v10963: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[91136]: pgmap v10963: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:29.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:28 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:29.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:28 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:28 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:29 smithi146 ceph-mon[103278]: pgmap v10964: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:30.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:29 smithi078 ceph-mon[99959]: pgmap v10964: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:29 smithi078 ceph-mon[91136]: pgmap v10964: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:25:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:25:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:31 smithi146 ceph-mon[103278]: pgmap v10965: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[91136]: pgmap v10965: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[99959]: pgmap v10965: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:31.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:31.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:31.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:33 smithi146 ceph-mon[103278]: pgmap v10966: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:33 smithi078 ceph-mon[91136]: pgmap v10966: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:33 smithi078 ceph-mon[99959]: pgmap v10966: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:35 smithi146 ceph-mon[103278]: pgmap v10967: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:35 smithi078 ceph-mon[91136]: pgmap v10967: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:35 smithi078 ceph-mon[99959]: pgmap v10967: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:37 smithi146 ceph-mon[103278]: pgmap v10968: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:37 smithi078 ceph-mon[91136]: pgmap v10968: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:37 smithi078 ceph-mon[99959]: pgmap v10968: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:38.530 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:25:39.060 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 6h - - 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 6h - - 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 12s ago 6h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 7s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 12s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 7s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 12s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 12s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 7s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 12s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 12s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:25:39.061 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 7s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 12s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (5h) 12s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 12s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 12s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 7s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 7s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 7s ago 6h 77.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 12s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 12s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:25:39.062 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:25:39.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:38 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:39.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:38 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:39.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:38 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:25:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:25:39.378 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:25:39.645 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:25:39.646 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:25:39.958 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:25:39.958 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:39.958 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:25:39.958 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:39.958 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:25:39.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[91136]: pgmap v10969: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[91136]: from='client.68919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[91136]: from='client.68925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[91136]: from='client.88850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2435032824' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[99959]: pgmap v10969: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[99959]: from='client.68919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[99959]: from='client.68925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[99959]: from='client.88850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:39.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2435032824' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:39 smithi146 ceph-mon[103278]: pgmap v10969: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:39 smithi146 ceph-mon[103278]: from='client.68919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:39 smithi146 ceph-mon[103278]: from='client.68925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:39 smithi146 ceph-mon[103278]: from='client.88850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2435032824' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:25:40.875 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:25:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:25:41.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:40 smithi146 ceph-mon[103278]: from='client.68943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4003053804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:40 smithi078 ceph-mon[91136]: from='client.68943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4003053804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:40 smithi078 ceph-mon[99959]: from='client.68943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:25:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4003053804' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:25:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:41 smithi078 ceph-mon[91136]: pgmap v10970: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:41 smithi078 ceph-mon[99959]: pgmap v10970: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:41.691Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:41.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:41.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:42.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:41 smithi146 ceph-mon[103278]: pgmap v10970: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:43 smithi146 ceph-mon[103278]: pgmap v10971: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:43 smithi078 ceph-mon[99959]: pgmap v10971: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:43 smithi078 ceph-mon[91136]: pgmap v10971: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:46.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:45 smithi078 ceph-mon[91136]: pgmap v10972: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:46.163 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:45 smithi078 ceph-mon[99959]: pgmap v10972: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:45 smithi146 ceph-mon[103278]: pgmap v10972: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[99959]: pgmap v10973: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[91136]: pgmap v10973: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:47 smithi146 ceph-mon[103278]: pgmap v10973: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:48 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:49.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:48 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:49.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:48 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:25:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:49 smithi146 ceph-mon[103278]: pgmap v10974: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:49 smithi078 ceph-mon[99959]: pgmap v10974: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:49 smithi078 ceph-mon[91136]: pgmap v10974: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:25:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:25:51.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[91136]: pgmap v10975: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:51.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:51.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:51.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:51.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:51.692Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:51.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:51.692Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:25:51.980 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:25:51.693Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:25:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:51 smithi146 ceph-mon[103278]: pgmap v10975: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[99959]: pgmap v10975: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:52.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:25:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:53 smithi146 ceph-mon[103278]: pgmap v10976: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:53 smithi078 ceph-mon[91136]: pgmap v10976: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:53 smithi078 ceph-mon[99959]: pgmap v10976: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:55 smithi146 ceph-mon[103278]: pgmap v10977: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:55 smithi078 ceph-mon[91136]: pgmap v10977: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:55 smithi078 ceph-mon[99959]: pgmap v10977: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:25:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:57 smithi146 ceph-mon[103278]: pgmap v10978: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:57 smithi078 ceph-mon[99959]: pgmap v10978: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:57 smithi078 ceph-mon[91136]: pgmap v10978: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:25:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:25:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:25:59 smithi146 ceph-mon[103278]: pgmap v10979: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:25:59 smithi078 ceph-mon[91136]: pgmap v10979: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:25:59 smithi078 ceph-mon[99959]: pgmap v10979: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:26:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:26:01.997 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:01.693Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:01.997 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:01.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:01.997 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:01.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:01 smithi146 ceph-mon[103278]: pgmap v10980: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:02.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:01 smithi078 ceph-mon[99959]: pgmap v10980: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:01 smithi078 ceph-mon[91136]: pgmap v10980: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:04 smithi146 ceph-mon[103278]: pgmap v10981: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:04 smithi078 ceph-mon[99959]: pgmap v10981: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:03 smithi078 ceph-mon[91136]: pgmap v10981: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:06.233 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:06 smithi078 ceph-mon[91136]: pgmap v10982: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:06.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:06 smithi078 ceph-mon[99959]: pgmap v10982: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:06 smithi146 ceph-mon[103278]: pgmap v10982: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:08.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[99959]: pgmap v10983: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:08.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.197 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[91136]: pgmap v10983: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:08.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.198 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:08 smithi146 ceph-mon[103278]: pgmap v10983: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:10.256 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:26:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:10 smithi146 ceph-mon[103278]: pgmap v10984: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:10.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:10 smithi078 ceph-mon[91136]: pgmap v10984: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:10 smithi078 ceph-mon[99959]: pgmap v10984: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 6h - - 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 6h - - 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 3s ago 6h 30.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 18s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 3s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 18s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 3s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:26:10.785 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 3s ago 6h 544M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 18s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 3s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 3s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 18s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 3s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 3s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 3s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 3s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 18s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 18s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 18s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 18s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:26:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 18s ago 6h 77.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:26:10.787 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 3s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:26:10.787 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:26:10.787 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 3s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:26:10.787 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:26:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:26:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:26:11.108 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:26:11.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:11.110 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:26:11.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:26:11.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:26:11.110 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:26:11.386 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:26:11.387 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:26:11.387 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:26:11.387 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:26:11.387 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:26:11.387 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:26:11.387 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:26:11.696 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:26:11.696 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:11.696 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:11.696 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:11.696 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:11.694Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:11.694Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:11.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='client.68955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: pgmap v10985: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='client.68961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='client.88886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/602469147' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1036276145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='client.68955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: pgmap v10985: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='client.68961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='client.88886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/602469147' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1036276145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='client.68955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: pgmap v10985: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='client.68961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='client.88886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/602469147' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1036276145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:13 smithi146 ceph-mon[103278]: from='client.68979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:13 smithi146 ceph-mon[103278]: pgmap v10986: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:13 smithi078 ceph-mon[91136]: from='client.68979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:13.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:13 smithi078 ceph-mon[91136]: pgmap v10986: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:13 smithi078 ceph-mon[99959]: from='client.68979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:13 smithi078 ceph-mon[99959]: pgmap v10986: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:15.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:15 smithi146 ceph-mon[103278]: pgmap v10987: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:15.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:15 smithi078 ceph-mon[91136]: pgmap v10987: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:15 smithi078 ceph-mon[99959]: pgmap v10987: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:17 smithi146 ceph-mon[103278]: pgmap v10988: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:17 smithi078 ceph-mon[91136]: pgmap v10988: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:17 smithi078 ceph-mon[99959]: pgmap v10988: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:19 smithi146 ceph-mon[103278]: pgmap v10989: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:19.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:19 smithi078 ceph-mon[91136]: pgmap v10989: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:19 smithi078 ceph-mon[99959]: pgmap v10989: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:26:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:26:21.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:21 smithi078 ceph-mon[91136]: pgmap v10990: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:21.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:21 smithi078 ceph-mon[99959]: pgmap v10990: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:21 smithi146 ceph-mon[103278]: pgmap v10990: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:21.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:21.695Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:21.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:23 smithi146 ceph-mon[103278]: pgmap v10991: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:23 smithi078 ceph-mon[91136]: pgmap v10991: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:23 smithi078 ceph-mon[99959]: pgmap v10991: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:25 smithi146 ceph-mon[103278]: pgmap v10992: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:25 smithi078 ceph-mon[91136]: pgmap v10992: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:25 smithi078 ceph-mon[99959]: pgmap v10992: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:27.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[91136]: pgmap v10993: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:27.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[99959]: pgmap v10993: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:27.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:27 smithi146 ceph-mon[103278]: pgmap v10993: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:29 smithi146 ceph-mon[103278]: pgmap v10994: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:29 smithi078 ceph-mon[91136]: pgmap v10994: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:29 smithi078 ceph-mon[99959]: pgmap v10994: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:26:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:26:31.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:31 smithi078 ceph-mon[91136]: pgmap v10995: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:31.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:31 smithi078 ceph-mon[99959]: pgmap v10995: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:31 smithi146 ceph-mon[103278]: pgmap v10995: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:31.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:31.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:31.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:34 smithi146 ceph-mon[103278]: pgmap v10996: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:34 smithi078 ceph-mon[91136]: pgmap v10996: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:34 smithi078 ceph-mon[99959]: pgmap v10996: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:36 smithi146 ceph-mon[103278]: pgmap v10997: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:36 smithi078 ceph-mon[91136]: pgmap v10997: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:36 smithi078 ceph-mon[99959]: pgmap v10997: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:38.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:38 smithi078 ceph-mon[91136]: pgmap v10998: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:38.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:38 smithi078 ceph-mon[99959]: pgmap v10998: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:38 smithi146 ceph-mon[103278]: pgmap v10998: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:40 smithi146 ceph-mon[103278]: pgmap v10999: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:40.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:40 smithi078 ceph-mon[91136]: pgmap v10999: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:40 smithi078 ceph-mon[99959]: pgmap v10999: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:26:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:26:41.989 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:41.695Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:41.989 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:41.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:41.989 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:41.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:41.989 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:26:42.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:42 smithi078 ceph-mon[99959]: pgmap v11000: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:42 smithi078 ceph-mon[91136]: pgmap v11000: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:42.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:42 smithi146 ceph-mon[103278]: pgmap v11000: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:42.514 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 6h - - 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 6h - - 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 15s ago 6h 30.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 10s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 15s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 10s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 15s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 15s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 10s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 15s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:26:42.515 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 15s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 10s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 15s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 15s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (5h) 15s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 15s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 10s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 10s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 10s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 10s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 10s ago 6h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 15s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 15s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:26:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:26:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:26:42.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:26:42.825 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:26:43.373 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:43 smithi146 ceph-mon[103278]: from='client.88910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:43.373 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1373342139' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:26:43.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:43 smithi078 ceph-mon[99959]: from='client.88910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:43.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1373342139' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:43.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:43 smithi078 ceph-mon[91136]: from='client.88910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:43.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1373342139' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:26:44.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:44 smithi146 ceph-mon[103278]: from='client.68997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:44 smithi146 ceph-mon[103278]: pgmap v11001: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:44.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:44 smithi146 ceph-mon[103278]: from='client.88922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:44 smithi146 ceph-mon[103278]: from='client.69015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1160940445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:44.400 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[91136]: from='client.68997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[91136]: pgmap v11001: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[91136]: from='client.88922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[91136]: from='client.69015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1160940445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[99959]: from='client.68997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[99959]: pgmap v11001: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[99959]: from='client.88922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[99959]: from='client.69015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:26:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1160940445' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:26:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:46 smithi146 ceph-mon[103278]: pgmap v11002: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:46 smithi078 ceph-mon[91136]: pgmap v11002: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:46 smithi078 ceph-mon[99959]: pgmap v11002: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:48.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[91136]: pgmap v11003: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:48.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.355 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.355 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[99959]: pgmap v11003: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:48.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:48 smithi146 ceph-mon[103278]: pgmap v11003: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:50 smithi146 ceph-mon[103278]: pgmap v11004: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:50 smithi078 ceph-mon[91136]: pgmap v11004: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:50 smithi078 ceph-mon[99959]: pgmap v11004: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:26:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:26:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:51 smithi146 ceph-mon[103278]: pgmap v11005: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:51 smithi078 ceph-mon[99959]: pgmap v11005: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:51 smithi078 ceph-mon[91136]: pgmap v11005: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:51.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:51.696Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:26:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:26:51.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:26:53.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:53 smithi146 ceph-mon[103278]: pgmap v11006: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[91136]: pgmap v11006: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:26:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:53 smithi078 ceph-mon[99959]: pgmap v11006: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:55 smithi146 ceph-mon[103278]: pgmap v11007: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:55.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:55 smithi078 ceph-mon[99959]: pgmap v11007: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:55 smithi078 ceph-mon[91136]: pgmap v11007: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:57 smithi146 ceph-mon[103278]: pgmap v11008: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:57 smithi078 ceph-mon[99959]: pgmap v11008: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:57 smithi078 ceph-mon[91136]: pgmap v11008: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:26:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:59 smithi146 ceph-mon[103278]: pgmap v11009: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:26:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:59 smithi078 ceph-mon[91136]: pgmap v11009: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:26:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:26:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:26:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:59 smithi078 ceph-mon[99959]: pgmap v11009: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:26:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:26:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:27:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:27:01.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:01 smithi078 ceph-mon[99959]: pgmap v11010: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:01.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:01 smithi078 ceph-mon[91136]: pgmap v11010: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:01.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:01 smithi146 ceph-mon[103278]: pgmap v11010: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:01.696Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:01.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:01.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:03 smithi078 ceph-mon[91136]: pgmap v11011: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:03 smithi078 ceph-mon[99959]: pgmap v11011: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:03 smithi146 ceph-mon[103278]: pgmap v11011: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:05 smithi146 ceph-mon[103278]: pgmap v11012: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:05 smithi078 ceph-mon[91136]: pgmap v11012: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:05 smithi078 ceph-mon[99959]: pgmap v11012: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:07.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:07 smithi078 ceph-mon[91136]: pgmap v11013: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:07.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:07 smithi078 ceph-mon[99959]: pgmap v11013: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:07.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:07 smithi146 ceph-mon[103278]: pgmap v11013: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:09.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:09 smithi146 ceph-mon[103278]: pgmap v11014: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:09.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:09 smithi078 ceph-mon[91136]: pgmap v11014: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:09 smithi078 ceph-mon[99959]: pgmap v11014: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:10.627 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:27:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:27:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:11 smithi146 ceph-mon[103278]: pgmap v11015: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:11 smithi078 ceph-mon[91136]: pgmap v11015: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:11 smithi078 ceph-mon[99959]: pgmap v11015: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:11.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:11.697Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:11.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:13.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[99959]: pgmap v11016: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:13.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:13 smithi078 ceph-mon[91136]: pgmap v11016: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:13.705 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:27:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:13 smithi146 ceph-mon[103278]: pgmap v11016: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:14.228 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:27:14.228 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 6h - - 2024-09-21T02:27:14.228 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 6h - - 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 6s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 1s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 6s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 1s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 6s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 6s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 1s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 6s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 6s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 1s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 6s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:27:14.229 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 6s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 6s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5h) 6s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 1s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 1s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 1s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 1s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 1s ago 6h 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 6s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 6s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:27:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:27:14.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:14 smithi078 ceph-mon[99959]: from='client.88946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:14 smithi078 ceph-mon[99959]: from='client.88952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:14 smithi078 ceph-mon[91136]: from='client.88946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:14 smithi078 ceph-mon[91136]: from='client.88952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:14.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:27:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:27:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:27:14.539 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:27:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:14 smithi146 ceph-mon[103278]: from='client.88946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:14.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:14 smithi146 ceph-mon[103278]: from='client.88952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:27:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:27:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:27:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:27:14.809 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:27:15.120 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:27:15.120 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:15.120 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:15.120 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:15.120 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[91136]: from='client.88958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[91136]: pgmap v11017: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3396581448' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[91136]: from='client.69051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3576850005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[99959]: from='client.88958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[99959]: pgmap v11017: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3396581448' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[99959]: from='client.69051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3576850005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:15 smithi146 ceph-mon[103278]: from='client.88958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:15 smithi146 ceph-mon[103278]: pgmap v11017: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3396581448' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:15 smithi146 ceph-mon[103278]: from='client.69051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3576850005' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:17.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:17 smithi146 ceph-mon[103278]: pgmap v11018: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:17.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:17 smithi078 ceph-mon[99959]: pgmap v11018: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:17 smithi078 ceph-mon[91136]: pgmap v11018: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:19 smithi146 ceph-mon[103278]: pgmap v11019: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:19.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:19 smithi078 ceph-mon[99959]: pgmap v11019: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:19 smithi078 ceph-mon[91136]: pgmap v11019: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:27:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:27:21.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:21 smithi078 ceph-mon[91136]: pgmap v11020: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:21.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:21 smithi078 ceph-mon[99959]: pgmap v11020: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:21.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:21 smithi146 ceph-mon[103278]: pgmap v11020: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:21.697Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:21.698Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:21.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:23.664 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:23 smithi146 ceph-mon[103278]: pgmap v11021: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:23 smithi078 ceph-mon[91136]: pgmap v11021: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:23 smithi078 ceph-mon[99959]: pgmap v11021: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:25.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:25 smithi146 ceph-mon[103278]: pgmap v11022: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:25.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:25 smithi078 ceph-mon[91136]: pgmap v11022: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:25 smithi078 ceph-mon[99959]: pgmap v11022: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:27 smithi146 ceph-mon[103278]: pgmap v11023: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:27 smithi078 ceph-mon[91136]: pgmap v11023: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:27 smithi078 ceph-mon[99959]: pgmap v11023: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:29 smithi078 ceph-mon[91136]: pgmap v11024: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:29 smithi078 ceph-mon[99959]: pgmap v11024: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:29 smithi146 ceph-mon[103278]: pgmap v11024: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:30.739 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:27:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:27:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:31 smithi146 ceph-mon[103278]: pgmap v11025: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:31 smithi078 ceph-mon[91136]: pgmap v11025: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:31 smithi078 ceph-mon[99959]: pgmap v11025: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:31.698Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:31.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:31.699Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:33.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:33 smithi146 ceph-mon[103278]: pgmap v11026: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[91136]: pgmap v11026: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[99959]: pgmap v11026: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:35.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:35 smithi146 ceph-mon[103278]: pgmap v11027: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:35.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:35 smithi078 ceph-mon[91136]: pgmap v11027: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:35 smithi078 ceph-mon[99959]: pgmap v11027: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:37 smithi146 ceph-mon[103278]: pgmap v11028: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:37.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:37 smithi078 ceph-mon[91136]: pgmap v11028: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:37 smithi078 ceph-mon[99959]: pgmap v11028: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:39 smithi146 ceph-mon[103278]: pgmap v11029: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:39 smithi078 ceph-mon[91136]: pgmap v11029: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:39 smithi078 ceph-mon[99959]: pgmap v11029: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:27:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:27:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:41 smithi146 ceph-mon[103278]: pgmap v11030: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:41.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:41 smithi078 ceph-mon[91136]: pgmap v11030: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:41 smithi078 ceph-mon[99959]: pgmap v11030: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:41.699Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:41.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:41.700Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:43 smithi146 ceph-mon[103278]: pgmap v11031: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:43 smithi078 ceph-mon[91136]: pgmap v11031: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:43 smithi078 ceph-mon[99959]: pgmap v11031: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:44.697 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:44.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:45.413 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:27:45.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:45 smithi146 ceph-mon[103278]: pgmap v11032: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:45.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:45 smithi078 ceph-mon[99959]: pgmap v11032: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:45 smithi078 ceph-mon[91136]: pgmap v11032: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 18s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 13s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 6h 371M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 18s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:27:45.942 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 13s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (5h) 13s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 13s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 13s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 13s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 6h 74.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 18s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 18s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:27:45.943 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:27:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:27:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:27:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:27:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:27:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:27:46.254 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:27:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:46 smithi146 ceph-mon[103278]: from='client.69063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:46 smithi146 ceph-mon[103278]: from='client.88988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:46 smithi146 ceph-mon[103278]: from='client.88994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/582491416' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:27:46.526 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[99959]: from='client.69063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[99959]: from='client.88988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[99959]: from='client.88994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/582491416' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[91136]: from='client.69063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[91136]: from='client.88988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[91136]: from='client.88994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:46.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/582491416' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:27:46.832 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:27:46.833 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:27:46.833 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:27:47.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:47 smithi078 ceph-mon[91136]: pgmap v11033: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:47.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:47 smithi078 ceph-mon[91136]: from='client.69087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:47.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1540484495' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:47.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:47 smithi078 ceph-mon[99959]: pgmap v11033: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:47.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:47 smithi078 ceph-mon[99959]: from='client.69087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:47.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1540484495' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:47 smithi146 ceph-mon[103278]: pgmap v11033: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:47 smithi146 ceph-mon[103278]: from='client.69087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:27:47.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1540484495' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:27:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:49 smithi146 ceph-mon[103278]: pgmap v11034: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:49 smithi078 ceph-mon[91136]: pgmap v11034: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:49 smithi078 ceph-mon[99959]: pgmap v11034: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:27:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:27:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:51 smithi146 ceph-mon[103278]: pgmap v11035: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:51 smithi078 ceph-mon[91136]: pgmap v11035: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:51 smithi078 ceph-mon[99959]: pgmap v11035: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:51.700Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:51.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:27:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:27:51.701Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:27:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:53 smithi146 ceph-mon[103278]: pgmap v11036: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[91136]: pgmap v11036: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[99959]: pgmap v11036: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:27:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:55 smithi146 ceph-mon[103278]: pgmap v11037: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:56.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:55 smithi078 ceph-mon[99959]: pgmap v11037: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:55 smithi078 ceph-mon[91136]: pgmap v11037: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:27:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:57 smithi146 ceph-mon[103278]: pgmap v11038: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:58.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:57 smithi078 ceph-mon[99959]: pgmap v11038: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:57 smithi078 ceph-mon[91136]: pgmap v11038: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:27:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:27:59.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:27:59.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:27:59 smithi146 ceph-mon[103278]: pgmap v11039: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:00.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:27:59 smithi078 ceph-mon[99959]: pgmap v11039: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:27:59 smithi078 ceph-mon[91136]: pgmap v11039: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:28:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:28:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:01 smithi146 ceph-mon[103278]: pgmap v11040: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:01 smithi078 ceph-mon[91136]: pgmap v11040: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:01 smithi078 ceph-mon[99959]: pgmap v11040: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:01.701Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:01.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:01.702Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:03 smithi146 ceph-mon[103278]: pgmap v11041: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:04.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:03 smithi078 ceph-mon[91136]: pgmap v11041: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:03 smithi078 ceph-mon[99959]: pgmap v11041: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:05.845 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:05 smithi146 ceph-mon[103278]: pgmap v11042: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:05 smithi078 ceph-mon[91136]: pgmap v11042: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:05 smithi078 ceph-mon[99959]: pgmap v11042: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:07.871 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:07 smithi078 ceph-mon[91136]: pgmap v11043: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:07.872 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:07 smithi078 ceph-mon[99959]: pgmap v11043: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:07 smithi146 ceph-mon[103278]: pgmap v11043: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:09.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:10.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:10.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:09 smithi078 ceph-mon[91136]: pgmap v11044: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:10.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:10.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:09 smithi078 ceph-mon[99959]: pgmap v11044: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:09 smithi146 ceph-mon[103278]: pgmap v11044: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:10.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:28:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:28:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:11 smithi078 ceph-mon[91136]: pgmap v11045: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:11 smithi078 ceph-mon[99959]: pgmap v11045: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:11.702Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:11.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:11.703Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:11 smithi146 ceph-mon[103278]: pgmap v11045: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:13 smithi146 ceph-mon[103278]: pgmap v11046: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:14.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[91136]: pgmap v11046: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[99959]: pgmap v11046: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:15 smithi146 ceph-mon[103278]: pgmap v11047: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:15 smithi078 ceph-mon[91136]: pgmap v11047: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:15 smithi078 ceph-mon[99959]: pgmap v11047: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:17.125 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:28:17.649 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:28:17.649 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 6h - - 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 4s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 4s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 4s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 4s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 9s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:28:17.650 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 9s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 9s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 4s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 4s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 4s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 4s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 4s ago 6h 71.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 9s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 9s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:28:17.651 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:28:17.961 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:28:17.961 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:28:17.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:28:17.963 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:28:17.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:17 smithi078 ceph-mon[99959]: pgmap v11048: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:17.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:17 smithi078 ceph-mon[99959]: from='client.89018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:17.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:17 smithi078 ceph-mon[91136]: pgmap v11048: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:17.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:17 smithi078 ceph-mon[91136]: from='client.89018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:17 smithi146 ceph-mon[103278]: pgmap v11048: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:17 smithi146 ceph-mon[103278]: from='client.89018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:18.229 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:28:18.230 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:28:18.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:28:18.539 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:28:18.539 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:18.539 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:18.539 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:18.539 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:18 smithi146 ceph-mon[103278]: from='client.89024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:18 smithi146 ceph-mon[103278]: from='client.89030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2031339128' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/201204050' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[91136]: from='client.89024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[91136]: from='client.89030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2031339128' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/201204050' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[99959]: from='client.89024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[99959]: from='client.89030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2031339128' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/201204050' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:19 smithi146 ceph-mon[103278]: from='client.69123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:19 smithi146 ceph-mon[103278]: pgmap v11049: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:19 smithi078 ceph-mon[91136]: from='client.69123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:19 smithi078 ceph-mon[91136]: pgmap v11049: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:19 smithi078 ceph-mon[99959]: from='client.69123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:19 smithi078 ceph-mon[99959]: pgmap v11049: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:28:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:28:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:21 smithi146 ceph-mon[103278]: pgmap v11050: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:22.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:21 smithi078 ceph-mon[99959]: pgmap v11050: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:21 smithi078 ceph-mon[91136]: pgmap v11050: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:21.703Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:21.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:21.704Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:23 smithi146 ceph-mon[103278]: pgmap v11051: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:24.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:23 smithi078 ceph-mon[91136]: pgmap v11051: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:23 smithi078 ceph-mon[99959]: pgmap v11051: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:25 smithi146 ceph-mon[103278]: pgmap v11052: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:25 smithi078 ceph-mon[91136]: pgmap v11052: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:25 smithi078 ceph-mon[99959]: pgmap v11052: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:28.009 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:27 smithi078 ceph-mon[99959]: pgmap v11053: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:28.009 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:27 smithi078 ceph-mon[91136]: pgmap v11053: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:27 smithi146 ceph-mon[103278]: pgmap v11053: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:29.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:30 smithi146 ceph-mon[103278]: pgmap v11054: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:30 smithi078 ceph-mon[91136]: pgmap v11054: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:30 smithi078 ceph-mon[99959]: pgmap v11054: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:30.662 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:28:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:28:31.533 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:31 smithi078 ceph-mon[99959]: pgmap v11055: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:31.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:31 smithi078 ceph-mon[91136]: pgmap v11055: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:31 smithi146 ceph-mon[103278]: pgmap v11055: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:31.704Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:31.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:31.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:31.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:31.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:33 smithi146 ceph-mon[103278]: pgmap v11056: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[91136]: pgmap v11056: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[99959]: pgmap v11056: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:35 smithi146 ceph-mon[103278]: pgmap v11057: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:36.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:35 smithi078 ceph-mon[99959]: pgmap v11057: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:35 smithi078 ceph-mon[91136]: pgmap v11057: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:38.254 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:37 smithi078 ceph-mon[99959]: pgmap v11058: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:38.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:37 smithi078 ceph-mon[91136]: pgmap v11058: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:37 smithi146 ceph-mon[103278]: pgmap v11058: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:39 smithi146 ceph-mon[103278]: pgmap v11059: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:40.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:39 smithi078 ceph-mon[99959]: pgmap v11059: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:39 smithi078 ceph-mon[91136]: pgmap v11059: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:28:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:28:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:41 smithi078 ceph-mon[91136]: pgmap v11060: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:41 smithi078 ceph-mon[99959]: pgmap v11060: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:41.705Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:41.705Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:41.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:41 smithi146 ceph-mon[103278]: pgmap v11060: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:43 smithi146 ceph-mon[103278]: pgmap v11061: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:43 smithi078 ceph-mon[91136]: pgmap v11061: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:43 smithi078 ceph-mon[99959]: pgmap v11061: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:46.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:45 smithi146 ceph-mon[103278]: pgmap v11062: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:45 smithi078 ceph-mon[91136]: pgmap v11062: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:45 smithi078 ceph-mon[99959]: pgmap v11062: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:48.135 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:47 smithi078 ceph-mon[99959]: pgmap v11063: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:48.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:47 smithi078 ceph-mon[91136]: pgmap v11063: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:47 smithi146 ceph-mon[103278]: pgmap v11063: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:48.826 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:28:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.359 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 6h - - 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 1s ago 6h 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 1s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 1s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 1s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:28:49.360 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 1s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 1s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 1s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 1s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 1s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 1s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (5h) 16s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 16s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 16s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 6h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 1s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:28:49.361 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 1s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:28:49.362 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:28:49.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:49.667 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:28:49.668 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:28:49.669 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:28:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:28:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:28:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:28:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:28:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:28:49.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:28:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:28:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:28:50.244 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:50.244 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: pgmap v11064: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: from='client.69135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: from='client.69141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: from='client.69147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2546780194' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:50.244 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[99959]: from='client.89078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: pgmap v11064: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: from='client.69135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: from='client.69141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: from='client.69147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2546780194' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:50.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-mon[91136]: from='client.89078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: pgmap v11064: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: from='client.69135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: from='client.69141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: from='client.69147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2546780194' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:28:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:50 smithi146 ceph-mon[103278]: from='client.89078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:28:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:28:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:28:51.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3857195610' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:51.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-mon[99959]: pgmap v11065: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:51.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3857195610' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:51.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-mon[91136]: pgmap v11065: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3857195610' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:28:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:51 smithi146 ceph-mon[103278]: pgmap v11065: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:51.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:51.706Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:28:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:28:51.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:28:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:53 smithi146 ceph-mon[103278]: pgmap v11066: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[99959]: pgmap v11066: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[91136]: pgmap v11066: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:28:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:55 smithi146 ceph-mon[103278]: pgmap v11067: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:56.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:55 smithi078 ceph-mon[99959]: pgmap v11067: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:55 smithi078 ceph-mon[91136]: pgmap v11067: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:28:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:57 smithi146 ceph-mon[103278]: pgmap v11068: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:58.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:57 smithi078 ceph-mon[91136]: pgmap v11068: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:58.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:57 smithi078 ceph-mon[99959]: pgmap v11068: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:28:59.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:59.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:28:59.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:28:59 smithi146 ceph-mon[103278]: pgmap v11069: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:28:59 smithi078 ceph-mon[91136]: pgmap v11069: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:28:59 smithi078 ceph-mon[99959]: pgmap v11069: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:29:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:29:01.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:01.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:01.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:01.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:29:01.973 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:01.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:01 smithi146 ceph-mon[103278]: pgmap v11070: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:01 smithi078 ceph-mon[99959]: pgmap v11070: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:01 smithi078 ceph-mon[91136]: pgmap v11070: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:03 smithi146 ceph-mon[103278]: pgmap v11071: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:04.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:03 smithi078 ceph-mon[99959]: pgmap v11071: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:04.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:03 smithi078 ceph-mon[91136]: pgmap v11071: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:05 smithi146 ceph-mon[103278]: pgmap v11072: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:05 smithi078 ceph-mon[99959]: pgmap v11072: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:05 smithi078 ceph-mon[91136]: pgmap v11072: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:08.248 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:07 smithi146 ceph-mon[103278]: pgmap v11073: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:08.261 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:07 smithi078 ceph-mon[91136]: pgmap v11073: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:08.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:07 smithi078 ceph-mon[99959]: pgmap v11073: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:09.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:09.521 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:09 smithi146 ceph-mon[103278]: pgmap v11074: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[91136]: pgmap v11074: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:09 smithi078 ceph-mon[99959]: pgmap v11074: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:29:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:29:11.707 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:11 smithi078 ceph-mon[99959]: pgmap v11075: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:11.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:11 smithi078 ceph-mon[91136]: pgmap v11075: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:11 smithi146 ceph-mon[103278]: pgmap v11075: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:12.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:11.706Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:12.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:11.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:29:12.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:11.707Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:14.402 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[99959]: pgmap v11076: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[91136]: pgmap v11076: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.404 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:14 smithi146 ceph-mon[103278]: pgmap v11076: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:16.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:16 smithi078 ceph-mon[91136]: pgmap v11077: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:16 smithi078 ceph-mon[99959]: pgmap v11077: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:16 smithi146 ceph-mon[103278]: pgmap v11077: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:18 smithi146 ceph-mon[103278]: pgmap v11078: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:18 smithi078 ceph-mon[91136]: pgmap v11078: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:18 smithi078 ceph-mon[99959]: pgmap v11078: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:20 smithi146 ceph-mon[103278]: pgmap v11079: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:20 smithi078 ceph-mon[99959]: pgmap v11079: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:20 smithi078 ceph-mon[91136]: pgmap v11079: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:20.545 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:29:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:29:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 6h - - 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 6h - - 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 12s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 7s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 12s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 7s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 12s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:29:21.082 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 12s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 7s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 12s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 12s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 7s ago 6h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 12s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 12s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 12s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 12s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 7s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 7s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (5h) 7s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 7s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 7s ago 6h 74.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 12s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:29:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:29:21.084 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 12s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:29:21.084 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:29:21.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:29:21.392 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:29:21.659 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:29:21.660 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:29:21.660 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:29:21.660 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:29:21.660 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:29:21.968 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:29:21.968 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:21.968 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:21.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:21.707Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:21.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:21.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:29:21.969 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:21.708Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[99959]: pgmap v11080: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[99959]: from='client.69171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[99959]: from='client.89096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[99959]: from='client.89102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3171213956' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1264282934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[91136]: pgmap v11080: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[91136]: from='client.69171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[91136]: from='client.89096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[91136]: from='client.89102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3171213956' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1264282934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:22 smithi146 ceph-mon[103278]: pgmap v11080: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:22 smithi146 ceph-mon[103278]: from='client.69171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:22 smithi146 ceph-mon[103278]: from='client.89096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:22 smithi146 ceph-mon[103278]: from='client.89102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3171213956' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1264282934' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:23 smithi146 ceph-mon[103278]: from='client.89114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:23.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:23 smithi078 ceph-mon[99959]: from='client.89114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:23 smithi078 ceph-mon[91136]: from='client.89114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:24 smithi146 ceph-mon[103278]: pgmap v11081: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:24 smithi078 ceph-mon[99959]: pgmap v11081: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:24 smithi078 ceph-mon[91136]: pgmap v11081: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:26 smithi146 ceph-mon[103278]: pgmap v11082: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:26.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:26 smithi078 ceph-mon[99959]: pgmap v11082: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:26.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:26 smithi078 ceph-mon[91136]: pgmap v11082: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:28 smithi146 ceph-mon[103278]: pgmap v11083: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:28.393 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:28 smithi078 ceph-mon[99959]: pgmap v11083: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:28.393 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:28 smithi078 ceph-mon[91136]: pgmap v11083: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:29 smithi146 ceph-mon[103278]: pgmap v11084: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[91136]: pgmap v11084: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[99959]: pgmap v11084: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:29:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:29:31.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:31 smithi078 ceph-mon[99959]: pgmap v11085: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:31.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:31 smithi078 ceph-mon[91136]: pgmap v11085: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:31 smithi146 ceph-mon[103278]: pgmap v11085: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:31.708Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:31.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:29:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:31.709Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:33 smithi146 ceph-mon[103278]: pgmap v11086: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[99959]: pgmap v11086: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[91136]: pgmap v11086: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:33.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:35 smithi146 ceph-mon[103278]: pgmap v11087: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:35.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:35 smithi078 ceph-mon[99959]: pgmap v11087: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:35 smithi078 ceph-mon[91136]: pgmap v11087: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:37 smithi146 ceph-mon[103278]: pgmap v11088: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:37 smithi078 ceph-mon[99959]: pgmap v11088: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:37 smithi078 ceph-mon[91136]: pgmap v11088: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:39.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:39 smithi146 ceph-mon[103278]: pgmap v11089: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:39.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:39 smithi078 ceph-mon[91136]: pgmap v11089: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:39 smithi078 ceph-mon[99959]: pgmap v11089: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:29:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:29:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:41 smithi146 ceph-mon[103278]: pgmap v11090: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:41.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:41 smithi078 ceph-mon[91136]: pgmap v11090: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:41 smithi078 ceph-mon[99959]: pgmap v11090: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:41.709Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:41.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:29:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:41.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:43.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:43 smithi146 ceph-mon[103278]: pgmap v11091: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:43.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:43 smithi078 ceph-mon[91136]: pgmap v11091: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:43 smithi078 ceph-mon[99959]: pgmap v11091: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:44.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:45.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:45 smithi146 ceph-mon[103278]: pgmap v11092: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:45.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:45 smithi078 ceph-mon[91136]: pgmap v11092: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:45 smithi078 ceph-mon[99959]: pgmap v11092: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:47 smithi146 ceph-mon[103278]: pgmap v11093: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:47 smithi078 ceph-mon[91136]: pgmap v11093: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:47 smithi078 ceph-mon[99959]: pgmap v11093: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:49 smithi146 ceph-mon[103278]: pgmap v11094: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:49.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[91136]: pgmap v11094: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[99959]: pgmap v11094: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:29:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:29:51.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:51 smithi078 ceph-mon[99959]: pgmap v11095: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:51 smithi078 ceph-mon[91136]: pgmap v11095: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:51.710Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:51.710Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:29:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:29:51.711Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:29:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:51 smithi146 ceph-mon[103278]: pgmap v11095: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:52.266 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 6h - - 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 6h - - 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 4s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 19s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 4s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 19s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 4s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 4s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 19s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 4s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 4s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 19s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 4s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 4s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 4s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 4s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 19s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 19s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 19s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 19s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 19s ago 6h 73.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 4s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 4s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:29:53.134 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:29:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:29:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:29:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:29:53.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:29:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:29:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:53.736 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:29:53.736 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:53.736 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='client.89126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: pgmap v11096: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='client.69213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='client.69219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1064786049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='client.89126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: pgmap v11096: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='client.69213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='client.69219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1064786049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='client.89126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: pgmap v11096: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='client.69213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='client.69219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1064786049' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:29:54.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:54 smithi078 ceph-mon[99959]: from='client.89150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:54.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/354485964' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:54.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:54 smithi078 ceph-mon[91136]: from='client.89150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:54.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/354485964' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:55.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:54 smithi146 ceph-mon[103278]: from='client.89150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:29:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/354485964' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:29:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:55 smithi078 ceph-mon[91136]: pgmap v11097: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:55 smithi078 ceph-mon[99959]: pgmap v11097: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:55 smithi146 ceph-mon[103278]: pgmap v11097: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:57 smithi078 ceph-mon[91136]: pgmap v11098: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:57 smithi078 ceph-mon[99959]: pgmap v11098: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:57 smithi146 ceph-mon[103278]: pgmap v11098: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:29:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:59 smithi078 ceph-mon[91136]: pgmap v11099: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:29:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:59 smithi078 ceph-mon[99959]: pgmap v11099: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:29:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:29:59.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:29:59.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:59 smithi146 ceph-mon[103278]: pgmap v11099: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:29:59.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:29:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T02:30:00.000+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:00.990 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:30:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:30:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:01 smithi146 ceph-mon[103278]: pgmap v11100: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:01 smithi078 ceph-mon[91136]: pgmap v11100: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:01 smithi078 ceph-mon[99959]: pgmap v11100: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:01.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:01.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:01.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:03 smithi146 ceph-mon[103278]: pgmap v11101: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:03 smithi078 ceph-mon[99959]: pgmap v11101: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:03 smithi078 ceph-mon[91136]: pgmap v11101: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:05 smithi146 ceph-mon[103278]: pgmap v11102: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:06.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:05 smithi078 ceph-mon[99959]: pgmap v11102: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:05 smithi078 ceph-mon[91136]: pgmap v11102: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:07.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:07 smithi146 ceph-mon[103278]: pgmap v11103: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:07 smithi078 ceph-mon[91136]: pgmap v11103: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:07 smithi078 ceph-mon[99959]: pgmap v11103: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:09.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:09.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:09 smithi146 ceph-mon[103278]: pgmap v11104: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:09.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:09.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:09.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[91136]: pgmap v11104: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[99959]: pgmap v11104: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:30:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:30:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:11 smithi146 ceph-mon[103278]: pgmap v11105: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:12.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:11 smithi078 ceph-mon[99959]: pgmap v11105: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:11 smithi078 ceph-mon[91136]: pgmap v11105: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:11.711Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:11.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:11.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:13 smithi146 ceph-mon[103278]: pgmap v11106: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:13.924 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[99959]: pgmap v11106: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:13.924 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.924 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.924 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.924 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:13.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[91136]: pgmap v11106: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:13.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:13.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:15 smithi146 ceph-mon[103278]: pgmap v11107: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:16.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:15 smithi078 ceph-mon[91136]: pgmap v11107: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:15 smithi078 ceph-mon[99959]: pgmap v11107: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:17 smithi146 ceph-mon[103278]: pgmap v11108: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:17 smithi078 ceph-mon[91136]: pgmap v11108: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:17 smithi078 ceph-mon[99959]: pgmap v11108: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:19 smithi146 ceph-mon[103278]: pgmap v11109: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:19 smithi078 ceph-mon[91136]: pgmap v11109: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:19 smithi078 ceph-mon[99959]: pgmap v11109: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:30:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:30:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:21 smithi078 ceph-mon[99959]: pgmap v11110: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:21.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:21 smithi078 ceph-mon[91136]: pgmap v11110: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:21.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:21.712Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:21.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:21 smithi146 ceph-mon[103278]: pgmap v11110: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:23 smithi146 ceph-mon[103278]: pgmap v11111: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:24.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:23 smithi078 ceph-mon[99959]: pgmap v11111: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:23 smithi078 ceph-mon[91136]: pgmap v11111: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:24.045 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 6h - - 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 6h - - 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 15s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 11s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 15s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 11s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 15s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 15s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 11s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 15s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:30:24.573 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 15s ago 6h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 11s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 15s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 15s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 15s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 15s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 11s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 11s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 11s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (5h) 11s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 11s ago 6h 73.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (5h) 15s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (5h) 15s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:30:24.574 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:30:24.883 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:30:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:30:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:30:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:30:24.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:30:24.884 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:30:24.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:24 smithi078 ceph-mon[99959]: from='client.89162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:24.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:24 smithi078 ceph-mon[91136]: from='client.89162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:25.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:24 smithi146 ceph-mon[103278]: from='client.89162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:30:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:30:25.152 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:30:25.464 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:25.465 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:25.465 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:25.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[99959]: from='client.69249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[99959]: pgmap v11112: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[99959]: from='client.69255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1683259101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/369985911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[91136]: from='client.69249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[91136]: pgmap v11112: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[91136]: from='client.69255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1683259101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/369985911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:25 smithi146 ceph-mon[103278]: from='client.69249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:25 smithi146 ceph-mon[103278]: pgmap v11112: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:25 smithi146 ceph-mon[103278]: from='client.69255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1683259101' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/369985911' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:26 smithi146 ceph-mon[103278]: from='client.89186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:26 smithi078 ceph-mon[99959]: from='client.89186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:26 smithi078 ceph-mon[91136]: from='client.89186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:27 smithi146 ceph-mon[103278]: pgmap v11113: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:27 smithi078 ceph-mon[99959]: pgmap v11113: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:27 smithi078 ceph-mon[91136]: pgmap v11113: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:29.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:29.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:29 smithi146 ceph-mon[103278]: pgmap v11114: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:29.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:29.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:29.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[99959]: pgmap v11114: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[91136]: pgmap v11114: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:30:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:30:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:31 smithi078 ceph-mon[91136]: pgmap v11115: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:31 smithi078 ceph-mon[99959]: pgmap v11115: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:31.712Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:31.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:31.713Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:31 smithi146 ceph-mon[103278]: pgmap v11115: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:33 smithi146 ceph-mon[103278]: pgmap v11116: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[91136]: pgmap v11116: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[99959]: pgmap v11116: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:36.046 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:35 smithi078 ceph-mon[99959]: pgmap v11117: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:36.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:35 smithi078 ceph-mon[91136]: pgmap v11117: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:35 smithi146 ceph-mon[103278]: pgmap v11117: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:37 smithi146 ceph-mon[103278]: pgmap v11118: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:37 smithi078 ceph-mon[91136]: pgmap v11118: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:37 smithi078 ceph-mon[99959]: pgmap v11118: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:39 smithi146 ceph-mon[103278]: pgmap v11119: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:39 smithi078 ceph-mon[91136]: pgmap v11119: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:39 smithi078 ceph-mon[99959]: pgmap v11119: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:30:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:30:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:41 smithi078 ceph-mon[99959]: pgmap v11120: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:41 smithi078 ceph-mon[91136]: pgmap v11120: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:41.713Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:41.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:41.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:41 smithi146 ceph-mon[103278]: pgmap v11120: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:43 smithi146 ceph-mon[103278]: pgmap v11121: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:43 smithi078 ceph-mon[91136]: pgmap v11121: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:44.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:43 smithi078 ceph-mon[99959]: pgmap v11121: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:45 smithi146 ceph-mon[103278]: pgmap v11122: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:46.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:45 smithi078 ceph-mon[91136]: pgmap v11122: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:45 smithi078 ceph-mon[99959]: pgmap v11122: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:47 smithi146 ceph-mon[103278]: pgmap v11123: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:47 smithi078 ceph-mon[91136]: pgmap v11123: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:47 smithi078 ceph-mon[99959]: pgmap v11123: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:49 smithi146 ceph-mon[103278]: pgmap v11124: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[91136]: pgmap v11124: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[99959]: pgmap v11124: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:30:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:30:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:51 smithi078 ceph-mon[91136]: pgmap v11125: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:51 smithi078 ceph-mon[99959]: pgmap v11125: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:51.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:51.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:30:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:30:51.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:30:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:51 smithi146 ceph-mon[103278]: pgmap v11125: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:53 smithi146 ceph-mon[103278]: pgmap v11126: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[99959]: pgmap v11126: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[91136]: pgmap v11126: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:30:55.768 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:30:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:55 smithi078 ceph-mon[91136]: pgmap v11127: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:30:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:55 smithi078 ceph-mon[99959]: pgmap v11127: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:30:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:55 smithi146 ceph-mon[103278]: pgmap v11127: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 6h - - 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 7s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 2s ago 6h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 7s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:30:56.305 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 7s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 7s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 7s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 7s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 2s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 7s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 7s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 7s ago 6h 82.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 7s ago 6h 99.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 6h 113M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 6h 89.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 6h 95.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:30:56.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 6h 91.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:30:56.307 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 6h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:30:56.307 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 7s ago 6h 89.3M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:30:56.307 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (5h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:30:56.307 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 7s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:30:56.307 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (5h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:30:56.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:30:56.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:30:56.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:30:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:30:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:30:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:30:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:30:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:30:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:30:56.907 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:30:57.176 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:56 smithi078 ceph-mon[99959]: from='client.69279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:57.177 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:56 smithi078 ceph-mon[99959]: from='client.69285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:57.177 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1172947026' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:57.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:56 smithi078 ceph-mon[91136]: from='client.69279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:57.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:56 smithi078 ceph-mon[91136]: from='client.69285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:57.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1172947026' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:57.236 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:30:57.236 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:30:57.236 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:30:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:56 smithi146 ceph-mon[103278]: from='client.69279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:57.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:56 smithi146 ceph-mon[103278]: from='client.69285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1172947026' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[91136]: from='client.89210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[91136]: pgmap v11128: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[91136]: from='client.89222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4250257454' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[99959]: from='client.89210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[99959]: pgmap v11128: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T02:30:58.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[99959]: from='client.89222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:58.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4250257454' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:57 smithi146 ceph-mon[103278]: from='client.89210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:57 smithi146 ceph-mon[103278]: pgmap v11128: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T02:30:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:57 smithi146 ceph-mon[103278]: from='client.89222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:30:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4250257454' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:30:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:30:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:30:59 smithi146 ceph-mon[103278]: pgmap v11129: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:31:00.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:30:59 smithi078 ceph-mon[91136]: pgmap v11129: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:31:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:30:59 smithi078 ceph-mon[99959]: pgmap v11129: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T02:31:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:31:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:31:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:01 smithi078 ceph-mon[99959]: pgmap v11130: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T02:31:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:01 smithi078 ceph-mon[91136]: pgmap v11130: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T02:31:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:01.714Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:01.714Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:01.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:01 smithi146 ceph-mon[103278]: pgmap v11130: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T02:31:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:03 smithi146 ceph-mon[103278]: pgmap v11131: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T02:31:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:03 smithi078 ceph-mon[91136]: pgmap v11131: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T02:31:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:03 smithi078 ceph-mon[99959]: pgmap v11131: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T02:31:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:05 smithi146 ceph-mon[103278]: pgmap v11132: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-21T02:31:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:05 smithi078 ceph-mon[99959]: pgmap v11132: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-21T02:31:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:05 smithi078 ceph-mon[91136]: pgmap v11132: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-21T02:31:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:07 smithi146 ceph-mon[103278]: pgmap v11133: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T02:31:08.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:07 smithi078 ceph-mon[91136]: pgmap v11133: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T02:31:08.292 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:07 smithi078 ceph-mon[99959]: pgmap v11133: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T02:31:10.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:10.208 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:10 smithi146 ceph-mon[103278]: pgmap v11134: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T02:31:10.208 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.208 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.208 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[99959]: pgmap v11134: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[91136]: pgmap v11134: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:10.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:31:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:31:12.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:11.715Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:11.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:11.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:12.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:12 smithi146 ceph-mon[103278]: pgmap v11135: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:31:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:12 smithi078 ceph-mon[99959]: pgmap v11135: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:31:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:12 smithi078 ceph-mon[91136]: pgmap v11135: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:31:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:14 smithi146 ceph-mon[103278]: pgmap v11136: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:31:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[99959]: pgmap v11136: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[91136]: pgmap v11136: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:16.366 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:16 smithi078 ceph-mon[99959]: pgmap v11137: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-21T02:31:16.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:16 smithi078 ceph-mon[91136]: pgmap v11137: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-21T02:31:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:16 smithi146 ceph-mon[103278]: pgmap v11137: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-09-21T02:31:18.175 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:18 smithi078 ceph-mon[99959]: pgmap v11138: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 177 op/s 2024-09-21T02:31:18.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:18 smithi078 ceph-mon[91136]: pgmap v11138: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 177 op/s 2024-09-21T02:31:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:18 smithi146 ceph-mon[103278]: pgmap v11138: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 177 op/s 2024-09-21T02:31:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:20 smithi146 ceph-mon[103278]: pgmap v11139: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-21T02:31:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:20 smithi078 ceph-mon[99959]: pgmap v11139: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-21T02:31:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:20 smithi078 ceph-mon[91136]: pgmap v11139: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-21T02:31:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:31:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:31:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:21.715Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:21.715Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:21.716Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:22 smithi146 ceph-mon[103278]: pgmap v11140: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T02:31:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:22 smithi078 ceph-mon[91136]: pgmap v11140: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T02:31:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:22 smithi078 ceph-mon[99959]: pgmap v11140: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T02:31:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:24 smithi146 ceph-mon[103278]: pgmap v11141: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-21T02:31:24.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:24 smithi078 ceph-mon[91136]: pgmap v11141: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-21T02:31:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:24 smithi078 ceph-mon[99959]: pgmap v11141: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 88 op/s 2024-09-21T02:31:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:26 smithi146 ceph-mon[103278]: pgmap v11142: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T02:31:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:26 smithi078 ceph-mon[91136]: pgmap v11142: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T02:31:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:26 smithi078 ceph-mon[99959]: pgmap v11142: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T02:31:27.530 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 6h - - 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 14s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 14s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 19s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:31:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 14s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 19s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 14s ago 6h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 19s ago 6h 93.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 19s ago 6h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 19s ago 6h 82.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 19s ago 6h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 14s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 14s ago 6h 90.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 14s ago 6h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 14s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 14s ago 6h 71.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 19s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 19s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:31:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:31:28.366 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:31:28.366 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:31:28.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:31:28.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:31:28.368 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:31:28.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:28 smithi078 ceph-mon[91136]: pgmap v11143: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:31:28.368 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:28 smithi078 ceph-mon[99959]: pgmap v11143: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:31:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:28 smithi146 ceph-mon[103278]: pgmap v11143: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:31:28.637 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:31:28.638 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:31:28.949 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:31:28.949 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:31:28.949 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:31:28.949 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:31:28.949 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[91136]: from='client.89234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[91136]: from='client.89240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[91136]: from='client.69327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/900840298' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/970652350' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[99959]: from='client.89234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[99959]: from='client.89240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[99959]: from='client.69327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/900840298' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/970652350' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:29 smithi146 ceph-mon[103278]: from='client.89234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:29 smithi146 ceph-mon[103278]: from='client.89240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:29 smithi146 ceph-mon[103278]: from='client.69327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/900840298' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:31:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/970652350' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:31:30.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:30.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:30 smithi146 ceph-mon[103278]: pgmap v11144: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:31:30.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:30 smithi146 ceph-mon[103278]: from='client.89258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:30.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[91136]: pgmap v11144: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[91136]: from='client.89258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[99959]: pgmap v11144: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-09-21T02:31:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[99959]: from='client.89258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:31:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:30.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:31:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:31:31.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:31 smithi146 ceph-mon[103278]: pgmap v11145: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:31:31.717 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:31 smithi078 ceph-mon[91136]: pgmap v11145: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:31:31.717 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:31 smithi078 ceph-mon[99959]: pgmap v11145: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:31:32.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:31.716Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:31.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:31.717Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:33 smithi146 ceph-mon[103278]: pgmap v11146: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:31:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:33 smithi078 ceph-mon[91136]: pgmap v11146: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:31:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:33 smithi078 ceph-mon[99959]: pgmap v11146: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:31:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:36 smithi146 ceph-mon[103278]: pgmap v11147: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:31:36.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:36 smithi078 ceph-mon[91136]: pgmap v11147: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:31:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:36 smithi078 ceph-mon[99959]: pgmap v11147: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:31:38.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:38 smithi078 ceph-mon[99959]: pgmap v11148: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:31:38.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:38 smithi078 ceph-mon[91136]: pgmap v11148: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:31:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:38 smithi146 ceph-mon[103278]: pgmap v11148: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:31:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:40 smithi146 ceph-mon[103278]: pgmap v11149: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:40 smithi078 ceph-mon[99959]: pgmap v11149: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:40 smithi078 ceph-mon[91136]: pgmap v11149: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:31:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:31:42.014 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:41.717Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:42.015 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:41.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:42.015 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:41.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:42 smithi146 ceph-mon[103278]: pgmap v11150: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:42 smithi078 ceph-mon[91136]: pgmap v11150: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:42 smithi078 ceph-mon[99959]: pgmap v11150: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:44 smithi078 ceph-mon[99959]: pgmap v11151: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:44 smithi078 ceph-mon[91136]: pgmap v11151: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:44.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:44 smithi146 ceph-mon[103278]: pgmap v11151: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:46 smithi078 ceph-mon[99959]: pgmap v11152: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:46 smithi078 ceph-mon[91136]: pgmap v11152: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:46 smithi146 ceph-mon[103278]: pgmap v11152: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:48 smithi078 ceph-mon[91136]: pgmap v11153: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:48 smithi078 ceph-mon[99959]: pgmap v11153: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:48 smithi146 ceph-mon[103278]: pgmap v11153: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[99959]: pgmap v11154: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[91136]: pgmap v11154: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:31:50.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:50 smithi146 ceph-mon[103278]: pgmap v11154: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:50.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:50.519 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:31:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:31:52.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:51.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:52.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:51.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:31:52.030 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:31:51.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:31:52.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:52 smithi146 ceph-mon[103278]: pgmap v11155: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:52.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:52 smithi078 ceph-mon[91136]: pgmap v11155: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:52 smithi078 ceph-mon[99959]: pgmap v11155: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:54 smithi078 ceph-mon[91136]: pgmap v11156: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:54 smithi078 ceph-mon[99959]: pgmap v11156: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:54 smithi146 ceph-mon[103278]: pgmap v11156: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:31:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:56 smithi146 ceph-mon[103278]: pgmap v11157: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:56 smithi078 ceph-mon[91136]: pgmap v11157: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:56 smithi078 ceph-mon[99959]: pgmap v11157: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:31:58.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:58 smithi078 ceph-mon[91136]: pgmap v11158: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:58.441 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:58 smithi078 ceph-mon[99959]: pgmap v11158: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:58 smithi146 ceph-mon[103278]: pgmap v11158: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:31:59.266 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:31:59.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:59.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:59.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:31:59.847 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:31:59.847 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 6h - - 2024-09-21T02:31:59.847 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 6h - - 2024-09-21T02:31:59.847 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 10s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 5s ago 6h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 10s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 5s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 10s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 10s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 5s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 10s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 10s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 5s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 10s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 10s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 10s ago 6h 82.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 10s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:31:59.848 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 5s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 5s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 5s ago 6h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 5s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 5s ago 6h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 10s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 10s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:31:59.849 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:32:00.437 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:32:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:32:00.438 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:32:00.438 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:32:00.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:00.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:00 smithi078 ceph-mon[91136]: pgmap v11159: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:00.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:00.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:00 smithi078 ceph-mon[99959]: pgmap v11159: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:00.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:00 smithi146 ceph-mon[103278]: pgmap v11159: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:00.748 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:32:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:32:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:01 smithi146 ceph-mon[103278]: from='client.69351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:01 smithi146 ceph-mon[103278]: from='client.89276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:01 smithi146 ceph-mon[103278]: from='client.89282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/630983006' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/850793041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[91136]: from='client.69351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[91136]: from='client.89276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[91136]: from='client.89282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/630983006' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/850793041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[99959]: from='client.69351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[99959]: from='client.89276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[99959]: from='client.89282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/630983006' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/850793041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:01.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:01.718Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:01.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:02 smithi146 ceph-mon[103278]: from='client.89294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:02 smithi146 ceph-mon[103278]: pgmap v11160: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:02 smithi078 ceph-mon[99959]: from='client.89294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:02 smithi078 ceph-mon[99959]: pgmap v11160: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:02 smithi078 ceph-mon[91136]: from='client.89294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:02 smithi078 ceph-mon[91136]: pgmap v11160: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:03 smithi146 ceph-mon[103278]: pgmap v11161: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:03 smithi078 ceph-mon[91136]: pgmap v11161: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:03 smithi078 ceph-mon[99959]: pgmap v11161: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:05.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:05 smithi146 ceph-mon[103278]: pgmap v11162: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:05.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:05 smithi078 ceph-mon[91136]: pgmap v11162: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:05.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:05 smithi078 ceph-mon[99959]: pgmap v11162: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:07 smithi146 ceph-mon[103278]: pgmap v11163: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:07 smithi078 ceph-mon[99959]: pgmap v11163: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:07 smithi078 ceph-mon[91136]: pgmap v11163: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:09 smithi146 ceph-mon[103278]: pgmap v11164: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[91136]: pgmap v11164: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[99959]: pgmap v11164: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:32:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:32:11.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:11 smithi146 ceph-mon[103278]: pgmap v11165: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:11 smithi078 ceph-mon[91136]: pgmap v11165: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:11 smithi078 ceph-mon[99959]: pgmap v11165: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:11.718Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:11.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:11.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:11.719Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:13 smithi146 ceph-mon[103278]: pgmap v11166: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:13 smithi078 ceph-mon[91136]: pgmap v11166: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:13 smithi078 ceph-mon[99959]: pgmap v11166: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:15.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:15 smithi078 ceph-mon[91136]: pgmap v11167: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:15 smithi078 ceph-mon[99959]: pgmap v11167: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:15 smithi146 ceph-mon[103278]: pgmap v11167: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:17 smithi078 ceph-mon[91136]: pgmap v11168: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:17 smithi078 ceph-mon[99959]: pgmap v11168: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:17 smithi146 ceph-mon[103278]: pgmap v11168: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:19.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:19 smithi078 ceph-mon[91136]: pgmap v11169: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:19.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:19.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:19 smithi078 ceph-mon[99959]: pgmap v11169: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:19.776 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:19 smithi146 ceph-mon[103278]: pgmap v11169: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:20.940 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:32:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:32:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:21 smithi078 ceph-mon[99959]: pgmap v11170: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:21 smithi078 ceph-mon[91136]: pgmap v11170: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:21.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:21.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:21.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:21 smithi146 ceph-mon[103278]: pgmap v11170: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:23 smithi078 ceph-mon[91136]: pgmap v11171: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:23 smithi078 ceph-mon[99959]: pgmap v11171: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:23.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:23 smithi146 ceph-mon[103278]: pgmap v11171: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:25 smithi078 ceph-mon[91136]: pgmap v11172: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:25 smithi078 ceph-mon[99959]: pgmap v11172: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:25 smithi146 ceph-mon[103278]: pgmap v11172: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:27 smithi146 ceph-mon[103278]: pgmap v11173: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:27 smithi078 ceph-mon[91136]: pgmap v11173: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:27 smithi078 ceph-mon[99959]: pgmap v11173: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:29.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:29 smithi146 ceph-mon[103278]: pgmap v11174: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:29.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:29.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:29 smithi078 ceph-mon[91136]: pgmap v11174: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:29 smithi078 ceph-mon[99959]: pgmap v11174: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:30.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:30.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:30.831 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:32:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:32:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:31.037 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 6h - - 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 17s ago 6h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:32:31.562 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 17s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 2s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 2s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 17s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 2s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 17s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 2s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 2s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 2s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 17s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 17s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 17s ago 6h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 17s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:32:31.563 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 17s ago 6h 75.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:32:31.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 2s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:32:31.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:32:31.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 2s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:32:31.564 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:32:31.873 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-mon[99959]: pgmap v11175: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:31.873 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-mon[99959]: from='client.89306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:31.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-mon[91136]: pgmap v11175: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:31.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-mon[91136]: from='client.89306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:31.873 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:32:31.873 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:32:31.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:32:31.875 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:32:31.875 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:31.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:31.875 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:31.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:31.875 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:31.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:31.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:31 smithi146 ceph-mon[103278]: pgmap v11175: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:31.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:31 smithi146 ceph-mon[103278]: from='client.89306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.141 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:32:32.141 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:32:32.142 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:32:32.448 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:32:32.449 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:32.449 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:32:32.449 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:32.449 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[99959]: from='client.89312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[99959]: from='client.89318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3724826911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[99959]: from='client.69411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1778214363' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[91136]: from='client.89312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[91136]: from='client.89318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3724826911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[91136]: from='client.69411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1778214363' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:33.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:32 smithi146 ceph-mon[103278]: from='client.89312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:33.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:32 smithi146 ceph-mon[103278]: from='client.89318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:33.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3724826911' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:32:33.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:32 smithi146 ceph-mon[103278]: from='client.69411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:32:33.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1778214363' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:32:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:33 smithi078 ceph-mon[99959]: pgmap v11176: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:33 smithi078 ceph-mon[91136]: pgmap v11176: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:33.813 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:33 smithi146 ceph-mon[103278]: pgmap v11176: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:35 smithi146 ceph-mon[103278]: pgmap v11177: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[99959]: pgmap v11177: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:35 smithi078 ceph-mon[91136]: pgmap v11177: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:37 smithi146 ceph-mon[103278]: pgmap v11178: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:37 smithi078 ceph-mon[99959]: pgmap v11178: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:37 smithi078 ceph-mon[91136]: pgmap v11178: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:39.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:39 smithi078 ceph-mon[99959]: pgmap v11179: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:39.621 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:39.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:39 smithi078 ceph-mon[91136]: pgmap v11179: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:39.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:39 smithi146 ceph-mon[103278]: pgmap v11179: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:40.918 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:32:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:32:41.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:41 smithi078 ceph-mon[91136]: pgmap v11180: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:41.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:41 smithi078 ceph-mon[99959]: pgmap v11180: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:41.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:41 smithi146 ceph-mon[103278]: pgmap v11180: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:42.034 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:41.719Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:41.720Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:41.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:43 smithi078 ceph-mon[91136]: pgmap v11181: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:43 smithi078 ceph-mon[99959]: pgmap v11181: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:43 smithi146 ceph-mon[103278]: pgmap v11181: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:45.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:45 smithi078 ceph-mon[91136]: pgmap v11182: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:45 smithi078 ceph-mon[99959]: pgmap v11182: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:45 smithi146 ceph-mon[103278]: pgmap v11182: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:47.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:47 smithi078 ceph-mon[99959]: pgmap v11183: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:47 smithi078 ceph-mon[91136]: pgmap v11183: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:47 smithi146 ceph-mon[103278]: pgmap v11183: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:49 smithi146 ceph-mon[103278]: pgmap v11184: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:49 smithi078 ceph-mon[99959]: pgmap v11184: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:50.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:49 smithi078 ceph-mon[91136]: pgmap v11184: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:32:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:32:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:51.959 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:51 smithi078 ceph-mon[99959]: pgmap v11185: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:51.959 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:51 smithi078 ceph-mon[91136]: pgmap v11185: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:51.959 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:51.720Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:51.960 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:51.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:32:51.960 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:32:51.721Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:32:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:51 smithi146 ceph-mon[103278]: pgmap v11185: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:53.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:53 smithi146 ceph-mon[103278]: pgmap v11186: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:53 smithi078 ceph-mon[91136]: pgmap v11186: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:53 smithi078 ceph-mon[99959]: pgmap v11186: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:55.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:55 smithi146 ceph-mon[103278]: pgmap v11187: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[91136]: pgmap v11187: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[99959]: pgmap v11187: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:32:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:57 smithi146 ceph-mon[103278]: pgmap v11188: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:57 smithi078 ceph-mon[99959]: pgmap v11188: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:57 smithi078 ceph-mon[91136]: pgmap v11188: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:32:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:59 smithi146 ceph-mon[103278]: pgmap v11189: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:32:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:59 smithi078 ceph-mon[91136]: pgmap v11189: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:59.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:32:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:32:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:59 smithi078 ceph-mon[99959]: pgmap v11189: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:32:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:32:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:32:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:33:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:33:01.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:01 smithi078 ceph-mon[99959]: pgmap v11190: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:01.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:01 smithi078 ceph-mon[91136]: pgmap v11190: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:01 smithi146 ceph-mon[103278]: pgmap v11190: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:01.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:01.721Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:01.971 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:01.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:01.972 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:01.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:02.741 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:33:03.269 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:33:03.269 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 6h - - 2024-09-21T02:33:03.269 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 6h - - 2024-09-21T02:33:03.269 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 13s ago 6h 29.3M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 8s ago 6h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 13s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 8s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 13s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 13s ago 6h 545M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 8s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 13s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 13s ago 6h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 8s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 13s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 13s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 13s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 13s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 8s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:33:03.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 8s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 8s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 8s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 8s ago 6h 76.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 13s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 13s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:33:03.271 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 8s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:33:03.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:03 smithi078 ceph-mon[91136]: pgmap v11191: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:03 smithi078 ceph-mon[91136]: from='client.69423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:03 smithi078 ceph-mon[91136]: from='client.69429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:03 smithi078 ceph-mon[99959]: pgmap v11191: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:03 smithi078 ceph-mon[99959]: from='client.69423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:03 smithi078 ceph-mon[99959]: from='client.69429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:33:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:33:03.585 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:33:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:03 smithi146 ceph-mon[103278]: pgmap v11191: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:03 smithi146 ceph-mon[103278]: from='client.69423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:03 smithi146 ceph-mon[103278]: from='client.69429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:33:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:33:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:33:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:33:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:33:03.853 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:33:04.161 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:33:04.161 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:33:04.161 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:33:04.162 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:33:04.162 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:33:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:04 smithi146 ceph-mon[103278]: from='client.89354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:04.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3445575994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:04 smithi146 ceph-mon[103278]: from='client.89366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3578883672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[91136]: from='client.89354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3445575994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[91136]: from='client.89366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3578883672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[99959]: from='client.89354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3445575994' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[99959]: from='client.89366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3578883672' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:05 smithi078 ceph-mon[91136]: pgmap v11192: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:05 smithi078 ceph-mon[99959]: pgmap v11192: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:05 smithi146 ceph-mon[103278]: pgmap v11192: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:07 smithi078 ceph-mon[91136]: pgmap v11193: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:07 smithi078 ceph-mon[99959]: pgmap v11193: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:07 smithi146 ceph-mon[103278]: pgmap v11193: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:09 smithi078 ceph-mon[99959]: pgmap v11194: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:09 smithi078 ceph-mon[91136]: pgmap v11194: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:09 smithi146 ceph-mon[103278]: pgmap v11194: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:33:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:11 smithi078 ceph-mon[91136]: pgmap v11195: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:11 smithi078 ceph-mon[99959]: pgmap v11195: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:11.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:11.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:11.722Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:11 smithi146 ceph-mon[103278]: pgmap v11195: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:14.137 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:13 smithi146 ceph-mon[103278]: pgmap v11196: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:14.137 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:13 smithi078 ceph-mon[99959]: pgmap v11196: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:13 smithi078 ceph-mon[91136]: pgmap v11196: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:15 smithi146 ceph-mon[103278]: pgmap v11197: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[91136]: pgmap v11197: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[99959]: pgmap v11197: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:17 smithi146 ceph-mon[103278]: pgmap v11198: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:18.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:17 smithi078 ceph-mon[91136]: pgmap v11198: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:17 smithi078 ceph-mon[99959]: pgmap v11198: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:19 smithi146 ceph-mon[103278]: pgmap v11199: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:19 smithi078 ceph-mon[91136]: pgmap v11199: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:19 smithi078 ceph-mon[99959]: pgmap v11199: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:20.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:33:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T02:33:21.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:21 smithi078 ceph-mon[91136]: pgmap v11200: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:21.899 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:21 smithi078 ceph-mon[99959]: pgmap v11200: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:21.899 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:21.722Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:21.899 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:21.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:21.899 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:21.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:21 smithi146 ceph-mon[103278]: pgmap v11200: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:23 smithi078 ceph-mon[91136]: pgmap v11201: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:23 smithi078 ceph-mon[99959]: pgmap v11201: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:23 smithi146 ceph-mon[103278]: pgmap v11201: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:25 smithi146 ceph-mon[103278]: pgmap v11202: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:26.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:25 smithi078 ceph-mon[91136]: pgmap v11202: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:25 smithi078 ceph-mon[99959]: pgmap v11202: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:27 smithi146 ceph-mon[103278]: pgmap v11203: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:28.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:27 smithi078 ceph-mon[91136]: pgmap v11203: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:27 smithi078 ceph-mon[99959]: pgmap v11203: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:29 smithi146 ceph-mon[103278]: pgmap v11204: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:29.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:29 smithi078 ceph-mon[99959]: pgmap v11204: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:29.948 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:29.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:29 smithi078 ceph-mon[91136]: pgmap v11204: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:29.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:30.935 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:33:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:33:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:31 smithi078 ceph-mon[91136]: pgmap v11205: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:31 smithi078 ceph-mon[99959]: pgmap v11205: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:31.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:31.723Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:31.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:31 smithi146 ceph-mon[103278]: pgmap v11205: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:33 smithi146 ceph-mon[103278]: pgmap v11206: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:34.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:33 smithi078 ceph-mon[91136]: pgmap v11206: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:33 smithi078 ceph-mon[99959]: pgmap v11206: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:34.457 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:33:34.985 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:33:34.985 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 6h - - 2024-09-21T02:33:34.985 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 6h - - 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 5s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 0s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 5s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 0s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 5s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 5s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 0s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 5s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 5s ago 6h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 0s ago 6h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 5s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 5s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:33:34.986 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 5s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 0s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 0s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 0s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 0s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 0s ago 6h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 5s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 5s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:33:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:33:35.299 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:33:35.574 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:33:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:33:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:33:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:33:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:33:35.574 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:33:35.575 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:33:35.885 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:33:35.885 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:33:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:33:35.885 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:33:35.885 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:33:35.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='client.89378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:35.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: pgmap v11207: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:35.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='client.89384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='client.89390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3188969220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='client.89378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: pgmap v11207: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='client.89384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='client.89390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:35.886 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3188969220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='client.89378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: pgmap v11207: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='client.89384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:36.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='client.89390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:36.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3188969220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:33:37.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:36 smithi146 ceph-mon[103278]: from='client.69483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1742058178' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:37.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:36 smithi078 ceph-mon[99959]: from='client.69483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1742058178' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:36 smithi078 ceph-mon[91136]: from='client.69483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:33:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1742058178' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:33:38.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:37 smithi146 ceph-mon[103278]: pgmap v11208: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:38.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:37 smithi078 ceph-mon[91136]: pgmap v11208: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:37 smithi078 ceph-mon[99959]: pgmap v11208: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:39 smithi146 ceph-mon[103278]: pgmap v11209: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:40.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:39 smithi078 ceph-mon[91136]: pgmap v11209: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:39 smithi078 ceph-mon[99959]: pgmap v11209: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:33:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:33:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:41.723Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:41.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:41.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:41.724Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:42.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:41 smithi078 ceph-mon[91136]: pgmap v11210: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:42.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:41 smithi078 ceph-mon[99959]: pgmap v11210: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:41 smithi146 ceph-mon[103278]: pgmap v11210: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:44.058 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:43 smithi078 ceph-mon[91136]: pgmap v11211: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:44.059 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:44.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:43 smithi078 ceph-mon[99959]: pgmap v11211: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:44.059 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:43 smithi146 ceph-mon[103278]: pgmap v11211: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:45 smithi146 ceph-mon[103278]: pgmap v11212: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:46.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:45 smithi078 ceph-mon[99959]: pgmap v11212: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:45 smithi078 ceph-mon[91136]: pgmap v11212: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:47 smithi146 ceph-mon[103278]: pgmap v11213: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:47 smithi078 ceph-mon[91136]: pgmap v11213: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:47 smithi078 ceph-mon[99959]: pgmap v11213: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:50.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:49 smithi078 ceph-mon[99959]: pgmap v11214: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:50.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:50.079 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:49 smithi078 ceph-mon[91136]: pgmap v11214: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:50.079 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:49 smithi146 ceph-mon[103278]: pgmap v11214: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:33:51.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:33:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:33:51.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.489 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:51.784 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:51.724Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:51.725Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:33:51.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:33:51.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:33:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:52 smithi146 ceph-mon[103278]: pgmap v11215: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:52.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:52 smithi078 ceph-mon[91136]: pgmap v11215: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:52 smithi078 ceph-mon[99959]: pgmap v11215: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:54.351 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:54 smithi146 ceph-mon[103278]: pgmap v11216: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:54 smithi078 ceph-mon[91136]: pgmap v11216: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:54 smithi078 ceph-mon[99959]: pgmap v11216: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:55 smithi146 ceph-mon[103278]: pgmap v11217: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[91136]: pgmap v11217: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[99959]: pgmap v11217: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:33:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:57 smithi146 ceph-mon[103278]: pgmap v11218: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:58.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:57 smithi078 ceph-mon[99959]: pgmap v11218: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:57 smithi078 ceph-mon[91136]: pgmap v11218: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:33:59.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:33:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:59 smithi146 ceph-mon[103278]: pgmap v11219: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:00.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:33:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:00.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:59 smithi078 ceph-mon[99959]: pgmap v11219: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:33:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:59 smithi078 ceph-mon[91136]: pgmap v11219: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:33:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:34:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:34:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:01 smithi078 ceph-mon[91136]: pgmap v11220: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:01 smithi078 ceph-mon[99959]: pgmap v11220: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:01.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:01.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:01.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:01 smithi146 ceph-mon[103278]: pgmap v11220: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:03 smithi146 ceph-mon[103278]: pgmap v11221: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:03 smithi078 ceph-mon[91136]: pgmap v11221: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:03 smithi078 ceph-mon[99959]: pgmap v11221: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:06.174 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:34:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:05 smithi146 ceph-mon[103278]: pgmap v11222: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:06.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:05 smithi078 ceph-mon[99959]: pgmap v11222: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:05 smithi078 ceph-mon[91136]: pgmap v11222: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:06.701 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:34:06.701 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 6h - - 2024-09-21T02:34:06.701 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 6h - - 2024-09-21T02:34:06.701 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 16s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:34:06.701 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 11s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:34:06.701 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 16s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 11s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 16s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 16s ago 6h 546M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 11s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 16s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 16s ago 6h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 11s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 16s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 16s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 16s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 16s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 11s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:34:06.702 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 11s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 11s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 11s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 11s ago 6h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 16s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 16s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:34:06.703 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 11s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:34:07.013 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:34:07.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:34:07.013 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:34:07.014 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:34:07.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:34:07.015 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:34:07.015 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:06 smithi078 ceph-mon[99959]: from='client.69495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:07.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:06 smithi078 ceph-mon[91136]: from='client.69495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:06 smithi146 ceph-mon[103278]: from='client.69495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:07.282 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:34:07.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:34:07.592 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:34:07.592 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:07.592 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:07 smithi146 ceph-mon[103278]: from='client.89420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:07 smithi146 ceph-mon[103278]: pgmap v11223: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:07 smithi146 ceph-mon[103278]: from='client.89426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1699391081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1793445615' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[91136]: from='client.89420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[91136]: pgmap v11223: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[91136]: from='client.89426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1699391081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1793445615' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[99959]: from='client.89420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[99959]: pgmap v11223: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[99959]: from='client.89426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1699391081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1793445615' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:08 smithi146 ceph-mon[103278]: from='client.69519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:09.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:08 smithi078 ceph-mon[99959]: from='client.69519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:08 smithi078 ceph-mon[91136]: from='client.69519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:10.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:09 smithi078 ceph-mon[99959]: pgmap v11224: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:10.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:10.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:09 smithi078 ceph-mon[91136]: pgmap v11224: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:10.210 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:09 smithi146 ceph-mon[103278]: pgmap v11224: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:11.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:34:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:34:11.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:11 smithi146 ceph-mon[103278]: pgmap v11225: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[91136]: pgmap v11225: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-mon[99959]: pgmap v11225: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:11.725Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:11.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:11.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:13 smithi146 ceph-mon[103278]: pgmap v11226: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:13 smithi078 ceph-mon[99959]: pgmap v11226: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:13 smithi078 ceph-mon[91136]: pgmap v11226: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:14.879 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:16.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:16 smithi146 ceph-mon[103278]: pgmap v11227: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:16.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[91136]: pgmap v11227: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[99959]: pgmap v11227: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:18 smithi146 ceph-mon[103278]: pgmap v11228: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:18 smithi078 ceph-mon[91136]: pgmap v11228: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:18 smithi078 ceph-mon[99959]: pgmap v11228: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:20 smithi146 ceph-mon[103278]: pgmap v11229: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:20.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:20 smithi078 ceph-mon[99959]: pgmap v11229: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:20 smithi078 ceph-mon[91136]: pgmap v11229: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:34:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:34:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:21.726Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:21.726Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:21.785 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:21.727Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:22 smithi146 ceph-mon[103278]: pgmap v11230: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:22.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:22 smithi078 ceph-mon[91136]: pgmap v11230: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:22 smithi078 ceph-mon[99959]: pgmap v11230: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:24 smithi078 ceph-mon[99959]: pgmap v11231: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:24 smithi078 ceph-mon[91136]: pgmap v11231: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:24.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:24 smithi146 ceph-mon[103278]: pgmap v11231: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:26 smithi146 ceph-mon[103278]: pgmap v11232: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 02:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T02:34:25.847775159Z level=info msg="Completed cleanup jobs" duration=11.973898ms 2024-09-21T02:34:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:26 smithi078 ceph-mon[99959]: pgmap v11232: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:26 smithi078 ceph-mon[91136]: pgmap v11232: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:28.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:28 smithi146 ceph-mon[103278]: pgmap v11233: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:28 smithi078 ceph-mon[99959]: pgmap v11233: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:28 smithi078 ceph-mon[91136]: pgmap v11233: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:34:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:30.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:30 smithi078 ceph-mon[99959]: pgmap v11234: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:30.335 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:30.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:30 smithi078 ceph-mon[91136]: pgmap v11234: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:30.335 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:30.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:30 smithi146 ceph-mon[103278]: pgmap v11234: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:30.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:34:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[99959]: pgmap v11235: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-mon[91136]: pgmap v11235: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:31 smithi146 ceph-mon[103278]: pgmap v11235: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:31.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:31.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:31.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:33 smithi146 ceph-mon[103278]: pgmap v11236: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:33 smithi078 ceph-mon[99959]: pgmap v11236: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:33 smithi078 ceph-mon[91136]: pgmap v11236: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:35 smithi146 ceph-mon[103278]: pgmap v11237: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[99959]: pgmap v11237: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[91136]: pgmap v11237: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:37.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:37 smithi146 ceph-mon[103278]: pgmap v11238: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:37.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:37 smithi078 ceph-mon[99959]: pgmap v11238: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:37.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:37 smithi078 ceph-mon[91136]: pgmap v11238: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:37.886 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 6h - - 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 6h - - 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 8s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 3s ago 6h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 8s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:34:38.412 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 3s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 8s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 8s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 3s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 8s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 8s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 3s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 8s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 8s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 8s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 3s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 3s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 3s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:34:38.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 3s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:34:38.414 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 3s ago 6h 77.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:34:38.414 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 8s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:34:38.414 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:34:38.414 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 8s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:34:38.414 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:34:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:34:38.723 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:34:38.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:38 smithi078 ceph-mon[99959]: from='client.89450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:38.723 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:38 smithi078 ceph-mon[99959]: from='client.89456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:38.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:38 smithi078 ceph-mon[91136]: from='client.89450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:38.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:38 smithi078 ceph-mon[91136]: from='client.89456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:38.991 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:34:38.992 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:34:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:38 smithi146 ceph-mon[103278]: from='client.89450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:38 smithi146 ceph-mon[103278]: from='client.89456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:39.300 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:34:39.300 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:34:39.301 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:34:39.951 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:39 smithi146 ceph-mon[103278]: from='client.69543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:39.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:39 smithi146 ceph-mon[103278]: pgmap v11239: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:39.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:39.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/972221032' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:39.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:39 smithi146 ceph-mon[103278]: from='client.69555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:39.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1500913378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[99959]: from='client.69543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[99959]: pgmap v11239: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/972221032' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[99959]: from='client.69555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1500913378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[91136]: from='client.69543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[91136]: pgmap v11239: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/972221032' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:34:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[91136]: from='client.69555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:34:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1500913378' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:34:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:34:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:34:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:41 smithi146 ceph-mon[103278]: pgmap v11240: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:41 smithi078 ceph-mon[91136]: pgmap v11240: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:41 smithi078 ceph-mon[99959]: pgmap v11240: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:41.727Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:41.728Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:41.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:43.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:43 smithi078 ceph-mon[99959]: pgmap v11241: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:43.891 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:43.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:43 smithi078 ceph-mon[91136]: pgmap v11241: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:43.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:43 smithi146 ceph-mon[103278]: pgmap v11241: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:45.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:45 smithi078 ceph-mon[91136]: pgmap v11242: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:45 smithi078 ceph-mon[99959]: pgmap v11242: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:45 smithi146 ceph-mon[103278]: pgmap v11242: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:47 smithi146 ceph-mon[103278]: pgmap v11243: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:48.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:47 smithi078 ceph-mon[91136]: pgmap v11243: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:47 smithi078 ceph-mon[99959]: pgmap v11243: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:49 smithi146 ceph-mon[103278]: pgmap v11244: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:49 smithi078 ceph-mon[91136]: pgmap v11244: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:49 smithi078 ceph-mon[99959]: pgmap v11244: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:34:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:34:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:34:51.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[91136]: pgmap v11245: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:51.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-mon[99959]: pgmap v11245: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:51.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:51 smithi146 ceph-mon[103278]: pgmap v11245: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:51.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:51.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:34:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:34:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:34:51.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:34:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:53 smithi146 ceph-mon[103278]: pgmap v11246: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:54.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:53 smithi078 ceph-mon[91136]: pgmap v11246: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:53 smithi078 ceph-mon[99959]: pgmap v11246: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:55 smithi146 ceph-mon[103278]: pgmap v11247: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[91136]: pgmap v11247: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[99959]: pgmap v11247: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:34:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:57 smithi146 ceph-mon[103278]: pgmap v11248: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:57 smithi078 ceph-mon[91136]: pgmap v11248: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:57 smithi078 ceph-mon[99959]: pgmap v11248: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:34:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:34:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:00.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:59 smithi146 ceph-mon[103278]: pgmap v11249: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:00.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:34:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:00.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:59 smithi078 ceph-mon[91136]: pgmap v11249: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:34:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:59 smithi078 ceph-mon[99959]: pgmap v11249: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:34:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:35:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:35:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:01 smithi146 ceph-mon[103278]: pgmap v11250: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:01 smithi078 ceph-mon[99959]: pgmap v11250: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:01 smithi078 ceph-mon[91136]: pgmap v11250: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:01.729Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:01.729Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:01.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:03.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:03 smithi078 ceph-mon[99959]: pgmap v11251: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:03.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:03 smithi078 ceph-mon[91136]: pgmap v11251: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:04.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:03 smithi146 ceph-mon[103278]: pgmap v11251: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:05 smithi146 ceph-mon[103278]: pgmap v11252: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:05 smithi078 ceph-mon[99959]: pgmap v11252: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:05 smithi078 ceph-mon[91136]: pgmap v11252: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:07 smithi146 ceph-mon[103278]: pgmap v11253: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:08.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:07 smithi078 ceph-mon[99959]: pgmap v11253: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:07 smithi078 ceph-mon[91136]: pgmap v11253: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:09.595 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:35:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:09 smithi146 ceph-mon[103278]: pgmap v11254: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:09 smithi078 ceph-mon[99959]: pgmap v11254: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:10.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:09 smithi078 ceph-mon[91136]: pgmap v11254: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 6h - - 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 14s ago 6h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 14s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:35:10.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 19s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 14s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 19s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 14s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 19s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 19s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 19s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 19s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 14s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 14s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 14s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 14s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 14s ago 6h 77.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 19s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:35:10.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:35:10.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 19s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:35:10.122 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 14s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:35:10.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:35:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:35:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:35:10.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:35:10.429 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:35:10.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[99959]: from='client.69567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:10.709 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[99959]: from='client.69573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[99959]: from='client.69579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1146246007' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:35:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[91136]: from='client.69567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[91136]: from='client.69573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:10.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[91136]: from='client.69579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:10.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1146246007' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:10.711 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:35:10.711 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:35:10.712 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:35:10.712 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:35:10.712 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:35:10.712 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:35:10.712 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:35:11.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:10 smithi146 ceph-mon[103278]: from='client.69567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:10 smithi146 ceph-mon[103278]: from='client.69573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:10 smithi146 ceph-mon[103278]: from='client.69579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1146246007' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:11.020 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:35:11.020 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:11.020 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:11.020 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:11 smithi146 ceph-mon[103278]: pgmap v11255: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:11 smithi146 ceph-mon[103278]: from='client.69591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/957780437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[91136]: pgmap v11255: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[91136]: from='client.69591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/957780437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[99959]: pgmap v11255: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[99959]: from='client.69591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/957780437' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:11.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:11.730Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:11.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:13 smithi146 ceph-mon[103278]: pgmap v11256: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:13 smithi078 ceph-mon[99959]: pgmap v11256: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:13 smithi078 ceph-mon[91136]: pgmap v11256: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:15 smithi146 ceph-mon[103278]: pgmap v11257: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[91136]: pgmap v11257: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[99959]: pgmap v11257: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:17.911 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:17 smithi146 ceph-mon[103278]: pgmap v11258: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:17 smithi078 ceph-mon[99959]: pgmap v11258: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:17 smithi078 ceph-mon[91136]: pgmap v11258: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:19 smithi146 ceph-mon[103278]: pgmap v11259: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:19 smithi078 ceph-mon[91136]: pgmap v11259: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:19 smithi078 ceph-mon[99959]: pgmap v11259: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:35:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T02:35:21.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:21 smithi078 ceph-mon[91136]: pgmap v11260: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:21 smithi078 ceph-mon[99959]: pgmap v11260: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:21.730Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:21.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:21.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:21.731Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:21 smithi146 ceph-mon[103278]: pgmap v11260: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:23 smithi146 ceph-mon[103278]: pgmap v11261: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:24.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:23 smithi078 ceph-mon[91136]: pgmap v11261: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:23 smithi078 ceph-mon[99959]: pgmap v11261: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:25.772 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:25 smithi078 ceph-mon[99959]: pgmap v11262: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:25.773 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:25 smithi078 ceph-mon[91136]: pgmap v11262: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:25 smithi146 ceph-mon[103278]: pgmap v11262: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:27 smithi146 ceph-mon[103278]: pgmap v11263: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:27 smithi078 ceph-mon[91136]: pgmap v11263: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:27 smithi078 ceph-mon[99959]: pgmap v11263: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:29 smithi146 ceph-mon[103278]: pgmap v11264: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:29 smithi078 ceph-mon[91136]: pgmap v11264: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:29 smithi078 ceph-mon[99959]: pgmap v11264: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:35:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:35:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:31 smithi146 ceph-mon[103278]: pgmap v11265: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[91136]: pgmap v11265: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[99959]: pgmap v11265: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:31.731Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:31.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:31.732Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:33 smithi146 ceph-mon[103278]: pgmap v11266: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:34.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:33 smithi078 ceph-mon[99959]: pgmap v11266: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:33 smithi078 ceph-mon[91136]: pgmap v11266: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:35 smithi146 ceph-mon[103278]: pgmap v11267: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[99959]: pgmap v11267: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[91136]: pgmap v11267: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:37 smithi146 ceph-mon[103278]: pgmap v11268: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:38.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:37 smithi078 ceph-mon[99959]: pgmap v11268: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:37 smithi078 ceph-mon[91136]: pgmap v11268: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:40.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:39 smithi146 ceph-mon[103278]: pgmap v11269: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:40.012 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:40.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:39 smithi078 ceph-mon[99959]: pgmap v11269: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:39 smithi078 ceph-mon[91136]: pgmap v11269: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:41.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:35:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:35:41.315 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 6h - - 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 6h - - 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 11s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 11s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 6s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 11s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 11s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:35:41.844 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 6s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 11s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 11s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 11s ago 6h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 11s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 11s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 6s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 6s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 6s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 6s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 6s ago 6h 77.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 11s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 11s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:35:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:35:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:41 smithi146 ceph-mon[103278]: pgmap v11270: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:41 smithi078 ceph-mon[99959]: pgmap v11270: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:41 smithi078 ceph-mon[91136]: pgmap v11270: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:41.732Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:41.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:41.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:35:42.158 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:35:42.159 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:35:42.429 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:35:42.429 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:35:42.429 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:35:42.429 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:35:42.430 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:35:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:35:42.431 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:35:42.431 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:35:42.741 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:35:42.742 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:42.742 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:35:42.742 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:42.742 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:35:43.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:42 smithi146 ceph-mon[103278]: from='client.69603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:42 smithi146 ceph-mon[103278]: from='client.89528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:42 smithi146 ceph-mon[103278]: from='client.69615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/61367963' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[99959]: from='client.69603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[99959]: from='client.89528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[99959]: from='client.69615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/61367963' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[91136]: from='client.69603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[91136]: from='client.89528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[91136]: from='client.69615 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/61367963' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:35:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:43 smithi146 ceph-mon[103278]: from='client.89546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:43 smithi146 ceph-mon[103278]: pgmap v11271: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2638855813' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[99959]: from='client.89546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[99959]: pgmap v11271: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:35:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2638855813' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[91136]: from='client.89546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:35:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[91136]: pgmap v11271: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:35:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2638855813' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:35:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:45 smithi146 ceph-mon[103278]: pgmap v11272: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:45 smithi078 ceph-mon[91136]: pgmap v11272: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:45 smithi078 ceph-mon[99959]: pgmap v11272: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:48.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:47 smithi146 ceph-mon[103278]: pgmap v11273: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:47 smithi078 ceph-mon[99959]: pgmap v11273: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:47 smithi078 ceph-mon[91136]: pgmap v11273: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:49 smithi146 ceph-mon[103278]: pgmap v11274: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:49 smithi078 ceph-mon[99959]: pgmap v11274: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:49 smithi078 ceph-mon[91136]: pgmap v11274: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:35:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:35:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[91136]: pgmap v11275: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[99959]: pgmap v11275: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:51.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:51.733Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:35:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:35:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:35:51.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:35:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:51 smithi146 ceph-mon[103278]: pgmap v11275: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:53 smithi146 ceph-mon[103278]: pgmap v11276: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:54.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:53 smithi078 ceph-mon[91136]: pgmap v11276: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:53 smithi078 ceph-mon[99959]: pgmap v11276: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:55 smithi146 ceph-mon[103278]: pgmap v11277: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[91136]: pgmap v11277: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[99959]: pgmap v11277: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:35:58.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:57 smithi146 ceph-mon[103278]: pgmap v11278: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:58.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:57 smithi078 ceph-mon[91136]: pgmap v11278: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:57 smithi078 ceph-mon[99959]: pgmap v11278: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:35:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:35:59.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:00.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:59 smithi146 ceph-mon[103278]: pgmap v11279: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:35:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:59 smithi078 ceph-mon[91136]: pgmap v11279: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:35:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:59 smithi078 ceph-mon[99959]: pgmap v11279: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:35:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:36:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:36:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:01 smithi078 ceph-mon[91136]: pgmap v11280: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:01 smithi078 ceph-mon[99959]: pgmap v11280: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:01.733Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:01.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:01.734Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:01 smithi146 ceph-mon[103278]: pgmap v11280: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:03 smithi146 ceph-mon[103278]: pgmap v11281: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:03 smithi078 ceph-mon[99959]: pgmap v11281: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:03 smithi078 ceph-mon[91136]: pgmap v11281: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:06.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:05 smithi078 ceph-mon[91136]: pgmap v11282: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:06.123 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:05 smithi078 ceph-mon[99959]: pgmap v11282: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:05 smithi146 ceph-mon[103278]: pgmap v11282: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:07.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:07 smithi078 ceph-mon[91136]: pgmap v11283: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:07.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:07 smithi078 ceph-mon[99959]: pgmap v11283: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:07 smithi146 ceph-mon[103278]: pgmap v11283: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:09 smithi146 ceph-mon[103278]: pgmap v11284: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:09 smithi078 ceph-mon[91136]: pgmap v11284: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:09 smithi078 ceph-mon[99959]: pgmap v11284: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:10.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:36:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:36:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[99959]: pgmap v11285: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[91136]: pgmap v11285: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:11.734Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:11.735Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:11.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:11 smithi146 ceph-mon[103278]: pgmap v11285: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:13.039 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:36:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:36:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T02:36:13.566 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 6h - - 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 17s ago 6h 87.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 17s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 2s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 2s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 17s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 2s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 17s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 2s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 2s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:36:13.567 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 2s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 17s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 17s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 17s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 17s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 17s ago 6h 77.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 2s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 2s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:36:13.568 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 17s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:36:13.875 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:36:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:36:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:36:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:36:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:36:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:36:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:36:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:36:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:36:14.151 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:36:14.151 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:36:14.151 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[99959]: pgmap v11286: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:14.151 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[99959]: from='client.69639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:14.151 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:14.151 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1977986652' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:14.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[91136]: pgmap v11286: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:14.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[91136]: from='client.69639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:14.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:14.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1977986652' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:13 smithi146 ceph-mon[103278]: pgmap v11286: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:13 smithi146 ceph-mon[103278]: from='client.69639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1977986652' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:14.457 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:36:14.457 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:14.458 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:14.458 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:14.458 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:14 smithi146 ceph-mon[103278]: from='client.89564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:14 smithi146 ceph-mon[103278]: from='client.69651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:14 smithi146 ceph-mon[103278]: from='client.89582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/185176304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[91136]: from='client.89564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[91136]: from='client.69651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[91136]: from='client.89582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/185176304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[99959]: from='client.89564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[99959]: from='client.69651 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[99959]: from='client.89582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:15.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/185176304' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:15 smithi146 ceph-mon[103278]: pgmap v11287: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[91136]: pgmap v11287: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[99959]: pgmap v11287: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:17 smithi146 ceph-mon[103278]: pgmap v11288: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:17 smithi078 ceph-mon[91136]: pgmap v11288: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:17 smithi078 ceph-mon[99959]: pgmap v11288: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:19 smithi146 ceph-mon[103278]: pgmap v11289: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:20.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:19 smithi078 ceph-mon[91136]: pgmap v11289: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:19 smithi078 ceph-mon[99959]: pgmap v11289: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:21.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:36:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:36:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:21 smithi078 ceph-mon[91136]: pgmap v11290: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:21 smithi078 ceph-mon[99959]: pgmap v11290: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:21.735Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:21.736Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:21.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:21 smithi146 ceph-mon[103278]: pgmap v11290: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:23 smithi146 ceph-mon[103278]: pgmap v11291: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:23 smithi078 ceph-mon[99959]: pgmap v11291: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:23 smithi078 ceph-mon[91136]: pgmap v11291: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:25 smithi146 ceph-mon[103278]: pgmap v11292: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:25 smithi078 ceph-mon[99959]: pgmap v11292: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:25 smithi078 ceph-mon[91136]: pgmap v11292: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:27 smithi146 ceph-mon[103278]: pgmap v11293: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:27 smithi078 ceph-mon[91136]: pgmap v11293: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:27 smithi078 ceph-mon[99959]: pgmap v11293: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:29.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:29.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:29.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:30 smithi146 ceph-mon[103278]: pgmap v11294: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:30.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:30 smithi078 ceph-mon[91136]: pgmap v11294: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:30 smithi078 ceph-mon[99959]: pgmap v11294: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:36:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:36:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:31.737Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:31.737Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:31.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:32 smithi146 ceph-mon[103278]: pgmap v11295: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[91136]: pgmap v11295: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[99959]: pgmap v11295: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:34 smithi146 ceph-mon[103278]: pgmap v11296: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:34.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:34 smithi078 ceph-mon[91136]: pgmap v11296: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:34 smithi078 ceph-mon[99959]: pgmap v11296: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:36 smithi146 ceph-mon[103278]: pgmap v11297: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[91136]: pgmap v11297: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[99959]: pgmap v11297: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:38 smithi146 ceph-mon[103278]: pgmap v11298: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:38 smithi078 ceph-mon[91136]: pgmap v11298: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:38 smithi078 ceph-mon[99959]: pgmap v11298: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:40.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:40 smithi146 ceph-mon[103278]: pgmap v11299: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:40.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:40 smithi078 ceph-mon[99959]: pgmap v11299: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:40 smithi078 ceph-mon[91136]: pgmap v11299: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:36:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:36:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:41.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:41.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:41.738Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:42 smithi146 ceph-mon[103278]: pgmap v11300: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:42.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:42 smithi078 ceph-mon[91136]: pgmap v11300: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:42 smithi078 ceph-mon[99959]: pgmap v11300: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:44 smithi146 ceph-mon[103278]: pgmap v11301: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:44.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:44 smithi078 ceph-mon[99959]: pgmap v11301: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:44 smithi078 ceph-mon[91136]: pgmap v11301: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:44.751 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:36:45.279 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 6h - - 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 6h - - 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 14s ago 6h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 9s ago 6h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 14s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 9s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 14s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 14s ago 6h 547M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 9s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 14s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 14s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 9s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:36:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 14s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 14s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 14s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 9s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 9s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 9s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 9s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 9s ago 6h 77.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 14s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 14s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:36:45.281 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 9s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:36:45.591 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:36:45.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:36:45.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:36:46.169 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:36:46.169 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:36:46.169 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:36:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:46 smithi146 ceph-mon[103278]: pgmap v11302: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:46.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:46 smithi146 ceph-mon[103278]: from='client.89594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:46 smithi146 ceph-mon[103278]: from='client.89600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/225349578' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[91136]: pgmap v11302: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:46.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[91136]: from='client.89594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[91136]: from='client.89600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/225349578' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[99959]: pgmap v11302: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[99959]: from='client.89594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[99959]: from='client.89600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/225349578' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:36:47.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:47 smithi078 ceph-mon[91136]: from='client.69687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:47.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:47 smithi078 ceph-mon[91136]: from='client.89618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:47.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2466776269' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:47.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:47 smithi078 ceph-mon[99959]: from='client.69687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:47.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:47 smithi078 ceph-mon[99959]: from='client.89618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:47.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2466776269' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:47.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:47 smithi146 ceph-mon[103278]: from='client.69687 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:47 smithi146 ceph-mon[103278]: from='client.89618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:36:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2466776269' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:36:48.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:48 smithi078 ceph-mon[91136]: pgmap v11303: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:48.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:48 smithi078 ceph-mon[99959]: pgmap v11303: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:48 smithi146 ceph-mon[103278]: pgmap v11303: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:50 smithi078 ceph-mon[91136]: pgmap v11304: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:50 smithi078 ceph-mon[99959]: pgmap v11304: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:50 smithi146 ceph-mon[103278]: pgmap v11304: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:36:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:36:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:36:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:51.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:51.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:36:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:36:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:36:51.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:36:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:52 smithi146 ceph-mon[103278]: pgmap v11305: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[91136]: pgmap v11305: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[99959]: pgmap v11305: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:54 smithi146 ceph-mon[103278]: pgmap v11306: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:54 smithi078 ceph-mon[91136]: pgmap v11306: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:54 smithi078 ceph-mon[99959]: pgmap v11306: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:56 smithi146 ceph-mon[103278]: pgmap v11307: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[91136]: pgmap v11307: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[99959]: pgmap v11307: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:36:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:58 smithi146 ceph-mon[103278]: pgmap v11308: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:58 smithi078 ceph-mon[91136]: pgmap v11308: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:58 smithi078 ceph-mon[99959]: pgmap v11308: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:36:59.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:59.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:36:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:00.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:00 smithi146 ceph-mon[103278]: pgmap v11309: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:00.405 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:00 smithi078 ceph-mon[99959]: pgmap v11309: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:00 smithi078 ceph-mon[91136]: pgmap v11309: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:01.034 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:37:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:37:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:01.738Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:01.739Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:01.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:02.232 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:02 smithi146 ceph-mon[103278]: pgmap v11310: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:02 smithi078 ceph-mon[91136]: pgmap v11310: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:02 smithi078 ceph-mon[99959]: pgmap v11310: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:04 smithi146 ceph-mon[103278]: pgmap v11311: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:04.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:04 smithi078 ceph-mon[91136]: pgmap v11311: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:04 smithi078 ceph-mon[99959]: pgmap v11311: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:06 smithi146 ceph-mon[103278]: pgmap v11312: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:06.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:06 smithi078 ceph-mon[99959]: pgmap v11312: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:06 smithi078 ceph-mon[91136]: pgmap v11312: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:08.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:08 smithi078 ceph-mon[91136]: pgmap v11313: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:08.415 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:08 smithi078 ceph-mon[99959]: pgmap v11313: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:08 smithi146 ceph-mon[103278]: pgmap v11313: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:10 smithi078 ceph-mon[91136]: pgmap v11314: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:10 smithi078 ceph-mon[99959]: pgmap v11314: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:10 smithi146 ceph-mon[103278]: pgmap v11314: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:10.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:37:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:37:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:11.740Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:11.740Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:11.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:12 smithi146 ceph-mon[103278]: pgmap v11315: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[99959]: pgmap v11315: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[91136]: pgmap v11315: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:14 smithi146 ceph-mon[103278]: pgmap v11316: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:14 smithi078 ceph-mon[99959]: pgmap v11316: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:14 smithi078 ceph-mon[91136]: pgmap v11316: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:16.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:16 smithi078 ceph-mon[99959]: pgmap v11317: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:16.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:16 smithi078 ceph-mon[91136]: pgmap v11317: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:16.467 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:37:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:16 smithi146 ceph-mon[103278]: pgmap v11317: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 6h - - 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 6h - - 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 5s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 0s ago 6h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 5s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:37:16.994 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 0s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 5s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 5s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 0s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 5s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 5s ago 6h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 0s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 5s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 5s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 5s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 0s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 0s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 0s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:37:16.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 0s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:37:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 0s ago 6h 77.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:37:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 5s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:37:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:37:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 5s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:37:16.996 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 0s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: from='client.89630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: pgmap v11318: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: from='client.69717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[91136]: from='client.69723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: from='client.89630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: pgmap v11318: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: from='client.69717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:17 smithi078 ceph-mon[99959]: from='client.69723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.308 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:37:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:37:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:37:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:37:17.309 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:37:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: from='client.89630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: pgmap v11318: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: from='client.69717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:17 smithi146 ceph-mon[103278]: from='client.69723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:37:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:37:17.582 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:37:17.891 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:37:17.891 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:17.891 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:17.891 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:17.891 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/988372268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:18 smithi146 ceph-mon[103278]: from='client.89654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/61573471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/988372268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:18 smithi078 ceph-mon[91136]: from='client.89654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/61573471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/988372268' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:18 smithi078 ceph-mon[99959]: from='client.89654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/61573471' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:19.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:19 smithi146 ceph-mon[103278]: pgmap v11319: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:19 smithi078 ceph-mon[99959]: pgmap v11319: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:19.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:19 smithi078 ceph-mon[91136]: pgmap v11319: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:37:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:37:21.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:21 smithi146 ceph-mon[103278]: pgmap v11320: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:21 smithi078 ceph-mon[99959]: pgmap v11320: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:21 smithi078 ceph-mon[91136]: pgmap v11320: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:21.741Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:21.741Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:21.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:23 smithi146 ceph-mon[103278]: pgmap v11321: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:23 smithi078 ceph-mon[99959]: pgmap v11321: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:23 smithi078 ceph-mon[91136]: pgmap v11321: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:25 smithi146 ceph-mon[103278]: pgmap v11322: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:26.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:25 smithi078 ceph-mon[99959]: pgmap v11322: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:25 smithi078 ceph-mon[91136]: pgmap v11322: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:27 smithi146 ceph-mon[103278]: pgmap v11323: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:28.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:27 smithi078 ceph-mon[99959]: pgmap v11323: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:27 smithi078 ceph-mon[91136]: pgmap v11323: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:28.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:28.885 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:29 smithi078 ceph-mon[99959]: pgmap v11324: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:29 smithi078 ceph-mon[91136]: pgmap v11324: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:29 smithi146 ceph-mon[103278]: pgmap v11324: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:37:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:37:31.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[99959]: pgmap v11325: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[91136]: pgmap v11325: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:31.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:31 smithi146 ceph-mon[103278]: pgmap v11325: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:31.741Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:31.742Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:31.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:33 smithi146 ceph-mon[103278]: pgmap v11326: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:34.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:33 smithi078 ceph-mon[91136]: pgmap v11326: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:33 smithi078 ceph-mon[99959]: pgmap v11326: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:35 smithi146 ceph-mon[103278]: pgmap v11327: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:36.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:35 smithi078 ceph-mon[91136]: pgmap v11327: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:35 smithi078 ceph-mon[99959]: pgmap v11327: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:37.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:37 smithi146 ceph-mon[103278]: pgmap v11328: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:37.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[99959]: pgmap v11328: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:37 smithi078 ceph-mon[91136]: pgmap v11328: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:39 smithi146 ceph-mon[103278]: pgmap v11329: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:39 smithi078 ceph-mon[99959]: pgmap v11329: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:39 smithi078 ceph-mon[91136]: pgmap v11329: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:37:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:37:41.884 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:41 smithi146 ceph-mon[103278]: pgmap v11330: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:41 smithi078 ceph-mon[99959]: pgmap v11330: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:41 smithi078 ceph-mon[91136]: pgmap v11330: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:41.743Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:41.743Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:41.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:43 smithi146 ceph-mon[103278]: pgmap v11331: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:43 smithi078 ceph-mon[99959]: pgmap v11331: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:43 smithi078 ceph-mon[91136]: pgmap v11331: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:45 smithi146 ceph-mon[103278]: pgmap v11332: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:45 smithi078 ceph-mon[99959]: pgmap v11332: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:45 smithi078 ceph-mon[91136]: pgmap v11332: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:47 smithi146 ceph-mon[103278]: pgmap v11333: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:47 smithi078 ceph-mon[91136]: pgmap v11333: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:47 smithi078 ceph-mon[99959]: pgmap v11333: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:48.185 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 6h - - 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 6h - - 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 17s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 12s ago 6h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 17s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 12s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 17s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 17s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 12s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:37:48.713 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 17s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 17s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 12s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 17s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 17s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 17s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 17s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 12s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 12s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 12s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 12s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 12s ago 6h 77.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 17s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:37:48.714 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 17s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:37:48.715 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 12s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:37:49.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:48 smithi146 ceph-mon[103278]: from='client.89666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:37:49.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:37:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:37:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:37:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:37:49.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:37:49.026 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:37:49.026 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:48 smithi078 ceph-mon[99959]: from='client.89666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.026 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:48 smithi078 ceph-mon[91136]: from='client.89666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.296 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:37:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:37:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:37:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:37:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:37:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:37:49.297 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:37:49.619 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:37:49.619 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:37:49.619 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[99959]: from='client.89672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[99959]: pgmap v11334: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[99959]: from='client.69759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2619326489' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3332078828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:49.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[91136]: from='client.89672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[91136]: pgmap v11334: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:49.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[91136]: from='client.69759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:49.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:49.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2619326489' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:49.985 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3332078828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:49 smithi146 ceph-mon[103278]: from='client.89672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:49 smithi146 ceph-mon[103278]: pgmap v11334: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:49 smithi146 ceph-mon[103278]: from='client.69759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:37:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2619326489' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:37:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3332078828' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:37:50.640 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:37:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:37:50.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:50 smithi078 ceph-mon[91136]: from='client.89690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:50.898 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:50 smithi078 ceph-mon[99959]: from='client.89690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:51.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:50 smithi146 ceph-mon[103278]: from='client.89690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:37:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:51 smithi146 ceph-mon[103278]: pgmap v11335: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[99959]: pgmap v11335: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[91136]: pgmap v11335: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:51.744Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:51.744Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:37:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:37:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:37:51.745Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:37:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:53 smithi146 ceph-mon[103278]: pgmap v11336: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:54.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:53 smithi078 ceph-mon[99959]: pgmap v11336: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:53 smithi078 ceph-mon[91136]: pgmap v11336: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:55 smithi146 ceph-mon[103278]: pgmap v11337: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:55 smithi078 ceph-mon[99959]: pgmap v11337: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:55 smithi078 ceph-mon[91136]: pgmap v11337: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:57.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:57 smithi146 ceph-mon[103278]: pgmap v11338: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:57.784 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[99959]: pgmap v11338: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:37:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:57 smithi078 ceph-mon[91136]: pgmap v11338: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:37:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:37:59.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:59 smithi146 ceph-mon[103278]: pgmap v11339: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:37:59.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:37:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:59 smithi078 ceph-mon[99959]: pgmap v11339: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:37:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:59 smithi078 ceph-mon[91136]: pgmap v11339: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:37:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:38:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:38:01.763 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:01 smithi146 ceph-mon[103278]: pgmap v11340: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:01.896 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:01 smithi078 ceph-mon[91136]: pgmap v11340: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:01.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:01 smithi078 ceph-mon[99959]: pgmap v11340: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:01.896 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:01.745Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:01.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:01.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:01.897 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:01.746Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:03 smithi146 ceph-mon[103278]: pgmap v11341: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:03 smithi078 ceph-mon[91136]: pgmap v11341: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:03 smithi078 ceph-mon[99959]: pgmap v11341: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:05 smithi146 ceph-mon[103278]: pgmap v11342: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:05 smithi078 ceph-mon[91136]: pgmap v11342: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:05 smithi078 ceph-mon[99959]: pgmap v11342: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:07 smithi146 ceph-mon[103278]: pgmap v11343: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:07 smithi078 ceph-mon[99959]: pgmap v11343: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:07 smithi078 ceph-mon[91136]: pgmap v11343: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:09 smithi146 ceph-mon[103278]: pgmap v11344: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:09 smithi078 ceph-mon[91136]: pgmap v11344: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:09 smithi078 ceph-mon[99959]: pgmap v11344: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:38:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:38:11.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:11 smithi078 ceph-mon[99959]: pgmap v11345: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:11.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:11 smithi078 ceph-mon[91136]: pgmap v11345: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:11 smithi146 ceph-mon[103278]: pgmap v11345: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:11.746Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:11.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:11.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:13.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:13 smithi146 ceph-mon[103278]: pgmap v11346: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:13 smithi078 ceph-mon[99959]: pgmap v11346: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:13 smithi078 ceph-mon[91136]: pgmap v11346: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:15 smithi146 ceph-mon[103278]: pgmap v11347: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:15 smithi078 ceph-mon[91136]: pgmap v11347: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:15 smithi078 ceph-mon[99959]: pgmap v11347: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:17 smithi146 ceph-mon[103278]: pgmap v11348: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[91136]: pgmap v11348: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[99959]: pgmap v11348: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:19.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:19 smithi078 ceph-mon[99959]: pgmap v11349: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:19.909 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:19.909 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:19 smithi078 ceph-mon[91136]: pgmap v11349: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:19.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:19.910 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:38:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:19 smithi146 ceph-mon[103278]: pgmap v11349: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 6h - - 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 6h - - 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 8s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 3s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 8s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:38:20.433 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 3s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 8s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 8s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 3s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 8s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 8s ago 6h 14.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 3s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 8s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 8s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 8s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 3s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 3s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 3s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:38:20.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 3s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:38:20.435 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 3s ago 6h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:38:20.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 8s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:38:20.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:38:20.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 8s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:38:20.435 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 3s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:38:20.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:20 smithi078 ceph-mon[91136]: from='client.69783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:20.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:20 smithi078 ceph-mon[91136]: from='client.69789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:20.748 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:38:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:38:20.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:20 smithi078 ceph-mon[99959]: from='client.69783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:20.749 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:20 smithi078 ceph-mon[99959]: from='client.69789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:38:20.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:38:20.750 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:38:21.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:20 smithi146 ceph-mon[103278]: from='client.69783 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:20 smithi146 ceph-mon[103278]: from='client.69789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:38:21.019 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:38:21.020 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:38:21.020 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:38:21.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:38:21.327 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:38:21.327 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:21.328 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:21.328 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:21.328 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:21 smithi146 ceph-mon[103278]: from='client.69795 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:21 smithi146 ceph-mon[103278]: pgmap v11350: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4065346529' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:21 smithi146 ceph-mon[103278]: from='client.69807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2306419914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:22.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[91136]: from='client.69795 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[91136]: pgmap v11350: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4065346529' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[91136]: from='client.69807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2306419914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[99959]: from='client.69795 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[99959]: pgmap v11350: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4065346529' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[99959]: from='client.69807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2306419914' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:21.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:21.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:21.747Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:23 smithi146 ceph-mon[103278]: pgmap v11351: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:23 smithi078 ceph-mon[99959]: pgmap v11351: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:23 smithi078 ceph-mon[91136]: pgmap v11351: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:25.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:25 smithi146 ceph-mon[103278]: pgmap v11352: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:25 smithi078 ceph-mon[91136]: pgmap v11352: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:25 smithi078 ceph-mon[99959]: pgmap v11352: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:28.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:27 smithi146 ceph-mon[103278]: pgmap v11353: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:27 smithi078 ceph-mon[99959]: pgmap v11353: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:27 smithi078 ceph-mon[91136]: pgmap v11353: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:29 smithi146 ceph-mon[103278]: pgmap v11354: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:29.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:29 smithi078 ceph-mon[99959]: pgmap v11354: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:29 smithi078 ceph-mon[91136]: pgmap v11354: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:30.944 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:38:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:38:31.967 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:31 smithi078 ceph-mon[99959]: pgmap v11355: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:31.967 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:31 smithi078 ceph-mon[91136]: pgmap v11355: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:31.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:31.747Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:31.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:31.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:31.968 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:31.748Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:31 smithi146 ceph-mon[103278]: pgmap v11355: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:33.094 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.094 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.094 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:33 smithi146 ceph-mon[103278]: pgmap v11356: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:33 smithi078 ceph-mon[91136]: pgmap v11356: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:33 smithi078 ceph-mon[99959]: pgmap v11356: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:35 smithi146 ceph-mon[103278]: pgmap v11357: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:35 smithi078 ceph-mon[99959]: pgmap v11357: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:35 smithi078 ceph-mon[91136]: pgmap v11357: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:38.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:37 smithi146 ceph-mon[103278]: pgmap v11358: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[99959]: pgmap v11358: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[91136]: pgmap v11358: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:39 smithi146 ceph-mon[103278]: pgmap v11359: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:39 smithi078 ceph-mon[91136]: pgmap v11359: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:39 smithi078 ceph-mon[99959]: pgmap v11359: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:38:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:38:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:41 smithi146 ceph-mon[103278]: pgmap v11360: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:41 smithi078 ceph-mon[91136]: pgmap v11360: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:41 smithi078 ceph-mon[99959]: pgmap v11360: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:38:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:41.748Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:41.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:41.749Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:43 smithi146 ceph-mon[103278]: pgmap v11361: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:43 smithi078 ceph-mon[91136]: pgmap v11361: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:43 smithi078 ceph-mon[99959]: pgmap v11361: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:45 smithi078 ceph-mon[99959]: pgmap v11362: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:45 smithi078 ceph-mon[91136]: pgmap v11362: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:45 smithi146 ceph-mon[103278]: pgmap v11362: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:47 smithi078 ceph-mon[99959]: pgmap v11363: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:47 smithi078 ceph-mon[91136]: pgmap v11363: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:47 smithi146 ceph-mon[103278]: pgmap v11363: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:49 smithi078 ceph-mon[91136]: pgmap v11364: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:49 smithi078 ceph-mon[99959]: pgmap v11364: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:49 smithi146 ceph-mon[103278]: pgmap v11364: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:38:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:38:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:38:51.622 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:38:51.873 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:51 smithi078 ceph-mon[99959]: pgmap v11365: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:51.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:51 smithi078 ceph-mon[91136]: pgmap v11365: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:51.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:51.749Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:51.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:51.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:38:51.874 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:38:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:38:51.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 6h - - 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 6h - - 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 0s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 15s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 0s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 15s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 0s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 0s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 15s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 0s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:38:52.217 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 0s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 15s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 0s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 0s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 0s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 15s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 15s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 15s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 15s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 15s ago 6h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 0s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:38:52.218 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 0s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:38:52.219 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 15s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:38:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:51 smithi146 ceph-mon[103278]: pgmap v11365: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:52.543 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:38:52.543 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:38:52.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:38:52.545 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:38:52.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:38:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:38:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:38:52.825 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='client.89738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='client.69825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='client.89750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.079 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2493225092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='client.89738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='client.69825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='client.89750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.080 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2493225092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:53.184 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:38:53.184 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:38:53.184 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='client.89738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='client.69825 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='client.89750 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2493225092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:38:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:54 smithi146 ceph-mon[103278]: pgmap v11366: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:54 smithi146 ceph-mon[103278]: from='client.69843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2720752189' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:54 smithi078 ceph-mon[91136]: pgmap v11366: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:54 smithi078 ceph-mon[91136]: from='client.69843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2720752189' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:54 smithi078 ceph-mon[99959]: pgmap v11366: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:54 smithi078 ceph-mon[99959]: from='client.69843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:38:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2720752189' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:38:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:56 smithi146 ceph-mon[103278]: pgmap v11367: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:56 smithi078 ceph-mon[99959]: pgmap v11367: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:56 smithi078 ceph-mon[91136]: pgmap v11367: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:38:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:57 smithi146 ceph-mon[103278]: pgmap v11368: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[91136]: pgmap v11368: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[99959]: pgmap v11368: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:38:59.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:38:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:00.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:59 smithi146 ceph-mon[103278]: pgmap v11369: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:00.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:38:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:59 smithi078 ceph-mon[99959]: pgmap v11369: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:38:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:59 smithi078 ceph-mon[91136]: pgmap v11369: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:38:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:39:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:39:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:01 smithi078 ceph-mon[99959]: pgmap v11370: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:01 smithi078 ceph-mon[91136]: pgmap v11370: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:01.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:01.750Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:01.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:01 smithi146 ceph-mon[103278]: pgmap v11370: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:03 smithi146 ceph-mon[103278]: pgmap v11371: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:03 smithi078 ceph-mon[91136]: pgmap v11371: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:03 smithi078 ceph-mon[99959]: pgmap v11371: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:05 smithi146 ceph-mon[103278]: pgmap v11372: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:05 smithi078 ceph-mon[91136]: pgmap v11372: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:05 smithi078 ceph-mon[99959]: pgmap v11372: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:07 smithi146 ceph-mon[103278]: pgmap v11373: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:07 smithi078 ceph-mon[91136]: pgmap v11373: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:07 smithi078 ceph-mon[99959]: pgmap v11373: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:09 smithi146 ceph-mon[103278]: pgmap v11374: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:09 smithi078 ceph-mon[91136]: pgmap v11374: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:09 smithi078 ceph-mon[99959]: pgmap v11374: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:39:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:39:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:11 smithi078 ceph-mon[91136]: pgmap v11375: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:11 smithi078 ceph-mon[99959]: pgmap v11375: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:11.750Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:11.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:11.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:11 smithi146 ceph-mon[103278]: pgmap v11375: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:13.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:14.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:14 smithi078 ceph-mon[99959]: pgmap v11376: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:14.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:14.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:14 smithi078 ceph-mon[91136]: pgmap v11376: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:14.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:14.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:14 smithi146 ceph-mon[103278]: pgmap v11376: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:16 smithi146 ceph-mon[103278]: pgmap v11377: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:16 smithi078 ceph-mon[91136]: pgmap v11377: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:16 smithi078 ceph-mon[99959]: pgmap v11377: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:18 smithi146 ceph-mon[103278]: pgmap v11378: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[99959]: pgmap v11378: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[91136]: pgmap v11378: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:20 smithi078 ceph-mon[91136]: pgmap v11379: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:20 smithi078 ceph-mon[99959]: pgmap v11379: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:20 smithi146 ceph-mon[103278]: pgmap v11379: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:39:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:39:22.020 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:21.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:22.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:21.751Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:22.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:21.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:22.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:22 smithi078 ceph-mon[99959]: pgmap v11380: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:22.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:22 smithi078 ceph-mon[91136]: pgmap v11380: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:22 smithi146 ceph-mon[103278]: pgmap v11380: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:23.479 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:39:24.004 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:39:24.004 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 6h - - 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 6h - - 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 11s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 11s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 6s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 11s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 11s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 6s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 11s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 11s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 11s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 11s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:39:24.005 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 11s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 6s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 6s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 6s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 6s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 6s ago 6h 77.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 11s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 11s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:39:24.006 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 6s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:39:24.117 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:24 smithi146 ceph-mon[103278]: pgmap v11381: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:24 smithi078 ceph-mon[99959]: pgmap v11381: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:24 smithi078 ceph-mon[91136]: pgmap v11381: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:24.315 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:39:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:39:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:39:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:39:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:39:24.316 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:39:24.584 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:39:24.585 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:39:24.896 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:39:24.896 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:24.896 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:24.896 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:25.101 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:25 smithi146 ceph-mon[103278]: from='client.69855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:25 smithi146 ceph-mon[103278]: from='client.89780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:25 smithi146 ceph-mon[103278]: from='client.69867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1097282980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:25.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/389205479' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[99959]: from='client.69855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[99959]: from='client.89780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[99959]: from='client.69867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1097282980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/389205479' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[91136]: from='client.69855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[91136]: from='client.89780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[91136]: from='client.69867 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1097282980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:25.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/389205479' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:26 smithi146 ceph-mon[103278]: from='client.69879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:26 smithi146 ceph-mon[103278]: pgmap v11382: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:26.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:26 smithi078 ceph-mon[99959]: from='client.69879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:26 smithi078 ceph-mon[99959]: pgmap v11382: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:26 smithi078 ceph-mon[91136]: from='client.69879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:26 smithi078 ceph-mon[91136]: pgmap v11382: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:28 smithi078 ceph-mon[99959]: pgmap v11383: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:28 smithi078 ceph-mon[91136]: pgmap v11383: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:28 smithi146 ceph-mon[103278]: pgmap v11383: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:29.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:30 smithi146 ceph-mon[103278]: pgmap v11384: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:30 smithi078 ceph-mon[99959]: pgmap v11384: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:30 smithi078 ceph-mon[91136]: pgmap v11384: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:39:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:39:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:31.751Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:31.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:31.752Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:32 smithi146 ceph-mon[103278]: pgmap v11385: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:32 smithi078 ceph-mon[99959]: pgmap v11385: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:32 smithi078 ceph-mon[91136]: pgmap v11385: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:33.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[91136]: pgmap v11386: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:33.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:33 smithi078 ceph-mon[99959]: pgmap v11386: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:33 smithi146 ceph-mon[103278]: pgmap v11386: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:36.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:35 smithi146 ceph-mon[103278]: pgmap v11387: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:35 smithi078 ceph-mon[91136]: pgmap v11387: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:35 smithi078 ceph-mon[99959]: pgmap v11387: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:38 smithi146 ceph-mon[103278]: pgmap v11388: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[91136]: pgmap v11388: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[99959]: pgmap v11388: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:40 smithi146 ceph-mon[103278]: pgmap v11389: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:40 smithi078 ceph-mon[99959]: pgmap v11389: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:40 smithi078 ceph-mon[91136]: pgmap v11389: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:39:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:39:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:41.753Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:41.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:41.753Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:42 smithi146 ceph-mon[103278]: pgmap v11390: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:42 smithi078 ceph-mon[91136]: pgmap v11390: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:42 smithi078 ceph-mon[99959]: pgmap v11390: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:44.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:44 smithi146 ceph-mon[103278]: pgmap v11391: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:44.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:44 smithi078 ceph-mon[99959]: pgmap v11391: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:44 smithi078 ceph-mon[91136]: pgmap v11391: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:46.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:46 smithi146 ceph-mon[103278]: pgmap v11392: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:46 smithi078 ceph-mon[99959]: pgmap v11392: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:46 smithi078 ceph-mon[91136]: pgmap v11392: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:48 smithi146 ceph-mon[103278]: pgmap v11393: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:48.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:48 smithi078 ceph-mon[99959]: pgmap v11393: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:48 smithi078 ceph-mon[91136]: pgmap v11393: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:50 smithi146 ceph-mon[103278]: pgmap v11394: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:50.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:50 smithi078 ceph-mon[91136]: pgmap v11394: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:50 smithi078 ceph-mon[99959]: pgmap v11394: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:39:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:39:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:39:52.038 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:51.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:52.039 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:51.754Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:39:52.039 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:39:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:39:51.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:39:52.385 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:52 smithi078 ceph-mon[99959]: pgmap v11395: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:52.385 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:52 smithi078 ceph-mon[91136]: pgmap v11395: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:52 smithi146 ceph-mon[103278]: pgmap v11395: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[91136]: pgmap v11396: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:53 smithi078 ceph-mon[99959]: pgmap v11396: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:53 smithi146 ceph-mon[103278]: pgmap v11396: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:55.200 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:39:55.758 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:39:55.758 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 6h - - 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 6h - - 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 3s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 18s ago 6h 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 3s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 18s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 3s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 3s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 18s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 3s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 3s ago 6h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 18s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 3s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 3s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 3s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 3s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:39:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 18s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 18s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 18s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 18s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 18s ago 6h 73.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 3s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 3s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:39:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 18s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:39:56.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:55 smithi146 ceph-mon[103278]: pgmap v11397: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:55 smithi146 ceph-mon[103278]: from='client.89810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:56.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:55 smithi078 ceph-mon[91136]: pgmap v11397: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:55 smithi078 ceph-mon[91136]: from='client.89810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:55 smithi078 ceph-mon[99959]: pgmap v11397: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:55 smithi078 ceph-mon[99959]: from='client.89810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:39:56.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:39:56.078 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:39:56.348 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:39:56.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:39:56.660 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:39:56.660 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:56.661 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:39:56.661 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:56.661 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:39:56.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:56 smithi146 ceph-mon[103278]: from='client.89816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:56.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:56 smithi146 ceph-mon[103278]: from='client.69903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:56.943 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1444618913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:56.944 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2826178416' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[99959]: from='client.89816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[99959]: from='client.69903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1444618913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2826178416' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[91136]: from='client.89816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[91136]: from='client.69903 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1444618913' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2826178416' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:39:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:57 smithi146 ceph-mon[103278]: from='client.89834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:57 smithi146 ceph-mon[103278]: pgmap v11398: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[91136]: from='client.89834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[91136]: pgmap v11398: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[99959]: from='client.89834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[99959]: pgmap v11398: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:39:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:59.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:39:59.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:59 smithi146 ceph-mon[103278]: pgmap v11399: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:39:59.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:39:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:00.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:59 smithi078 ceph-mon[91136]: pgmap v11399: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:00.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:39:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:00.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:59 smithi078 ceph-mon[99959]: pgmap v11399: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:00.000 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:39:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T02:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:01.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:01.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:40:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:40:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:01 smithi146 ceph-mon[103278]: pgmap v11400: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:02.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:01 smithi078 ceph-mon[99959]: pgmap v11400: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:01 smithi078 ceph-mon[91136]: pgmap v11400: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:01.754Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:01.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:01.755Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:03 smithi146 ceph-mon[103278]: pgmap v11401: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:03 smithi078 ceph-mon[91136]: pgmap v11401: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:03 smithi078 ceph-mon[99959]: pgmap v11401: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:05.999 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:05 smithi146 ceph-mon[103278]: pgmap v11402: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:06.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:05 smithi078 ceph-mon[91136]: pgmap v11402: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:05 smithi078 ceph-mon[99959]: pgmap v11402: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:07 smithi146 ceph-mon[103278]: pgmap v11403: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:07 smithi078 ceph-mon[91136]: pgmap v11403: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:07 smithi078 ceph-mon[99959]: pgmap v11403: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:09 smithi146 ceph-mon[103278]: pgmap v11404: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:09 smithi078 ceph-mon[91136]: pgmap v11404: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:09 smithi078 ceph-mon[99959]: pgmap v11404: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:40:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:40:12.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:11 smithi146 ceph-mon[103278]: pgmap v11405: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:11 smithi078 ceph-mon[91136]: pgmap v11405: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:11 smithi078 ceph-mon[99959]: pgmap v11405: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:11.755Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:11.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:11.756Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:13.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:13 smithi146 ceph-mon[103278]: pgmap v11406: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:13.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[91136]: pgmap v11406: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:13 smithi078 ceph-mon[99959]: pgmap v11406: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:14.775 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:14.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:15.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:15 smithi078 ceph-mon[99959]: pgmap v11407: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:15 smithi078 ceph-mon[91136]: pgmap v11407: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:15 smithi146 ceph-mon[103278]: pgmap v11407: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:17 smithi146 ceph-mon[103278]: pgmap v11408: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[91136]: pgmap v11408: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[99959]: pgmap v11408: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:20.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:19 smithi146 ceph-mon[103278]: pgmap v11409: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:19 smithi078 ceph-mon[99959]: pgmap v11409: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:19 smithi078 ceph-mon[91136]: pgmap v11409: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:20.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:40:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:40:21.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:21 smithi078 ceph-mon[99959]: pgmap v11410: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:21.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:21 smithi078 ceph-mon[91136]: pgmap v11410: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:21.756Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:21.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:21.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:21.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:22.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:21 smithi146 ceph-mon[103278]: pgmap v11410: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:23 smithi146 ceph-mon[103278]: pgmap v11411: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:23 smithi078 ceph-mon[91136]: pgmap v11411: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:23 smithi078 ceph-mon[99959]: pgmap v11411: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:25.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:25 smithi146 ceph-mon[103278]: pgmap v11412: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:25 smithi078 ceph-mon[91136]: pgmap v11412: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:25 smithi078 ceph-mon[99959]: pgmap v11412: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:26.955 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 6h - - 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 6h - - 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 14s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 10s ago 6h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 14s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 10s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 14s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 14s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 10s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 14s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:40:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 14s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 10s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 14s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 14s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 14s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 10s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 10s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 10s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 10s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 10s ago 6h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 14s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 14s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:40:27.481 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:40:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:27 smithi078 ceph-mon[91136]: pgmap v11413: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:27 smithi078 ceph-mon[91136]: from='client.69927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:27 smithi078 ceph-mon[91136]: from='client.89852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:27 smithi078 ceph-mon[99959]: pgmap v11413: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:27 smithi078 ceph-mon[99959]: from='client.69927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:27 smithi078 ceph-mon[99959]: from='client.89852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:40:27.787 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:40:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:40:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:27 smithi146 ceph-mon[103278]: pgmap v11413: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:27 smithi146 ceph-mon[103278]: from='client.69927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:27 smithi146 ceph-mon[103278]: from='client.89852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:40:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:40:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:40:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:40:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:40:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:40:28.056 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:40:28.056 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:40:28.363 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:40:28.363 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:40:28.363 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:40:28.363 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:40:28.364 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:40:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:28 smithi146 ceph-mon[103278]: from='client.89858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1066309608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:28 smithi146 ceph-mon[103278]: from='client.69951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3692259236' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[91136]: from='client.89858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:29.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1066309608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[91136]: from='client.69951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3692259236' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[99959]: from='client.89858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1066309608' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[99959]: from='client.69951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3692259236' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:40:29.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:29 smithi146 ceph-mon[103278]: pgmap v11414: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:29 smithi078 ceph-mon[91136]: pgmap v11414: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:29 smithi078 ceph-mon[99959]: pgmap v11414: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:40:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:40:31.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:31 smithi078 ceph-mon[91136]: pgmap v11415: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:31.758 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:31 smithi078 ceph-mon[99959]: pgmap v11415: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:31.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:31 smithi146 ceph-mon[103278]: pgmap v11415: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:31.757Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:31.757Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:31.758Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:34.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:33 smithi146 ceph-mon[103278]: pgmap v11416: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[91136]: pgmap v11416: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:34.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[99959]: pgmap v11416: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:34.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:34.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:35 smithi078 ceph-mon[91136]: pgmap v11417: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:35 smithi078 ceph-mon[99959]: pgmap v11417: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:35 smithi146 ceph-mon[103278]: pgmap v11417: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[91136]: pgmap v11418: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[99959]: pgmap v11418: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:37 smithi146 ceph-mon[103278]: pgmap v11418: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:40.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:39 smithi146 ceph-mon[103278]: pgmap v11419: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:39 smithi078 ceph-mon[99959]: pgmap v11419: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:39 smithi078 ceph-mon[91136]: pgmap v11419: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:40:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:40:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:41 smithi146 ceph-mon[103278]: pgmap v11420: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:42.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:41 smithi078 ceph-mon[99959]: pgmap v11420: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:41 smithi078 ceph-mon[91136]: pgmap v11420: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:41.758Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:41.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:41.759Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:43 smithi146 ceph-mon[103278]: pgmap v11421: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:44.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:43 smithi078 ceph-mon[99959]: pgmap v11421: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:43 smithi078 ceph-mon[91136]: pgmap v11421: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:45 smithi146 ceph-mon[103278]: pgmap v11422: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:45 smithi078 ceph-mon[99959]: pgmap v11422: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:45 smithi078 ceph-mon[91136]: pgmap v11422: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:47.927 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:47 smithi146 ceph-mon[103278]: pgmap v11423: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:47 smithi078 ceph-mon[99959]: pgmap v11423: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:47 smithi078 ceph-mon[91136]: pgmap v11423: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:50.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:49 smithi146 ceph-mon[103278]: pgmap v11424: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:50.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:49 smithi078 ceph-mon[91136]: pgmap v11424: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:50.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:49 smithi078 ceph-mon[99959]: pgmap v11424: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:50.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:40:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:40:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:40:51.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:51 smithi078 ceph-mon[99959]: pgmap v11425: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:51.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:51 smithi078 ceph-mon[91136]: pgmap v11425: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:51.936 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:51.759Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:51.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:40:51.937 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:40:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:40:51.760Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:40:52.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:51 smithi146 ceph-mon[103278]: pgmap v11425: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:40:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:53 smithi146 ceph-mon[103278]: pgmap v11426: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:40:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[91136]: pgmap v11426: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[99959]: pgmap v11426: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:55 smithi146 ceph-mon[103278]: pgmap v11427: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:55 smithi078 ceph-mon[91136]: pgmap v11427: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:55 smithi078 ceph-mon[99959]: pgmap v11427: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:40:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:57 smithi146 ceph-mon[103278]: pgmap v11428: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:40:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[99959]: pgmap v11428: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[91136]: pgmap v11428: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:40:58.655 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:40:58.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:58.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:59.193 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:40:59.193 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 6h - - 2024-09-21T02:40:59.193 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 6h - - 2024-09-21T02:40:59.193 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 6s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:40:59.193 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 1s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 6s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 1s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 6s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 6s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 1s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 6s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 6s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 1s ago 6h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 6s ago 6h 93.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 6s ago 6h 99.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 6s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 6s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 1s ago 6h 115M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 1s ago 6h 90.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:40:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 1s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:40:59.195 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 1s ago 6h 92.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:40:59.195 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 1s ago 6h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:40:59.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 6s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:40:59.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:40:59.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 6s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:40:59.195 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 1s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:40:59.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:40:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:40:59.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:40:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:40:59.776 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:40:59.776 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:40:59.776 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:40:59.776 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:40:59.776 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[99959]: pgmap v11429: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[99959]: from='client.69963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[99959]: from='client.89888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[99959]: from='client.69975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1952424388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[91136]: pgmap v11429: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:41:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[91136]: from='client.69963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[91136]: from='client.89888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[91136]: from='client.69975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:40:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1952424388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:00.085 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:41:00.085 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:00.085 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:59 smithi146 ceph-mon[103278]: pgmap v11429: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:41:00.170 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:59 smithi146 ceph-mon[103278]: from='client.69963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:59 smithi146 ceph-mon[103278]: from='client.89888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:59 smithi146 ceph-mon[103278]: from='client.69975 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:40:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1952424388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:00 smithi078 ceph-mon[91136]: from='client.89906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3768954799' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:41:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:41:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:00 smithi078 ceph-mon[99959]: from='client.89906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3768954799' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:00 smithi146 ceph-mon[103278]: from='client.89906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3768954799' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:01 smithi078 ceph-mon[91136]: pgmap v11430: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:41:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:01 smithi078 ceph-mon[99959]: pgmap v11430: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:41:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:01.760Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:01.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:01.761Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:01 smithi146 ceph-mon[103278]: pgmap v11430: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:41:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:03 smithi146 ceph-mon[103278]: pgmap v11431: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:41:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:03 smithi078 ceph-mon[99959]: pgmap v11431: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:41:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:03 smithi078 ceph-mon[91136]: pgmap v11431: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:41:06.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:05 smithi146 ceph-mon[103278]: pgmap v11432: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:41:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:05 smithi078 ceph-mon[99959]: pgmap v11432: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:41:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:05 smithi078 ceph-mon[91136]: pgmap v11432: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:41:08.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:07 smithi146 ceph-mon[103278]: pgmap v11433: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:41:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:07 smithi078 ceph-mon[91136]: pgmap v11433: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:41:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:07 smithi078 ceph-mon[99959]: pgmap v11433: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:41:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:09 smithi146 ceph-mon[103278]: pgmap v11434: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:41:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:09 smithi078 ceph-mon[99959]: pgmap v11434: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:41:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:09 smithi078 ceph-mon[91136]: pgmap v11434: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:41:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:41:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:41:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:11 smithi078 ceph-mon[99959]: pgmap v11435: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:41:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:11 smithi078 ceph-mon[91136]: pgmap v11435: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:41:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:11.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:11.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:11.762Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:11 smithi146 ceph-mon[103278]: pgmap v11435: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T02:41:14.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:13 smithi146 ceph-mon[103278]: pgmap v11436: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:41:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[99959]: pgmap v11436: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[91136]: pgmap v11436: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:15 smithi146 ceph-mon[103278]: pgmap v11437: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:41:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:15 smithi078 ceph-mon[99959]: pgmap v11437: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:41:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:15 smithi078 ceph-mon[91136]: pgmap v11437: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 0 B/s wr, 122 op/s 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[91136]: pgmap v11438: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[99959]: pgmap v11438: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:17 smithi146 ceph-mon[103278]: pgmap v11438: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 152 op/s 2024-09-21T02:41:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:19 smithi146 ceph-mon[103278]: pgmap v11439: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T02:41:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:19 smithi078 ceph-mon[91136]: pgmap v11439: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T02:41:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:19 smithi078 ceph-mon[99959]: pgmap v11439: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-09-21T02:41:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:41:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:41:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:21 smithi078 ceph-mon[91136]: pgmap v11440: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:41:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:21 smithi078 ceph-mon[99959]: pgmap v11440: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:41:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:21.762Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:21.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:21.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:22.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:21 smithi146 ceph-mon[103278]: pgmap v11440: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:41:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:23 smithi146 ceph-mon[103278]: pgmap v11441: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T02:41:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:23 smithi078 ceph-mon[91136]: pgmap v11441: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T02:41:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:23 smithi078 ceph-mon[99959]: pgmap v11441: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s 2024-09-21T02:41:26.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:25 smithi146 ceph-mon[103278]: pgmap v11442: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T02:41:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:25 smithi078 ceph-mon[91136]: pgmap v11442: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T02:41:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:25 smithi078 ceph-mon[99959]: pgmap v11442: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T02:41:28.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:27 smithi146 ceph-mon[103278]: pgmap v11443: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:41:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:27 smithi078 ceph-mon[91136]: pgmap v11443: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:41:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:27 smithi078 ceph-mon[99959]: pgmap v11443: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-09-21T02:41:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:29 smithi146 ceph-mon[103278]: pgmap v11444: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T02:41:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:29 smithi078 ceph-mon[91136]: pgmap v11444: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T02:41:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:29 smithi078 ceph-mon[99959]: pgmap v11444: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2024-09-21T02:41:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:30.376 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:41:30.784 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:41:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:41:30.903 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 18s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 13s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 18s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 13s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 6h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:41:30.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 6h 99.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 6h 82.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 13s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 13s ago 6h 91.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 13s ago 6h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 6h 92.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 6h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:41:30.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:41:31.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:41:31.217 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:41:31.483 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:41:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:41:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:41:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:41:31.484 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:41:31.794 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:41:31.794 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:41:31.794 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:41:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:31 smithi146 ceph-mon[103278]: from='client.89918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:31 smithi146 ceph-mon[103278]: from='client.89924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:31 smithi146 ceph-mon[103278]: pgmap v11445: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-21T02:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:31 smithi146 ceph-mon[103278]: from='client.70011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2296070366' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3541233145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:31.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:31.763Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:31.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[91136]: from='client.89918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[91136]: from='client.89924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[91136]: pgmap v11445: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[91136]: from='client.70011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2296070366' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3541233145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[99959]: from='client.89918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[99959]: from='client.89924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[99959]: pgmap v11445: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 56 op/s 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[99959]: from='client.70011 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2296070366' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:41:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3541233145' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:41:33.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:32 smithi146 ceph-mon[103278]: from='client.70023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:33.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:32 smithi078 ceph-mon[91136]: from='client.70023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:33.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:32 smithi078 ceph-mon[99959]: from='client.70023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:41:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:34 smithi146 ceph-mon[103278]: pgmap v11446: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:41:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.284 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[99959]: pgmap v11446: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[91136]: pgmap v11446: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:36 smithi146 ceph-mon[103278]: pgmap v11447: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:41:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:36 smithi078 ceph-mon[99959]: pgmap v11447: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:41:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:36 smithi078 ceph-mon[91136]: pgmap v11447: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:41:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:38 smithi146 ceph-mon[103278]: pgmap v11448: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:41:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[91136]: pgmap v11448: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[99959]: pgmap v11448: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:40.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:40 smithi078 ceph-mon[91136]: pgmap v11449: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:40.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:40.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:40 smithi078 ceph-mon[99959]: pgmap v11449: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:40.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:40.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:40 smithi146 ceph-mon[103278]: pgmap v11449: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:41:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:41:42.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:41.763Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:42.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:41.764Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:42.024 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:41.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:42 smithi146 ceph-mon[103278]: pgmap v11450: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:42 smithi078 ceph-mon[91136]: pgmap v11450: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:42 smithi078 ceph-mon[99959]: pgmap v11450: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:44 smithi078 ceph-mon[91136]: pgmap v11451: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:44 smithi078 ceph-mon[99959]: pgmap v11451: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:44.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:44 smithi146 ceph-mon[103278]: pgmap v11451: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:46 smithi078 ceph-mon[91136]: pgmap v11452: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:46 smithi078 ceph-mon[99959]: pgmap v11452: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:46 smithi146 ceph-mon[103278]: pgmap v11452: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:48 smithi078 ceph-mon[91136]: pgmap v11453: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:48 smithi078 ceph-mon[99959]: pgmap v11453: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:48.376 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:48 smithi146 ceph-mon[103278]: pgmap v11453: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:50.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:50 smithi146 ceph-mon[103278]: pgmap v11454: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:50.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:50 smithi078 ceph-mon[91136]: pgmap v11454: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:50 smithi078 ceph-mon[99959]: pgmap v11454: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:41:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:41:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:41:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:51.764Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:51.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:41:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:41:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:41:51.765Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:41:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:52 smithi146 ceph-mon[103278]: pgmap v11455: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:52 smithi078 ceph-mon[91136]: pgmap v11455: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:52 smithi078 ceph-mon[99959]: pgmap v11455: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:54 smithi146 ceph-mon[103278]: pgmap v11456: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[91136]: pgmap v11456: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[99959]: pgmap v11456: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:56 smithi146 ceph-mon[103278]: pgmap v11457: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:56 smithi078 ceph-mon[91136]: pgmap v11457: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:56 smithi078 ceph-mon[99959]: pgmap v11457: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:58 smithi146 ceph-mon[103278]: pgmap v11458: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[99959]: pgmap v11458: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[91136]: pgmap v11458: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:41:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:41:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:41:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:41:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:41:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:41:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:00.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:00 smithi078 ceph-mon[91136]: pgmap v11459: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:00.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:00.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:00 smithi078 ceph-mon[99959]: pgmap v11459: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:00.302 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:00.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:00 smithi146 ceph-mon[103278]: pgmap v11459: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:00.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:00.889 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:42:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:42:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:01.765Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:01.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:01.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:02.082 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:42:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:02 smithi146 ceph-mon[103278]: pgmap v11460: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:02.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:02 smithi078 ceph-mon[91136]: pgmap v11460: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:02 smithi078 ceph-mon[99959]: pgmap v11460: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 6h - - 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 4s ago 6h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 4s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:42:02.611 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 4s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 4s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 9s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 9s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 9s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 4s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 4s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 4s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 4s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 4s ago 6h 76.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:42:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:42:02.613 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 4s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:42:02.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:42:02.919 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:42:03.190 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:42:03.190 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:42:03.190 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:42:03.190 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:42:03.190 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:42:03.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:42:03.191 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:42:03.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:03 smithi078 ceph-mon[99959]: from='client.70035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:03.192 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2473424478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:03.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:03 smithi078 ceph-mon[91136]: from='client.70035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:03.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2473424478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:03.496 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:42:03.496 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:42:03.496 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:42:03.496 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:42:03.496 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:42:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:03 smithi146 ceph-mon[103278]: from='client.70035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:03.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2473424478' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:04 smithi146 ceph-mon[103278]: from='client.70041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:04 smithi146 ceph-mon[103278]: from='client.89966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:04 smithi146 ceph-mon[103278]: pgmap v11461: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:04 smithi146 ceph-mon[103278]: from='client.70059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3577120848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[99959]: from='client.70041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[99959]: from='client.89966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[99959]: pgmap v11461: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[99959]: from='client.70059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3577120848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[91136]: from='client.70041 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[91136]: from='client.89966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[91136]: pgmap v11461: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[91136]: from='client.70059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3577120848' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:06 smithi146 ceph-mon[103278]: pgmap v11462: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:06 smithi078 ceph-mon[91136]: pgmap v11462: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:06 smithi078 ceph-mon[99959]: pgmap v11462: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:08 smithi146 ceph-mon[103278]: pgmap v11463: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:08 smithi078 ceph-mon[91136]: pgmap v11463: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:08 smithi078 ceph-mon[99959]: pgmap v11463: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:10 smithi146 ceph-mon[103278]: pgmap v11464: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:10 smithi078 ceph-mon[99959]: pgmap v11464: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:10 smithi078 ceph-mon[91136]: pgmap v11464: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:42:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:42:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:11.766Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:11.766Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:11.767Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:12 smithi146 ceph-mon[103278]: pgmap v11465: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:12 smithi078 ceph-mon[99959]: pgmap v11465: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:12 smithi078 ceph-mon[91136]: pgmap v11465: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:14 smithi146 ceph-mon[103278]: pgmap v11466: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[99959]: pgmap v11466: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[91136]: pgmap v11466: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:16 smithi146 ceph-mon[103278]: pgmap v11467: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:16 smithi078 ceph-mon[99959]: pgmap v11467: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:16 smithi078 ceph-mon[91136]: pgmap v11467: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:18 smithi146 ceph-mon[103278]: pgmap v11468: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:18 smithi078 ceph-mon[99959]: pgmap v11468: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:18 smithi078 ceph-mon[91136]: pgmap v11468: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:19.237 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:19.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:20.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:20 smithi078 ceph-mon[99959]: pgmap v11469: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:20.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:20.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:20 smithi078 ceph-mon[91136]: pgmap v11469: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:20.509 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:20 smithi146 ceph-mon[103278]: pgmap v11469: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:20.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:42:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:42:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:21.767Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:21.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:21.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:22 smithi146 ceph-mon[103278]: pgmap v11470: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:22 smithi078 ceph-mon[91136]: pgmap v11470: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:22 smithi078 ceph-mon[99959]: pgmap v11470: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:24 smithi146 ceph-mon[103278]: pgmap v11471: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:24 smithi078 ceph-mon[91136]: pgmap v11471: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:24 smithi078 ceph-mon[99959]: pgmap v11471: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:26.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:26 smithi146 ceph-mon[103278]: pgmap v11472: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:26 smithi078 ceph-mon[91136]: pgmap v11472: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:26 smithi078 ceph-mon[99959]: pgmap v11472: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:28 smithi146 ceph-mon[103278]: pgmap v11473: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:28 smithi078 ceph-mon[91136]: pgmap v11473: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:28 smithi078 ceph-mon[99959]: pgmap v11473: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:30 smithi146 ceph-mon[103278]: pgmap v11474: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:30 smithi078 ceph-mon[99959]: pgmap v11474: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:30 smithi078 ceph-mon[91136]: pgmap v11474: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:42:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:42:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:31.768Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:31.768Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:31.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:32 smithi146 ceph-mon[103278]: pgmap v11475: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:32 smithi078 ceph-mon[91136]: pgmap v11475: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:32 smithi078 ceph-mon[99959]: pgmap v11475: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:33.782 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 6h - - 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 0s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 87.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 0s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 0s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:42:34.303 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 0s ago 6h 548M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 0s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 0s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 0s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 0s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 0s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 16s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 16s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 16s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 6h 76.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 0s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:42:34.304 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:42:34.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 0s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:42:34.305 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:42:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:34 smithi146 ceph-mon[103278]: pgmap v11476: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[99959]: pgmap v11476: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[91136]: pgmap v11476: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:42:34.614 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:42:34.884 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:42:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:42:34.885 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:42:34.885 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:42:35.197 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:42:35.197 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:42:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:42:35.197 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:42:35.197 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:42:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:35 smithi146 ceph-mon[103278]: from='client.70071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:35 smithi146 ceph-mon[103278]: from='client.70077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2528003588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1584580664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[91136]: from='client.70071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[91136]: from='client.70077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2528003588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1584580664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[99959]: from='client.70071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[99959]: from='client.70077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2528003588' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:42:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1584580664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:42:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:36 smithi146 ceph-mon[103278]: from='client.70083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:36 smithi146 ceph-mon[103278]: pgmap v11477: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:36 smithi146 ceph-mon[103278]: from='client.70095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:36 smithi078 ceph-mon[91136]: from='client.70083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:36 smithi078 ceph-mon[91136]: pgmap v11477: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:36 smithi078 ceph-mon[91136]: from='client.70095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:36 smithi078 ceph-mon[99959]: from='client.70083 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:36 smithi078 ceph-mon[99959]: pgmap v11477: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:36 smithi078 ceph-mon[99959]: from='client.70095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:42:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:38 smithi146 ceph-mon[103278]: pgmap v11478: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:38 smithi078 ceph-mon[91136]: pgmap v11478: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:38 smithi078 ceph-mon[99959]: pgmap v11478: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:39 smithi146 ceph-mon[103278]: pgmap v11479: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[91136]: pgmap v11479: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[99959]: pgmap v11479: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:41.015 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:42:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:42:42.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:41 smithi146 ceph-mon[103278]: pgmap v11480: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:41 smithi078 ceph-mon[91136]: pgmap v11480: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:41 smithi078 ceph-mon[99959]: pgmap v11480: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:41.769Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:41.769Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:41.770Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:43 smithi146 ceph-mon[103278]: pgmap v11481: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:43 smithi078 ceph-mon[91136]: pgmap v11481: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:43 smithi078 ceph-mon[99959]: pgmap v11481: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:46.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:45 smithi146 ceph-mon[103278]: pgmap v11482: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:45 smithi078 ceph-mon[91136]: pgmap v11482: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:45 smithi078 ceph-mon[99959]: pgmap v11482: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:47 smithi146 ceph-mon[103278]: pgmap v11483: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:47 smithi078 ceph-mon[91136]: pgmap v11483: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:47 smithi078 ceph-mon[99959]: pgmap v11483: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:50.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:49 smithi146 ceph-mon[103278]: pgmap v11484: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:49 smithi078 ceph-mon[91136]: pgmap v11484: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:49 smithi078 ceph-mon[99959]: pgmap v11484: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:42:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:42:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:51 smithi146 ceph-mon[103278]: pgmap v11485: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:51 smithi078 ceph-mon[91136]: pgmap v11485: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:51 smithi078 ceph-mon[99959]: pgmap v11485: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:51.770Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:51.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:42:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:42:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:42:51.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:53 smithi146 ceph-mon[103278]: pgmap v11486: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[91136]: pgmap v11486: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[99959]: pgmap v11486: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:55 smithi146 ceph-mon[103278]: pgmap v11487: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:55 smithi078 ceph-mon[99959]: pgmap v11487: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:55 smithi078 ceph-mon[91136]: pgmap v11487: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:58.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:57 smithi146 ceph-mon[103278]: pgmap v11488: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:57 smithi078 ceph-mon[99959]: pgmap v11488: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:57 smithi078 ceph-mon[91136]: pgmap v11488: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:42:59.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:59 smithi146 ceph-mon[103278]: pgmap v11489: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:42:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[91136]: pgmap v11489: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:42:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:42:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:42:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[99959]: pgmap v11489: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:42:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:42:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:00.886 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:43:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:43:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:01 smithi146 ceph-mon[103278]: pgmap v11490: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:01 smithi078 ceph-mon[91136]: pgmap v11490: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:01 smithi078 ceph-mon[99959]: pgmap v11490: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:01.771Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:01.771Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:01.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:03 smithi146 ceph-mon[103278]: pgmap v11491: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:43:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:03 smithi078 ceph-mon[91136]: pgmap v11491: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:43:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:03 smithi078 ceph-mon[99959]: pgmap v11491: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:43:05.489 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:43:06.012 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:05 smithi078 ceph-mon[91136]: pgmap v11492: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:06.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:05 smithi078 ceph-mon[99959]: pgmap v11492: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 6h - - 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 6h - - 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 12s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 7s ago 6h 89.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 12s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 7s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 12s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 12s ago 6h 549M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 7s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 12s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:43:06.013 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 12s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 7s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 12s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 12s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 12s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 7s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 7s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 7s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 7s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 7s ago 6h 77.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 12s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 12s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:43:06.014 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 7s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:43:06.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:05 smithi146 ceph-mon[103278]: pgmap v11492: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:06.324 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:43:06.324 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:43:06.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:43:06.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:06.324 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:43:06.325 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:43:06.593 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:43:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:43:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:43:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:43:06.594 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:43:06.909 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:43:06.909 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:06.909 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:06.909 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:06.910 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[99959]: from='client.70107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[99959]: from='client.90032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[99959]: from='client.90038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/596258072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[91136]: from='client.70107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[91136]: from='client.90032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:06.910 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[91136]: from='client.90038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:06.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/596258072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:07.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:06 smithi146 ceph-mon[103278]: from='client.70107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:06 smithi146 ceph-mon[103278]: from='client.90032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:06 smithi146 ceph-mon[103278]: from='client.90038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/596258072' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:07 smithi146 ceph-mon[103278]: from='client.70131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:07 smithi146 ceph-mon[103278]: pgmap v11493: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3866551237' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:07 smithi078 ceph-mon[99959]: from='client.70131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:07 smithi078 ceph-mon[99959]: pgmap v11493: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3866551237' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:07 smithi078 ceph-mon[91136]: from='client.70131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:07 smithi078 ceph-mon[91136]: pgmap v11493: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3866551237' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:10.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:09 smithi146 ceph-mon[103278]: pgmap v11494: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:09 smithi078 ceph-mon[91136]: pgmap v11494: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:09 smithi078 ceph-mon[99959]: pgmap v11494: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:43:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:43:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:11 smithi146 ceph-mon[103278]: pgmap v11495: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:11 smithi078 ceph-mon[91136]: pgmap v11495: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:11 smithi078 ceph-mon[99959]: pgmap v11495: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:11.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:11.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:11.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:14.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:13 smithi146 ceph-mon[103278]: pgmap v11496: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[91136]: pgmap v11496: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[99959]: pgmap v11496: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:16.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:15 smithi146 ceph-mon[103278]: pgmap v11497: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:15 smithi078 ceph-mon[91136]: pgmap v11497: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:15 smithi078 ceph-mon[99959]: pgmap v11497: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:18.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:17 smithi146 ceph-mon[103278]: pgmap v11498: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:17 smithi078 ceph-mon[99959]: pgmap v11498: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:17 smithi078 ceph-mon[91136]: pgmap v11498: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:19 smithi146 ceph-mon[103278]: pgmap v11499: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[91136]: pgmap v11499: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:20.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[99959]: pgmap v11499: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:43:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:43:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:21 smithi146 ceph-mon[103278]: pgmap v11500: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:21 smithi078 ceph-mon[91136]: pgmap v11500: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:21 smithi078 ceph-mon[99959]: pgmap v11500: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:21.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:21.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:21.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:23 smithi078 ceph-mon[91136]: pgmap v11501: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:23 smithi078 ceph-mon[99959]: pgmap v11501: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:23 smithi146 ceph-mon[103278]: pgmap v11501: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:26.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:25 smithi146 ceph-mon[103278]: pgmap v11502: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:25 smithi078 ceph-mon[91136]: pgmap v11502: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:25 smithi078 ceph-mon[99959]: pgmap v11502: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:28.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:27 smithi146 ceph-mon[103278]: pgmap v11503: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:27 smithi078 ceph-mon[91136]: pgmap v11503: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:28.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:27 smithi078 ceph-mon[99959]: pgmap v11503: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:29.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:29.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:29.030 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:30.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:29 smithi146 ceph-mon[103278]: pgmap v11504: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:29 smithi078 ceph-mon[99959]: pgmap v11504: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:29 smithi078 ceph-mon[91136]: pgmap v11504: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:43:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:43:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:31 smithi146 ceph-mon[103278]: pgmap v11505: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:31 smithi078 ceph-mon[91136]: pgmap v11505: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:31 smithi078 ceph-mon[99959]: pgmap v11505: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:31.772Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:31.772Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:31.773Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:33.848 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:33 smithi146 ceph-mon[103278]: pgmap v11506: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:33 smithi078 ceph-mon[91136]: pgmap v11506: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:33 smithi078 ceph-mon[99959]: pgmap v11506: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:35.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:36.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:35 smithi146 ceph-mon[103278]: pgmap v11507: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:35 smithi078 ceph-mon[91136]: pgmap v11507: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:35 smithi078 ceph-mon[99959]: pgmap v11507: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:37.211 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:43:37.733 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:43:37.733 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 6h - - 2024-09-21T02:43:37.733 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 6h - - 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 3s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 19s ago 6h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 3s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 19s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 3s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 3s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 19s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 3s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 3s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 19s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 3s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 3s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 3s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:43:37.734 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 3s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 19s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 19s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 19s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 19s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 19s ago 6h 77.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 3s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 3s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:43:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 19s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:43:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:37 smithi078 ceph-mon[91136]: pgmap v11508: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:37 smithi078 ceph-mon[91136]: from='client.70143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:37 smithi078 ceph-mon[99959]: pgmap v11508: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:38.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:37 smithi078 ceph-mon[99959]: from='client.70143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:38.044 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:43:38.044 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:43:38.044 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:43:38.045 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:43:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:37 smithi146 ceph-mon[103278]: pgmap v11508: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:37 smithi146 ceph-mon[103278]: from='client.70143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:43:38.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:43:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:43:38.621 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:43:38.621 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:38.621 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:43:38.621 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:38.621 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:43:39.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='client.90068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='client.70155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/134766411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2337428703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='client.90068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='client.70155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/134766411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2337428703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='client.90068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='client.70155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/134766411' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:43:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2337428703' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:43:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:40.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:39 smithi146 ceph-mon[103278]: from='client.70167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:39 smithi146 ceph-mon[103278]: pgmap v11509: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:39 smithi078 ceph-mon[99959]: from='client.70167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:39 smithi078 ceph-mon[99959]: pgmap v11509: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:39 smithi078 ceph-mon[91136]: from='client.70167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:43:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:39 smithi078 ceph-mon[91136]: pgmap v11509: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:43:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:43:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:41 smithi078 ceph-mon[91136]: pgmap v11510: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:41 smithi078 ceph-mon[99959]: pgmap v11510: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:41.773Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:41.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:42.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:41.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:41 smithi146 ceph-mon[103278]: pgmap v11510: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:44.160 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:43 smithi078 ceph-mon[91136]: pgmap v11511: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:44.160 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:44.161 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:43 smithi078 ceph-mon[99959]: pgmap v11511: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:44.161 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:44.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:43 smithi146 ceph-mon[103278]: pgmap v11511: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:46.073 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:45 smithi078 ceph-mon[99959]: pgmap v11512: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:46.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:45 smithi078 ceph-mon[91136]: pgmap v11512: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:45 smithi146 ceph-mon[103278]: pgmap v11512: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:47 smithi146 ceph-mon[103278]: pgmap v11513: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:47 smithi078 ceph-mon[91136]: pgmap v11513: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:47 smithi078 ceph-mon[99959]: pgmap v11513: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:43:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:49 smithi146 ceph-mon[103278]: pgmap v11514: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:49 smithi078 ceph-mon[99959]: pgmap v11514: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:49 smithi078 ceph-mon[91136]: pgmap v11514: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:43:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:43:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:43:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:51 smithi078 ceph-mon[91136]: pgmap v11515: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:51 smithi078 ceph-mon[99959]: pgmap v11515: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:51.774Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:51.774Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:43:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:43:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:43:51.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:43:52.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:51 smithi146 ceph-mon[103278]: pgmap v11515: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:53 smithi146 ceph-mon[103278]: pgmap v11516: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:53 smithi078 ceph-mon[91136]: pgmap v11516: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:53 smithi078 ceph-mon[99959]: pgmap v11516: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:55.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:55 smithi146 ceph-mon[103278]: pgmap v11517: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:55 smithi078 ceph-mon[91136]: pgmap v11517: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:55 smithi078 ceph-mon[99959]: pgmap v11517: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:43:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:57 smithi146 ceph-mon[103278]: pgmap v11518: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:57 smithi078 ceph-mon[99959]: pgmap v11518: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:57 smithi078 ceph-mon[91136]: pgmap v11518: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:43:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:43:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:43:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:59 smithi146 ceph-mon[103278]: pgmap v11519: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:43:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:59 smithi078 ceph-mon[91136]: pgmap v11519: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:43:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:59 smithi078 ceph-mon[99959]: pgmap v11519: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:43:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:44:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:44:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:01 smithi078 ceph-mon[91136]: pgmap v11520: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:01 smithi078 ceph-mon[99959]: pgmap v11520: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:01.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:01.775Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:01.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:01 smithi146 ceph-mon[103278]: pgmap v11520: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:03 smithi146 ceph-mon[103278]: pgmap v11521: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:03 smithi078 ceph-mon[91136]: pgmap v11521: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:03 smithi078 ceph-mon[99959]: pgmap v11521: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:05 smithi146 ceph-mon[103278]: pgmap v11522: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:05 smithi078 ceph-mon[91136]: pgmap v11522: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:05 smithi078 ceph-mon[99959]: pgmap v11522: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:07 smithi146 ceph-mon[103278]: pgmap v11523: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:07 smithi078 ceph-mon[91136]: pgmap v11523: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:07 smithi078 ceph-mon[99959]: pgmap v11523: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:08.914 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:44:09.440 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 6h - - 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 6h - - 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 15s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 10s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 15s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 10s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 15s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 15s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 10s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 15s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 15s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 10s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:44:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 15s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 15s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 15s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 10s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 10s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 10s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 10s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 10s ago 6h 77.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 15s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 15s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:44:09.442 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 10s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:44:09.747 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:44:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:44:10.016 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:44:10.017 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:44:10.017 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:44:10.017 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:44:10.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:44:10.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[99959]: pgmap v11524: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:10.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[99959]: from='client.70179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:10.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:10.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[99959]: from='client.90104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:10.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/206010034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:10.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[91136]: pgmap v11524: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:10.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[91136]: from='client.70179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:10.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:10.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[91136]: from='client.90104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:10.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/206010034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:09 smithi146 ceph-mon[103278]: pgmap v11524: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:09 smithi146 ceph-mon[103278]: from='client.70179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:09 smithi146 ceph-mon[103278]: from='client.90104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/206010034' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:10.325 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:44:10.325 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:10.326 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:10.326 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:10.326 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:10.963 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:44:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:44:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:10 smithi146 ceph-mon[103278]: from='client.70191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:10 smithi146 ceph-mon[103278]: from='client.90122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1839890442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-mon[91136]: from='client.70191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-mon[91136]: from='client.90122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1839890442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-mon[99959]: from='client.70191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-mon[99959]: from='client.90122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1839890442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:11 smithi078 ceph-mon[91136]: pgmap v11525: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:11 smithi078 ceph-mon[99959]: pgmap v11525: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:11.775Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:11.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:11.776Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:11 smithi146 ceph-mon[103278]: pgmap v11525: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:14.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:13 smithi146 ceph-mon[103278]: pgmap v11526: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:14.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:13 smithi078 ceph-mon[91136]: pgmap v11526: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:13 smithi078 ceph-mon[99959]: pgmap v11526: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:16 smithi146 ceph-mon[103278]: pgmap v11527: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:16 smithi078 ceph-mon[91136]: pgmap v11527: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:16 smithi078 ceph-mon[99959]: pgmap v11527: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:18 smithi146 ceph-mon[103278]: pgmap v11528: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:18 smithi078 ceph-mon[91136]: pgmap v11528: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:18 smithi078 ceph-mon[99959]: pgmap v11528: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:19 smithi146 ceph-mon[103278]: pgmap v11529: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[99959]: pgmap v11529: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[91136]: pgmap v11529: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:44:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:44:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:21 smithi078 ceph-mon[91136]: pgmap v11530: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:21 smithi078 ceph-mon[99959]: pgmap v11530: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:21.776Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:21.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:21.777Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:21 smithi146 ceph-mon[103278]: pgmap v11530: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:24.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:23 smithi146 ceph-mon[103278]: pgmap v11531: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:23 smithi078 ceph-mon[91136]: pgmap v11531: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:23 smithi078 ceph-mon[99959]: pgmap v11531: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:25.983 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:25 smithi078 ceph-mon[99959]: pgmap v11532: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:25.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:25 smithi078 ceph-mon[91136]: pgmap v11532: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:25 smithi146 ceph-mon[103278]: pgmap v11532: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 02:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T02:44:25.849255424Z level=info msg="Completed cleanup jobs" duration=13.191559ms 2024-09-21T02:44:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:27 smithi146 ceph-mon[103278]: pgmap v11533: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:27 smithi078 ceph-mon[91136]: pgmap v11533: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:27 smithi078 ceph-mon[99959]: pgmap v11533: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:29 smithi146 ceph-mon[103278]: pgmap v11534: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:29 smithi078 ceph-mon[91136]: pgmap v11534: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:29 smithi078 ceph-mon[99959]: pgmap v11534: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:44:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:44:31.778 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:31 smithi078 ceph-mon[99959]: pgmap v11535: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:31.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:31 smithi078 ceph-mon[91136]: pgmap v11535: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:31 smithi146 ceph-mon[103278]: pgmap v11535: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:31.777Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:31.778Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:31.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:33 smithi078 ceph-mon[91136]: pgmap v11536: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:33 smithi078 ceph-mon[99959]: pgmap v11536: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:34.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:33 smithi146 ceph-mon[103278]: pgmap v11536: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:35.302 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.302 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.303 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:36 smithi146 ceph-mon[103278]: pgmap v11537: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:36 smithi078 ceph-mon[91136]: pgmap v11537: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:36 smithi078 ceph-mon[99959]: pgmap v11537: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:38.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:38 smithi146 ceph-mon[103278]: pgmap v11538: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:38 smithi078 ceph-mon[91136]: pgmap v11538: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:38 smithi078 ceph-mon[99959]: pgmap v11538: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:40 smithi146 ceph-mon[103278]: pgmap v11539: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[91136]: pgmap v11539: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[99959]: pgmap v11539: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:40.622 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:44:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:44:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:44:41.148 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:44:41.148 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 6h - - 2024-09-21T02:44:41.148 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 6s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 2s ago 6h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 6s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 6s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 6s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 6s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 6s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 2s ago 6h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 6s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 6s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 6s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:44:41.149 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 6s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 6h 77.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 6s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 6s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:44:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 2s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:44:41.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:44:41.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:44:41.462 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:44:41.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:44:41.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:44:41.462 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:44:41.731 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:44:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:44:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:44:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:44:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:44:41.732 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:44:41.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:44:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:41.778Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:41.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:41.779Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:42.043 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:44:42.043 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:42.043 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:44:42.043 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:42.043 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:44:42.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:42 smithi146 ceph-mon[103278]: from='client.90134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:42 smithi146 ceph-mon[103278]: pgmap v11540: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:42 smithi146 ceph-mon[103278]: from='client.70221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:42 smithi146 ceph-mon[103278]: from='client.90146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1945194220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2839289157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[91136]: from='client.90134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[91136]: pgmap v11540: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[91136]: from='client.70221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[91136]: from='client.90146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1945194220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2839289157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[99959]: from='client.90134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[99959]: pgmap v11540: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[99959]: from='client.70221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[99959]: from='client.90146 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1945194220' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:44:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2839289157' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:44:43.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:43 smithi146 ceph-mon[103278]: from='client.70239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:43 smithi078 ceph-mon[91136]: from='client.70239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:43 smithi078 ceph-mon[99959]: from='client.70239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:44:44.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:44 smithi078 ceph-mon[91136]: pgmap v11541: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:44.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:44.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:44 smithi078 ceph-mon[99959]: pgmap v11541: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:44.356 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:44 smithi146 ceph-mon[103278]: pgmap v11541: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:46 smithi078 ceph-mon[91136]: pgmap v11542: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:46 smithi078 ceph-mon[99959]: pgmap v11542: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:46 smithi146 ceph-mon[103278]: pgmap v11542: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:48 smithi146 ceph-mon[103278]: pgmap v11543: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:48 smithi078 ceph-mon[91136]: pgmap v11543: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:48 smithi078 ceph-mon[99959]: pgmap v11543: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:50 smithi146 ceph-mon[103278]: pgmap v11544: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:50 smithi078 ceph-mon[91136]: pgmap v11544: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:50 smithi078 ceph-mon[99959]: pgmap v11544: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:44:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:44:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:44:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:51.779Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:51.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:44:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:44:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:44:51.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:44:52.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:52 smithi078 ceph-mon[99959]: pgmap v11545: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:52.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:52 smithi078 ceph-mon[91136]: pgmap v11545: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:52 smithi146 ceph-mon[103278]: pgmap v11545: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:54.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:54 smithi146 ceph-mon[103278]: pgmap v11546: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:54 smithi078 ceph-mon[91136]: pgmap v11546: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:54 smithi078 ceph-mon[99959]: pgmap v11546: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:55.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:55 smithi146 ceph-mon[103278]: pgmap v11547: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[91136]: pgmap v11547: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:44:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:55 smithi078 ceph-mon[99959]: pgmap v11547: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:44:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:57 smithi078 ceph-mon[91136]: pgmap v11548: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:58.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:57 smithi078 ceph-mon[99959]: pgmap v11548: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:57 smithi146 ceph-mon[103278]: pgmap v11548: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:44:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:44:59.233 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:00 smithi146 ceph-mon[103278]: pgmap v11549: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[91136]: pgmap v11549: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[99959]: pgmap v11549: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:45:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:45:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:01.780Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:01.780Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:01.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:02 smithi146 ceph-mon[103278]: pgmap v11550: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:02 smithi078 ceph-mon[99959]: pgmap v11550: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:02 smithi078 ceph-mon[91136]: pgmap v11550: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:04 smithi146 ceph-mon[103278]: pgmap v11551: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:45:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:04 smithi078 ceph-mon[91136]: pgmap v11551: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:45:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:04 smithi078 ceph-mon[99959]: pgmap v11551: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:45:06.394 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:06 smithi078 ceph-mon[91136]: pgmap v11552: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:06.394 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:06 smithi078 ceph-mon[99959]: pgmap v11552: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:06 smithi146 ceph-mon[103278]: pgmap v11552: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:08 smithi146 ceph-mon[103278]: pgmap v11553: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:45:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:08 smithi078 ceph-mon[91136]: pgmap v11553: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:45:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:08 smithi078 ceph-mon[99959]: pgmap v11553: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:45:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:10 smithi146 ceph-mon[103278]: pgmap v11554: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:10 smithi078 ceph-mon[91136]: pgmap v11554: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:10 smithi078 ceph-mon[99959]: pgmap v11554: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:45:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:45:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:11.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:11.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:11.781Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:12.340 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:45:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:12 smithi146 ceph-mon[103278]: pgmap v11555: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:12 smithi078 ceph-mon[91136]: pgmap v11555: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:12 smithi078 ceph-mon[99959]: pgmap v11555: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:12.866 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:45:12.866 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T02:45:12.866 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T02:45:12.866 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 18s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 13s ago 6h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 18s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 13s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:45:12.867 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 13s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 13s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 13s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 6h 77.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:45:12.868 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:45:13.173 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:45:13.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:45:13.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:45:13.175 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:45:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/963374839' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:13.443 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:45:13.444 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:45:13.445 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/963374839' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:13.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/963374839' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:13.757 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:45:13.757 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:13.757 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:14 smithi146 ceph-mon[103278]: from='client.70251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:14 smithi146 ceph-mon[103278]: from='client.90176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:14 smithi146 ceph-mon[103278]: pgmap v11556: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:14 smithi146 ceph-mon[103278]: from='client.70263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3591183808' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[91136]: from='client.70251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[91136]: from='client.90176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[91136]: pgmap v11556: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[91136]: from='client.70263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3591183808' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[99959]: from='client.70251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[99959]: from='client.90176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[99959]: pgmap v11556: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[99959]: from='client.70263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3591183808' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:15.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:15 smithi146 ceph-mon[103278]: from='client.70275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:15.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.440 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[91136]: from='client.70275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[99959]: from='client.70275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:16.284 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:16 smithi146 ceph-mon[103278]: pgmap v11557: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:16 smithi078 ceph-mon[91136]: pgmap v11557: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:16 smithi078 ceph-mon[99959]: pgmap v11557: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:18 smithi146 ceph-mon[103278]: pgmap v11558: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:18 smithi078 ceph-mon[91136]: pgmap v11558: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:18 smithi078 ceph-mon[99959]: pgmap v11558: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:20 smithi146 ceph-mon[103278]: pgmap v11559: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[91136]: pgmap v11559: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[99959]: pgmap v11559: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:45:20] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:45:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:21.781Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:21.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:21.782Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:22 smithi146 ceph-mon[103278]: pgmap v11560: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:22 smithi078 ceph-mon[91136]: pgmap v11560: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:22 smithi078 ceph-mon[99959]: pgmap v11560: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:24 smithi146 ceph-mon[103278]: pgmap v11561: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:24 smithi078 ceph-mon[91136]: pgmap v11561: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:24 smithi078 ceph-mon[99959]: pgmap v11561: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:26 smithi146 ceph-mon[103278]: pgmap v11562: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:26 smithi078 ceph-mon[91136]: pgmap v11562: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:26 smithi078 ceph-mon[99959]: pgmap v11562: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:28 smithi078 ceph-mon[91136]: pgmap v11563: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:28 smithi078 ceph-mon[99959]: pgmap v11563: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:28 smithi146 ceph-mon[103278]: pgmap v11563: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:30 smithi078 ceph-mon[91136]: pgmap v11564: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:30 smithi078 ceph-mon[99959]: pgmap v11564: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:30 smithi146 ceph-mon[103278]: pgmap v11564: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:45:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:45:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:31.782Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:31.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:31.783Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:32 smithi078 ceph-mon[99959]: pgmap v11565: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:32 smithi078 ceph-mon[91136]: pgmap v11565: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:32.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:32 smithi146 ceph-mon[103278]: pgmap v11565: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:34 smithi078 ceph-mon[99959]: pgmap v11566: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:34 smithi078 ceph-mon[91136]: pgmap v11566: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:34.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:34 smithi146 ceph-mon[103278]: pgmap v11566: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:35 smithi146 ceph-mon[103278]: pgmap v11567: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[91136]: pgmap v11567: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:35 smithi078 ceph-mon[99959]: pgmap v11567: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:37 smithi146 ceph-mon[103278]: pgmap v11568: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:37 smithi078 ceph-mon[91136]: pgmap v11568: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:37 smithi078 ceph-mon[99959]: pgmap v11568: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:39 smithi146 ceph-mon[103278]: pgmap v11569: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[91136]: pgmap v11569: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[99959]: pgmap v11569: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:45:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:45:42.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:41 smithi078 ceph-mon[91136]: pgmap v11570: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:42.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:41 smithi078 ceph-mon[99959]: pgmap v11570: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:41.783Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:41.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:42.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:41.784Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:41 smithi146 ceph-mon[103278]: pgmap v11570: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:44.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:43 smithi078 ceph-mon[91136]: pgmap v11571: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:44.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:44.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:43 smithi078 ceph-mon[99959]: pgmap v11571: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:44.056 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:44.056 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:45:44.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:43 smithi146 ceph-mon[103278]: pgmap v11571: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:44.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:45:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:45:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 6h - - 2024-09-21T02:45:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:45:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 5s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:45:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 5s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 5s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 5s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 9s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 9s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 9s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:45:44.582 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 5s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 5s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 5s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 5s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 5s ago 6h 77.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:45:44.583 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 5s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:45:44.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:44 smithi078 ceph-mon[91136]: from='client.70287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:44.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:44 smithi078 ceph-mon[99959]: from='client.70287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:45:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:45:44.897 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:45:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:45:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:45:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:45:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:45:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:45:45.166 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:45:45.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:44 smithi146 ceph-mon[103278]: from='client.70287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:45.474 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:45:45.474 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:45:45.474 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:45:46.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[91136]: from='client.70293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[91136]: from='client.90218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[91136]: pgmap v11572: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:46.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/892660384' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:46.098 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[91136]: from='client.70311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1409053803' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[99959]: from='client.70293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[99959]: from='client.90218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[99959]: pgmap v11572: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/892660384' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[99959]: from='client.70311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1409053803' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:45 smithi146 ceph-mon[103278]: from='client.70293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:45 smithi146 ceph-mon[103278]: from='client.90218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:45 smithi146 ceph-mon[103278]: pgmap v11572: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/892660384' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:45:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:45 smithi146 ceph-mon[103278]: from='client.70311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:45:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1409053803' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:45:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:47 smithi078 ceph-mon[91136]: pgmap v11573: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:47 smithi078 ceph-mon[99959]: pgmap v11573: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:47 smithi146 ceph-mon[103278]: pgmap v11573: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:49 smithi146 ceph-mon[103278]: pgmap v11574: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:49 smithi078 ceph-mon[91136]: pgmap v11574: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:49 smithi078 ceph-mon[99959]: pgmap v11574: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:45:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:45:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:45:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:51 smithi146 ceph-mon[103278]: pgmap v11575: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:51 smithi078 ceph-mon[91136]: pgmap v11575: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:51 smithi078 ceph-mon[99959]: pgmap v11575: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:51.784Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:51.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:45:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:45:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:45:51.785Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:45:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:53 smithi146 ceph-mon[103278]: pgmap v11576: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:53 smithi078 ceph-mon[99959]: pgmap v11576: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:53 smithi078 ceph-mon[91136]: pgmap v11576: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:55.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:55.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:55.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:55.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:55 smithi146 ceph-mon[103278]: pgmap v11577: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[99959]: pgmap v11577: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:45:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:55 smithi078 ceph-mon[91136]: pgmap v11577: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:45:57.881 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:57 smithi146 ceph-mon[103278]: pgmap v11578: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:57 smithi078 ceph-mon[91136]: pgmap v11578: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:57 smithi078 ceph-mon[99959]: pgmap v11578: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:45:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:45:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:59 smithi146 ceph-mon[103278]: pgmap v11579: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:45:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[99959]: pgmap v11579: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[91136]: pgmap v11579: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:45:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:46:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:46:02.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:01 smithi146 ceph-mon[103278]: pgmap v11580: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:01 smithi078 ceph-mon[91136]: pgmap v11580: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:01 smithi078 ceph-mon[99959]: pgmap v11580: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:01.786Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:01.786Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:01.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:04.067 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:03 smithi146 ceph-mon[103278]: pgmap v11581: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:03 smithi078 ceph-mon[99959]: pgmap v11581: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:03 smithi078 ceph-mon[91136]: pgmap v11581: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:05 smithi146 ceph-mon[103278]: pgmap v11582: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:05 smithi078 ceph-mon[91136]: pgmap v11582: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:05 smithi078 ceph-mon[99959]: pgmap v11582: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:07 smithi078 ceph-mon[91136]: pgmap v11583: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:07 smithi078 ceph-mon[99959]: pgmap v11583: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:07 smithi146 ceph-mon[103278]: pgmap v11583: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:09 smithi146 ceph-mon[103278]: pgmap v11584: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:09 smithi078 ceph-mon[99959]: pgmap v11584: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:09 smithi078 ceph-mon[91136]: pgmap v11584: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:46:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:46:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:11 smithi146 ceph-mon[103278]: pgmap v11585: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:11 smithi078 ceph-mon[91136]: pgmap v11585: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:11 smithi078 ceph-mon[99959]: pgmap v11585: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:11.787Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:11.787Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:11.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:13 smithi146 ceph-mon[103278]: pgmap v11586: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:13 smithi078 ceph-mon[91136]: pgmap v11586: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:13 smithi078 ceph-mon[99959]: pgmap v11586: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:15.766 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[91136]: pgmap v11587: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[99959]: pgmap v11587: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:15 smithi146 ceph-mon[103278]: pgmap v11587: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:16.294 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:46:16.294 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 6h - - 2024-09-21T02:46:16.294 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T02:46:16.294 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 1s ago 6h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:46:16.294 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 1s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 1s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 1s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 6h 372M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 1s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 1s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 1s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 1s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 1s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 1s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 16s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:46:16.295 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 16s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:46:16.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 16s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:46:16.296 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 6h 77.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:46:16.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 1s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:46:16.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:46:16.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 1s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:46:16.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 16s ago 6h 89.9M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:46:16.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:46:16.605 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:46:16.876 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:46:16.877 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:46:16.877 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:46:16.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:46:16.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[91136]: from='client.70323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[91136]: from='client.90248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[91136]: from='client.90254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2733028335' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:16.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[99959]: from='client.70323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[99959]: from='client.90248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[99959]: from='client.90254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.878 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2733028335' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:16.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:16 smithi146 ceph-mon[103278]: from='client.70323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:16 smithi146 ceph-mon[103278]: from='client.90248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:16 smithi146 ceph-mon[103278]: from='client.90254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:16.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2733028335' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:17.187 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:46:17.187 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:17.187 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:17 smithi146 ceph-mon[103278]: pgmap v11588: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:17 smithi146 ceph-mon[103278]: from='client.70347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/365995039' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:17 smithi078 ceph-mon[91136]: pgmap v11588: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:17 smithi078 ceph-mon[91136]: from='client.70347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/365995039' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:17 smithi078 ceph-mon[99959]: pgmap v11588: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:17 smithi078 ceph-mon[99959]: from='client.70347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/365995039' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:19 smithi146 ceph-mon[103278]: pgmap v11589: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[91136]: pgmap v11589: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[99959]: pgmap v11589: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:46:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:46:22.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:21 smithi078 ceph-mon[91136]: pgmap v11590: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:22.236 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:21 smithi078 ceph-mon[99959]: pgmap v11590: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:22.236 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:21.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:22.236 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:21.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:22.236 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:21.788Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:21 smithi146 ceph-mon[103278]: pgmap v11590: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:23 smithi146 ceph-mon[103278]: pgmap v11591: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:23 smithi078 ceph-mon[91136]: pgmap v11591: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:23 smithi078 ceph-mon[99959]: pgmap v11591: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:25 smithi146 ceph-mon[103278]: pgmap v11592: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:25 smithi078 ceph-mon[91136]: pgmap v11592: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:25 smithi078 ceph-mon[99959]: pgmap v11592: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:27.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:27 smithi078 ceph-mon[99959]: pgmap v11593: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:27.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:27 smithi078 ceph-mon[91136]: pgmap v11593: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:27 smithi146 ceph-mon[103278]: pgmap v11593: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:29.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:29.132 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:30.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:29 smithi078 ceph-mon[91136]: pgmap v11594: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:30.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:30.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:29 smithi078 ceph-mon[99959]: pgmap v11594: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:30.194 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:29 smithi146 ceph-mon[103278]: pgmap v11594: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:46:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:46:32.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:31 smithi146 ceph-mon[103278]: pgmap v11595: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:31 smithi078 ceph-mon[91136]: pgmap v11595: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:31 smithi078 ceph-mon[99959]: pgmap v11595: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:31.788Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:31.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:31.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:34.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:33 smithi146 ceph-mon[103278]: pgmap v11596: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:33 smithi078 ceph-mon[99959]: pgmap v11596: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:33 smithi078 ceph-mon[91136]: pgmap v11596: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:36.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:35 smithi146 ceph-mon[103278]: pgmap v11597: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:36.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[99959]: pgmap v11597: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[91136]: pgmap v11597: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:38.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:37 smithi146 ceph-mon[103278]: pgmap v11598: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:37 smithi078 ceph-mon[99959]: pgmap v11598: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:37 smithi078 ceph-mon[91136]: pgmap v11598: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:39 smithi146 ceph-mon[103278]: pgmap v11599: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:39 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[91136]: pgmap v11599: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[99959]: pgmap v11599: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:46:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:46:42.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:41 smithi146 ceph-mon[103278]: pgmap v11600: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:41 smithi078 ceph-mon[91136]: pgmap v11600: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:41 smithi078 ceph-mon[99959]: pgmap v11600: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:41.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:41.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:41.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:43 smithi146 ceph-mon[103278]: pgmap v11601: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:43 smithi078 ceph-mon[91136]: pgmap v11601: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:43 smithi078 ceph-mon[99959]: pgmap v11601: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:45 smithi146 ceph-mon[103278]: pgmap v11602: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:45 smithi078 ceph-mon[91136]: pgmap v11602: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:45 smithi078 ceph-mon[99959]: pgmap v11602: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:47.490 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 6h - - 2024-09-21T02:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 6h - - 2024-09-21T02:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 13s ago 6h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 8s ago 6h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 13s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 8s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 13s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 13s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 8s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 13s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 13s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 8s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 13s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 13s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 13s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 13s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 8s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 8s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 8s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 8s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 8s ago 6h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 13s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 13s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:46:48.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:47 smithi146 ceph-mon[103278]: pgmap v11603: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:47 smithi078 ceph-mon[91136]: pgmap v11603: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:47 smithi078 ceph-mon[99959]: pgmap v11603: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:46:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:46:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:46:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:46:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:46:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:46:48.332 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:46:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:46:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:46:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:46:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:46:48.610 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:46:48.921 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:46:48.921 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:48.922 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:46:48.922 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:48.922 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[91136]: from='client.70359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[91136]: from='client.90284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[91136]: from='client.70371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1942401948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2907619625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[99959]: from='client.70359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[99959]: from='client.90284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[99959]: from='client.70371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1942401948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:49.212 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2907619625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:49 smithi146 ceph-mon[103278]: from='client.70359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:49 smithi146 ceph-mon[103278]: from='client.90284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:49 smithi146 ceph-mon[103278]: from='client.70371 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1942401948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:46:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2907619625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:46:50.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:50 smithi146 ceph-mon[103278]: from='client.90302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:50 smithi146 ceph-mon[103278]: pgmap v11604: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:50.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-mon[99959]: from='client.90302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:50.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-mon[99959]: pgmap v11604: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:50.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:50.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-mon[91136]: from='client.90302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:46:50.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-mon[91136]: pgmap v11604: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:50.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:46:50.934 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:46:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:46:52.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:52 smithi078 ceph-mon[99959]: pgmap v11605: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:52 smithi078 ceph-mon[91136]: pgmap v11605: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:51.789Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:51.789Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:46:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:46:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:46:51.790Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:46:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:52 smithi146 ceph-mon[103278]: pgmap v11605: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:54 smithi146 ceph-mon[103278]: pgmap v11606: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:54 smithi078 ceph-mon[91136]: pgmap v11606: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:54 smithi078 ceph-mon[99959]: pgmap v11606: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:56 smithi146 ceph-mon[103278]: pgmap v11607: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[91136]: pgmap v11607: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[99959]: pgmap v11607: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:46:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:46:58.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:58 smithi146 ceph-mon[103278]: pgmap v11608: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:58 smithi078 ceph-mon[91136]: pgmap v11608: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:58 smithi078 ceph-mon[99959]: pgmap v11608: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:46:59.252 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:46:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:00 smithi146 ceph-mon[103278]: pgmap v11609: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[91136]: pgmap v11609: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[99959]: pgmap v11609: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:47:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:47:02.131 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:01.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:02.131 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:01.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:02.132 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:01.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:02 smithi146 ceph-mon[103278]: pgmap v11610: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:02 smithi078 ceph-mon[91136]: pgmap v11610: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:02 smithi078 ceph-mon[99959]: pgmap v11610: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:04 smithi146 ceph-mon[103278]: pgmap v11611: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:04 smithi078 ceph-mon[99959]: pgmap v11611: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:04 smithi078 ceph-mon[91136]: pgmap v11611: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:06.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:06 smithi146 ceph-mon[103278]: pgmap v11612: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:06 smithi078 ceph-mon[99959]: pgmap v11612: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:06 smithi078 ceph-mon[91136]: pgmap v11612: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:08 smithi146 ceph-mon[103278]: pgmap v11613: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:08 smithi078 ceph-mon[99959]: pgmap v11613: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:08 smithi078 ceph-mon[91136]: pgmap v11613: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:10.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:10 smithi078 ceph-mon[99959]: pgmap v11614: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:10.377 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:10.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:10 smithi078 ceph-mon[91136]: pgmap v11614: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:10.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:10 smithi146 ceph-mon[103278]: pgmap v11614: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:10.629 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:47:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:47:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:11.790Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:11.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:11.791Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:12.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:12 smithi146 ceph-mon[103278]: pgmap v11615: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:47:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:12 smithi078 ceph-mon[99959]: pgmap v11615: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:47:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:12 smithi078 ceph-mon[91136]: pgmap v11615: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:47:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:14 smithi146 ceph-mon[103278]: pgmap v11616: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:47:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:14 smithi078 ceph-mon[91136]: pgmap v11616: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:47:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:14 smithi078 ceph-mon[99959]: pgmap v11616: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:47:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:16 smithi146 ceph-mon[103278]: pgmap v11617: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[91136]: pgmap v11617: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[99959]: pgmap v11617: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:18 smithi146 ceph-mon[103278]: pgmap v11618: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:18 smithi078 ceph-mon[91136]: pgmap v11618: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:18 smithi078 ceph-mon[99959]: pgmap v11618: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:19.215 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 6h - - 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 6h - - 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 4s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 19s ago 6h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 4s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 19s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 4s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 4s ago 6h 550M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:47:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 19s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 4s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 4s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 19s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 4s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 4s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 4s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 4s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 19s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 19s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 19s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 19s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 19s ago 6h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 4s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:47:19.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:47:19.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 4s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:47:19.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:47:20.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:47:20.049 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:47:20.321 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:47:20.322 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: pgmap v11619: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: from='client.70395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3808623462' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: pgmap v11619: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: from='client.70395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:20.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3808623462' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:20.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.324 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: pgmap v11619: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: from='client.70395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3808623462' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.468 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:20.635 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:47:20.635 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:20.635 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:47:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:47:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:21 smithi146 ceph-mon[103278]: from='client.90320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:21 smithi146 ceph-mon[103278]: from='client.90326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3761155329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-mon[91136]: from='client.90320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-mon[91136]: from='client.90326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3761155329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-mon[99959]: from='client.90320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-mon[99959]: from='client.90326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3761155329' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:21.791Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:21.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:21.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:22.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:22 smithi146 ceph-mon[103278]: from='client.90338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:22 smithi146 ceph-mon[103278]: pgmap v11620: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:22 smithi078 ceph-mon[91136]: from='client.90338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:22 smithi078 ceph-mon[91136]: pgmap v11620: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:22 smithi078 ceph-mon[99959]: from='client.90338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:22 smithi078 ceph-mon[99959]: pgmap v11620: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:24 smithi078 ceph-mon[91136]: pgmap v11621: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:24 smithi078 ceph-mon[99959]: pgmap v11621: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:24 smithi146 ceph-mon[103278]: pgmap v11621: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:26 smithi078 ceph-mon[91136]: pgmap v11622: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:26 smithi078 ceph-mon[99959]: pgmap v11622: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:26 smithi146 ceph-mon[103278]: pgmap v11622: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:28 smithi078 ceph-mon[91136]: pgmap v11623: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:28 smithi078 ceph-mon[99959]: pgmap v11623: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:28 smithi146 ceph-mon[103278]: pgmap v11623: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:30.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:30 smithi078 ceph-mon[99959]: pgmap v11624: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:30.477 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:30.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:30 smithi078 ceph-mon[91136]: pgmap v11624: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:30.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:30 smithi146 ceph-mon[103278]: pgmap v11624: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:47:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:47:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:31.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:31.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:31.792Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:32.392 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:32 smithi078 ceph-mon[99959]: pgmap v11625: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:32.392 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:32 smithi078 ceph-mon[91136]: pgmap v11625: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:32 smithi146 ceph-mon[103278]: pgmap v11625: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:34.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:34 smithi146 ceph-mon[103278]: pgmap v11626: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:34 smithi078 ceph-mon[91136]: pgmap v11626: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:34 smithi078 ceph-mon[99959]: pgmap v11626: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:36 smithi146 ceph-mon[103278]: pgmap v11627: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[91136]: pgmap v11627: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[99959]: pgmap v11627: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:38 smithi146 ceph-mon[103278]: pgmap v11628: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:38 smithi078 ceph-mon[91136]: pgmap v11628: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:38 smithi078 ceph-mon[99959]: pgmap v11628: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:40 smithi146 ceph-mon[103278]: pgmap v11629: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:40.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:40 smithi078 ceph-mon[99959]: pgmap v11629: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:40 smithi078 ceph-mon[91136]: pgmap v11629: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:40.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:47:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:47:41.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.612 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:41 smithi146 ceph-mon[103278]: pgmap v11630: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[99959]: pgmap v11630: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-mon[91136]: pgmap v11630: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:41.792Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:41.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:41.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:43 smithi146 ceph-mon[103278]: pgmap v11631: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:43 smithi078 ceph-mon[99959]: pgmap v11631: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:43 smithi078 ceph-mon[91136]: pgmap v11631: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:45 smithi146 ceph-mon[103278]: pgmap v11632: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:45 smithi078 ceph-mon[91136]: pgmap v11632: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:45 smithi078 ceph-mon[99959]: pgmap v11632: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:47 smithi146 ceph-mon[103278]: pgmap v11633: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:47 smithi078 ceph-mon[91136]: pgmap v11633: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:47 smithi078 ceph-mon[99959]: pgmap v11633: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:49 smithi146 ceph-mon[103278]: pgmap v11634: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:49 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:49 smithi078 ceph-mon[91136]: pgmap v11634: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:49 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:49 smithi078 ceph-mon[99959]: pgmap v11634: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:49 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:47:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:47:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:47:50.944 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:47:51.480 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:47:51.480 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 6h - - 2024-09-21T02:47:51.480 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 6h - - 2024-09-21T02:47:51.481 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 16s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:47:51.481 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 11s ago 6h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:47:51.481 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 16s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:47:51.481 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 11s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:47:51.481 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 16s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:47:51.481 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 16s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 11s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 16s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 16s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 11s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 16s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 16s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 16s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 16s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:47:51.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 11s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 11s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 11s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 11s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 11s ago 6h 75.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 16s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 16s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:47:51.483 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:47:51.828 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:47:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:47:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:47:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:47:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:47:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:47:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:47:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:47:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:47:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:47:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:47:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:47:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:47:51.830 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:47:52.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[99959]: pgmap v11635: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:52.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[99959]: from='client.90350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:52.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[99959]: from='client.70437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:52.071 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3185069315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:52.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[91136]: pgmap v11635: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:52.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[91136]: from='client.90350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:52.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[91136]: from='client.70437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:52.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3185069315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:52.072 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:51.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:52.072 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:51.793Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:47:52.072 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:47:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:47:51.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:47:52.108 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:47:52.109 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:47:52.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:51 smithi146 ceph-mon[103278]: pgmap v11635: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:51 smithi146 ceph-mon[103278]: from='client.90350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:51 smithi146 ceph-mon[103278]: from='client.70437 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3185069315' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:47:52.454 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:47:52.454 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:52.454 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:47:52.454 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:52.455 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:47:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:52 smithi078 ceph-mon[91136]: from='client.90362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:52 smithi078 ceph-mon[91136]: from='client.90374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1020422343' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:52 smithi078 ceph-mon[99959]: from='client.90362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:52 smithi078 ceph-mon[99959]: from='client.90374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1020422343' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:52 smithi146 ceph-mon[103278]: from='client.90362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:52 smithi146 ceph-mon[103278]: from='client.90374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:47:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1020422343' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:47:54.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:53 smithi146 ceph-mon[103278]: pgmap v11636: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:53 smithi078 ceph-mon[99959]: pgmap v11636: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:53 smithi078 ceph-mon[91136]: pgmap v11636: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:55 smithi146 ceph-mon[103278]: pgmap v11637: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[99959]: pgmap v11637: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[91136]: pgmap v11637: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:47:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:57 smithi146 ceph-mon[103278]: pgmap v11638: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:57 smithi078 ceph-mon[91136]: pgmap v11638: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:57 smithi078 ceph-mon[99959]: pgmap v11638: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:47:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:59.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:47:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:59 smithi146 ceph-mon[103278]: pgmap v11639: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:47:59 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:59 smithi078 ceph-mon[91136]: pgmap v11639: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:47:59 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:59 smithi078 ceph-mon[99959]: pgmap v11639: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:47:59 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:48:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T02:48:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:01 smithi146 ceph-mon[103278]: pgmap v11640: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[99959]: pgmap v11640: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-mon[91136]: pgmap v11640: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:01.793Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:01.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:01.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:04.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:03 smithi146 ceph-mon[103278]: pgmap v11641: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:03 smithi078 ceph-mon[91136]: pgmap v11641: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:03 smithi078 ceph-mon[99959]: pgmap v11641: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:05 smithi146 ceph-mon[103278]: pgmap v11642: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:05 smithi078 ceph-mon[91136]: pgmap v11642: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:05 smithi078 ceph-mon[99959]: pgmap v11642: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:07 smithi146 ceph-mon[103278]: pgmap v11643: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:07 smithi078 ceph-mon[91136]: pgmap v11643: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:07 smithi078 ceph-mon[99959]: pgmap v11643: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:09 smithi146 ceph-mon[103278]: pgmap v11644: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:09 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:09 smithi078 ceph-mon[91136]: pgmap v11644: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:09 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:09 smithi078 ceph-mon[99959]: pgmap v11644: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:09 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:48:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T02:48:11.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:11 smithi078 ceph-mon[91136]: pgmap v11645: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:11.853 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:11.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:11.853 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:11.794Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:11.853 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:11.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:11 smithi146 ceph-mon[103278]: pgmap v11645: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:11 smithi078 ceph-mon[99959]: pgmap v11645: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:13 smithi146 ceph-mon[103278]: pgmap v11646: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:13 smithi078 ceph-mon[91136]: pgmap v11646: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:13 smithi078 ceph-mon[99959]: pgmap v11646: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[99959]: pgmap v11647: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[91136]: pgmap v11647: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:15 smithi146 ceph-mon[103278]: pgmap v11647: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:18.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:17 smithi146 ceph-mon[103278]: pgmap v11648: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:17 smithi078 ceph-mon[99959]: pgmap v11648: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:17 smithi078 ceph-mon[91136]: pgmap v11648: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:19 smithi146 ceph-mon[103278]: pgmap v11649: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:19 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:19 smithi078 ceph-mon[91136]: pgmap v11649: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:19 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:19 smithi078 ceph-mon[99959]: pgmap v11649: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:19 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:48:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:48:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:21 smithi146 ceph-mon[103278]: pgmap v11650: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[99959]: pgmap v11650: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-mon[91136]: pgmap v11650: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:21.794Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:21.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:21.795Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:22.746 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 6h - - 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 7s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 2s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 7s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 7s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 7s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:48:23.295 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 7s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 7s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 2s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 7s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 7s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 7s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 7s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 6h 76.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 7s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:48:23.296 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 7s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:48:23.297 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:48:23.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:48:23.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:48:23.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:48:23.636 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:48:23.914 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:48:23.915 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:48:23.915 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:48:23.915 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:48:23.915 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:48:23.915 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:48:23.915 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:48:23.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[99959]: from='client.90386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:23.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[99959]: pgmap v11651: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:23.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[99959]: from='client.70473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:23.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[99959]: from='client.90398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:23.915 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2268278510' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:23.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[91136]: from='client.90386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:23.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[91136]: pgmap v11651: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:23.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[91136]: from='client.70473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:23.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[91136]: from='client.90398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:23.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2268278510' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:48:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:24.245 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:24.246 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:23 smithi146 ceph-mon[103278]: from='client.90386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:23 smithi146 ceph-mon[103278]: pgmap v11651: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:23 smithi146 ceph-mon[103278]: from='client.70473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:23 smithi146 ceph-mon[103278]: from='client.90398 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2268278510' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:25.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:24 smithi146 ceph-mon[103278]: from='client.90410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2584984011' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:24 smithi078 ceph-mon[99959]: from='client.90410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2584984011' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:24 smithi078 ceph-mon[91136]: from='client.90410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2584984011' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:25 smithi146 ceph-mon[103278]: pgmap v11652: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:25 smithi078 ceph-mon[91136]: pgmap v11652: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:25 smithi078 ceph-mon[99959]: pgmap v11652: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:27 smithi146 ceph-mon[103278]: pgmap v11653: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:27 smithi078 ceph-mon[91136]: pgmap v11653: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:27 smithi078 ceph-mon[99959]: pgmap v11653: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:29 smithi146 ceph-mon[103278]: pgmap v11654: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:29 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:29 smithi078 ceph-mon[99959]: pgmap v11654: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:29 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:29 smithi078 ceph-mon[91136]: pgmap v11654: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:29 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:31.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:48:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:48:31.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:31 smithi078 ceph-mon[91136]: pgmap v11655: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:31.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:31 smithi078 ceph-mon[99959]: pgmap v11655: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:31.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:31.795Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:31.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:31.796Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:31.961 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:31.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:32.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:31 smithi146 ceph-mon[103278]: pgmap v11655: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:33 smithi078 ceph-mon[91136]: pgmap v11656: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:34.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:33 smithi078 ceph-mon[99959]: pgmap v11656: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:33 smithi146 ceph-mon[103278]: pgmap v11656: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:35 smithi146 ceph-mon[103278]: pgmap v11657: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[91136]: pgmap v11657: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[99959]: pgmap v11657: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:37 smithi146 ceph-mon[103278]: pgmap v11658: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:37 smithi078 ceph-mon[91136]: pgmap v11658: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:37 smithi078 ceph-mon[99959]: pgmap v11658: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:39 smithi146 ceph-mon[103278]: pgmap v11659: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:39 smithi078 ceph-mon[91136]: pgmap v11659: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:39 smithi078 ceph-mon[99959]: pgmap v11659: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:40.861 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:48:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:48:40.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:40.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:40.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:40.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:41 smithi146 ceph-mon[103278]: pgmap v11660: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:41 smithi078 ceph-mon[91136]: pgmap v11660: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:41 smithi078 ceph-mon[99959]: pgmap v11660: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:41.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:41.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:41.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:43 smithi146 ceph-mon[103278]: pgmap v11661: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:43 smithi078 ceph-mon[99959]: pgmap v11661: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:43 smithi078 ceph-mon[91136]: pgmap v11661: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:45 smithi146 ceph-mon[103278]: pgmap v11662: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:45 smithi078 ceph-mon[91136]: pgmap v11662: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:45 smithi078 ceph-mon[99959]: pgmap v11662: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:47 smithi146 ceph-mon[103278]: pgmap v11663: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:47 smithi078 ceph-mon[91136]: pgmap v11663: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:47 smithi078 ceph-mon[99959]: pgmap v11663: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:49 smithi146 ceph-mon[103278]: pgmap v11664: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:49 smithi078 ceph-mon[91136]: pgmap v11664: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:49 smithi078 ceph-mon[99959]: pgmap v11664: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:50.880 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:48:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:48:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:48:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:51 smithi078 ceph-mon[99959]: pgmap v11665: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:51 smithi078 ceph-mon[91136]: pgmap v11665: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:51.797Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:51.797Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:48:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:48:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:48:51.798Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:48:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:51 smithi146 ceph-mon[103278]: pgmap v11665: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:54.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:53 smithi146 ceph-mon[103278]: pgmap v11666: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:54.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:53 smithi078 ceph-mon[91136]: pgmap v11666: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:54.125 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:53 smithi078 ceph-mon[99959]: pgmap v11666: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:54.541 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:48:55.078 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 6h - - 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 14s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 14s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 19s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 14s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 19s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 14s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:48:55.079 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 19s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 19s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 19s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 19s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 14s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 14s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 14s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 14s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 14s ago 6h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 19s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 19s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:48:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:48:55.396 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:48:55.397 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:48:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:48:55.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:48:55.398 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:48:55.668 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:48:55.669 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:48:55.669 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:48:55.991 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:48:55.991 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:48:55.991 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:55.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:48:55.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[91136]: from='client.70503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[91136]: pgmap v11667: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[91136]: from='client.70509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[91136]: from='client.70515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3185755801' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[99959]: from='client.70503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[99959]: pgmap v11667: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[99959]: from='client.70509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[99959]: from='client.70515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:55.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3185755801' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:55 smithi146 ceph-mon[103278]: from='client.70503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:55 smithi146 ceph-mon[103278]: pgmap v11667: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:48:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:55 smithi146 ceph-mon[103278]: from='client.70509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:55 smithi146 ceph-mon[103278]: from='client.70515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3185755801' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:56 smithi146 ceph-mon[103278]: from='client.90446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2310319483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[91136]: from='client.90446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2310319483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[99959]: from='client.90446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:48:57.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:48:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2310319483' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:48:58.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:57 smithi146 ceph-mon[103278]: pgmap v11668: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:57 smithi078 ceph-mon[91136]: pgmap v11668: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:57 smithi078 ceph-mon[99959]: pgmap v11668: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:48:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:48:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:48:59 smithi146 ceph-mon[103278]: pgmap v11669: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:48:59 smithi078 ceph-mon[91136]: pgmap v11669: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:48:59 smithi078 ceph-mon[99959]: pgmap v11669: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:00.929 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:49:00] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:49:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:01 smithi146 ceph-mon[103278]: pgmap v11670: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:01 smithi078 ceph-mon[91136]: pgmap v11670: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:01 smithi078 ceph-mon[99959]: pgmap v11670: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:01.798Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:01.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:01.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:03 smithi146 ceph-mon[103278]: pgmap v11671: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:03 smithi078 ceph-mon[91136]: pgmap v11671: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:03 smithi078 ceph-mon[99959]: pgmap v11671: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:05 smithi146 ceph-mon[103278]: pgmap v11672: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:05 smithi078 ceph-mon[91136]: pgmap v11672: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:05 smithi078 ceph-mon[99959]: pgmap v11672: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:08.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:07 smithi146 ceph-mon[103278]: pgmap v11673: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:07 smithi078 ceph-mon[91136]: pgmap v11673: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:07 smithi078 ceph-mon[99959]: pgmap v11673: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:09 smithi146 ceph-mon[103278]: pgmap v11674: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:09 smithi078 ceph-mon[91136]: pgmap v11674: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:09 smithi078 ceph-mon[99959]: pgmap v11674: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:10.949 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:49:10] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:49:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:11 smithi146 ceph-mon[103278]: pgmap v11675: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:11 smithi078 ceph-mon[91136]: pgmap v11675: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:11 smithi078 ceph-mon[99959]: pgmap v11675: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:11.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:11.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:11.799Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:13 smithi078 ceph-mon[91136]: pgmap v11676: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:49:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:13 smithi078 ceph-mon[99959]: pgmap v11676: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:49:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:13 smithi146 ceph-mon[103278]: pgmap v11676: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T02:49:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:16.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:15 smithi078 ceph-mon[99959]: pgmap v11677: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:16.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:15 smithi078 ceph-mon[91136]: pgmap v11677: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:15 smithi146 ceph-mon[103278]: pgmap v11677: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:18 smithi078 ceph-mon[91136]: pgmap v11678: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:18 smithi078 ceph-mon[99959]: pgmap v11678: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:18 smithi146 ceph-mon[103278]: pgmap v11678: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:20 smithi078 ceph-mon[91136]: pgmap v11679: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:20 smithi078 ceph-mon[99959]: pgmap v11679: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:20 smithi146 ceph-mon[103278]: pgmap v11679: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:49:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:49:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:22 smithi078 ceph-mon[91136]: pgmap v11680: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:22 smithi078 ceph-mon[99959]: pgmap v11680: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:21.799Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:21.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:21.800Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:22.332 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:22 smithi146 ceph-mon[103278]: pgmap v11680: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:24.342 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:24 smithi146 ceph-mon[103278]: pgmap v11681: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:24 smithi078 ceph-mon[91136]: pgmap v11681: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:24 smithi078 ceph-mon[99959]: pgmap v11681: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:26 smithi146 ceph-mon[103278]: pgmap v11682: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:26.282 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:26 smithi078 ceph-mon[99959]: pgmap v11682: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:26.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:26 smithi078 ceph-mon[91136]: pgmap v11682: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:26.283 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 6h - - 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 6h - - 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 10s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 5s ago 6h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 10s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 5s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 10s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 10s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:49:26.810 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 5s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 10s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 10s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 5s ago 6h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 10s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 10s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 10s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 10s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 5s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 5s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 5s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:49:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 5s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:49:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 5s ago 6h 74.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:49:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 10s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:49:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:49:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 10s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:49:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:49:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:49:27.122 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:49:27.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:27 smithi078 ceph-mon[99959]: from='client.90458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:27.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:27 smithi078 ceph-mon[91136]: from='client.90458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:49:27.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:49:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:27 smithi146 ceph-mon[103278]: from='client.90458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:27.697 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:49:27.697 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:27.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:27.698 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:27.698 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:28 smithi146 ceph-mon[103278]: from='client.70545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:28 smithi146 ceph-mon[103278]: from='client.70551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:28 smithi146 ceph-mon[103278]: pgmap v11683: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1327881756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1926629193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[99959]: from='client.70545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[99959]: from='client.70551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[99959]: pgmap v11683: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1327881756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1926629193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[91136]: from='client.70545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[91136]: from='client.70551 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[91136]: pgmap v11683: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1327881756' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1926629193' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:49:29.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:29 smithi078 ceph-mon[99959]: from='client.70563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:29.343 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:29.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:29 smithi078 ceph-mon[91136]: from='client.70563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:29.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:29 smithi146 ceph-mon[103278]: from='client.70563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:30 smithi146 ceph-mon[103278]: pgmap v11684: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:30 smithi078 ceph-mon[91136]: pgmap v11684: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:30 smithi078 ceph-mon[99959]: pgmap v11684: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:49:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:49:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:32.069 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:31.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:32.069 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:31.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:32.069 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:31.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:32 smithi146 ceph-mon[103278]: pgmap v11685: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:32 smithi078 ceph-mon[91136]: pgmap v11685: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:32 smithi078 ceph-mon[99959]: pgmap v11685: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:34.434 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:34 smithi078 ceph-mon[99959]: pgmap v11686: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:34.434 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:34 smithi078 ceph-mon[91136]: pgmap v11686: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:34 smithi146 ceph-mon[103278]: pgmap v11686: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:36.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:36 smithi078 ceph-mon[91136]: pgmap v11687: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:36.482 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:36 smithi078 ceph-mon[99959]: pgmap v11687: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:36 smithi146 ceph-mon[103278]: pgmap v11687: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:37.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:38 smithi146 ceph-mon[103278]: pgmap v11688: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:38 smithi078 ceph-mon[91136]: pgmap v11688: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:38 smithi078 ceph-mon[99959]: pgmap v11688: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:40 smithi146 ceph-mon[103278]: pgmap v11689: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:40 smithi078 ceph-mon[91136]: pgmap v11689: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:40 smithi078 ceph-mon[99959]: pgmap v11689: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:49:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:49:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:42.180 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:41.800Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:42.181 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:41.801Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:42.181 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:41.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:42.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:42 smithi146 ceph-mon[103278]: pgmap v11690: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:42 smithi078 ceph-mon[91136]: pgmap v11690: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:42 smithi078 ceph-mon[99959]: pgmap v11690: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:44 smithi146 ceph-mon[103278]: pgmap v11691: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:44 smithi078 ceph-mon[91136]: pgmap v11691: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:44 smithi078 ceph-mon[99959]: pgmap v11691: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:46 smithi146 ceph-mon[103278]: pgmap v11692: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:46 smithi078 ceph-mon[91136]: pgmap v11692: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:46 smithi078 ceph-mon[99959]: pgmap v11692: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:48 smithi146 ceph-mon[103278]: pgmap v11693: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:48 smithi078 ceph-mon[91136]: pgmap v11693: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:48 smithi078 ceph-mon[99959]: pgmap v11693: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:50 smithi146 ceph-mon[103278]: pgmap v11694: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:50 smithi078 ceph-mon[91136]: pgmap v11694: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:50 smithi078 ceph-mon[99959]: pgmap v11694: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:49:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:49:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:49:52.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:51.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:52.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:51.802Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:49:52.199 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:49:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:49:51.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:49:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:52 smithi146 ceph-mon[103278]: pgmap v11695: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:52 smithi078 ceph-mon[91136]: pgmap v11695: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:52 smithi078 ceph-mon[99959]: pgmap v11695: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:54 smithi146 ceph-mon[103278]: pgmap v11696: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:54 smithi078 ceph-mon[91136]: pgmap v11696: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:54 smithi078 ceph-mon[99959]: pgmap v11696: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:56.383 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:56 smithi078 ceph-mon[99959]: pgmap v11697: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:56.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:56 smithi078 ceph-mon[91136]: pgmap v11697: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:56.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:56 smithi146 ceph-mon[103278]: pgmap v11697: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:49:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:49:57.990 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 6h - - 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 17s ago 6h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 17s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 2s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 2s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 17s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:49:58.522 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 2s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 17s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 2s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 2s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 2s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 17s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 17s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 17s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 17s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 17s ago 6h 76.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 2s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:49:58.523 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 2s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:49:58.524 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:49:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:58 smithi146 ceph-mon[103278]: pgmap v11698: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:58 smithi078 ceph-mon[91136]: pgmap v11698: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:58 smithi078 ceph-mon[99959]: pgmap v11698: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:49:58.836 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:49:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:49:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:49:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:49:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:49:59.104 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[99959]: from='client.70575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[99959]: from='client.70581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1686924301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[91136]: from='client.70575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[91136]: from='client.70581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:59.373 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:59.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:49:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1686924301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:49:59.420 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:49:59.420 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:59.420 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:49:59.420 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:59.420 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:49:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:59 smithi146 ceph-mon[103278]: from='client.70575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:59 smithi146 ceph-mon[103278]: from='client.70581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:49:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:49:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:49:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1686924301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T02:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[99959]: from='client.70587 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[99959]: pgmap v11699: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[99959]: from='client.90518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3229177611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[91136]: from='client.70587 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[91136]: pgmap v11699: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[91136]: from='client.90518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3229177611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:00 smithi146 ceph-mon[103278]: from='client.70587 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:00 smithi146 ceph-mon[103278]: pgmap v11699: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:00 smithi146 ceph-mon[103278]: from='client.90518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3229177611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:50:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:50:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:01.802Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:01.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:01.803Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:02 smithi146 ceph-mon[103278]: pgmap v11700: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:02 smithi078 ceph-mon[91136]: pgmap v11700: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:02 smithi078 ceph-mon[99959]: pgmap v11700: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:04.439 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:04 smithi146 ceph-mon[103278]: pgmap v11701: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:04 smithi078 ceph-mon[91136]: pgmap v11701: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:04 smithi078 ceph-mon[99959]: pgmap v11701: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:06 smithi146 ceph-mon[103278]: pgmap v11702: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:06 smithi078 ceph-mon[91136]: pgmap v11702: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:06 smithi078 ceph-mon[99959]: pgmap v11702: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:08.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:08 smithi146 ceph-mon[103278]: pgmap v11703: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:08 smithi078 ceph-mon[91136]: pgmap v11703: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:08 smithi078 ceph-mon[99959]: pgmap v11703: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:10.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:10 smithi078 ceph-mon[99959]: pgmap v11704: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:10.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:10 smithi078 ceph-mon[91136]: pgmap v11704: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:10.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:10 smithi146 ceph-mon[103278]: pgmap v11704: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:50:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:50:11.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:11.803Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:11.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:11.804Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:12 smithi146 ceph-mon[103278]: pgmap v11705: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:12 smithi078 ceph-mon[99959]: pgmap v11705: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:12 smithi078 ceph-mon[91136]: pgmap v11705: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:13 smithi146 ceph-mon[103278]: pgmap v11706: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:13 smithi078 ceph-mon[91136]: pgmap v11706: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:13 smithi078 ceph-mon[99959]: pgmap v11706: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:15.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:15 smithi078 ceph-mon[91136]: pgmap v11707: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:15.760 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:15 smithi078 ceph-mon[99959]: pgmap v11707: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:15.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:15 smithi146 ceph-mon[103278]: pgmap v11707: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:17 smithi146 ceph-mon[103278]: pgmap v11708: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[91136]: pgmap v11708: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:17 smithi078 ceph-mon[99959]: pgmap v11708: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:19 smithi146 ceph-mon[103278]: pgmap v11709: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:19 smithi078 ceph-mon[99959]: pgmap v11709: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:19 smithi078 ceph-mon[91136]: pgmap v11709: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:20.894 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:50:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:50:21.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[99959]: pgmap v11710: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[91136]: pgmap v11710: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:21.804Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:22.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:21.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:22.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:21.805Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:21 smithi146 ceph-mon[103278]: pgmap v11710: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:24.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:23 smithi146 ceph-mon[103278]: pgmap v11711: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:23 smithi078 ceph-mon[91136]: pgmap v11711: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:23 smithi078 ceph-mon[99959]: pgmap v11711: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:26.121 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:25 smithi146 ceph-mon[103278]: pgmap v11712: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:25 smithi078 ceph-mon[91136]: pgmap v11712: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:25 smithi078 ceph-mon[99959]: pgmap v11712: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:28.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:27 smithi146 ceph-mon[103278]: pgmap v11713: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:27 smithi078 ceph-mon[91136]: pgmap v11713: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:27 smithi078 ceph-mon[99959]: pgmap v11713: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:29.713 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:50:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:29 smithi078 ceph-mon[91136]: pgmap v11714: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:29 smithi078 ceph-mon[99959]: pgmap v11714: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 6h - - 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 6h - - 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 13s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 8s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 13s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 8s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 13s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 13s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:50:30.244 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 8s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 13s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 13s ago 6h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 8s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 13s ago 6h 95.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 13s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 13s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 13s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 8s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 8s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 8s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 8s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 8s ago 6h 77.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 13s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 13s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:50:30.245 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:50:30.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:29 smithi146 ceph-mon[103278]: pgmap v11714: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:30.557 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:50:30.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:50:30.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:50:30.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:50:30.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:50:30.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:50:30.558 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:50:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:50:30.825 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:50:30.825 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:50:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:50:31.139 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:50:31.139 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:50:31.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:50:31.139 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:50:31.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:50:31.139 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[99959]: from='client.70611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[99959]: from='client.90536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[99959]: from='client.70623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3196414859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[91136]: from='client.70611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[91136]: from='client.90536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[91136]: from='client.70623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3196414859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:30 smithi146 ceph-mon[103278]: from='client.70611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:30 smithi146 ceph-mon[103278]: from='client.90536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:30 smithi146 ceph-mon[103278]: from='client.70623 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3196414859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:50:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:31 smithi146 ceph-mon[103278]: from='client.70635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:31 smithi146 ceph-mon[103278]: pgmap v11715: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/281423593' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-mon[99959]: from='client.70635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-mon[99959]: pgmap v11715: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/281423593' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-mon[91136]: from='client.70635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:50:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-mon[91136]: pgmap v11715: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/281423593' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:50:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:31.806Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:31.806Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:31.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:33 smithi146 ceph-mon[103278]: pgmap v11716: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:33 smithi078 ceph-mon[91136]: pgmap v11716: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:33 smithi078 ceph-mon[99959]: pgmap v11716: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:35 smithi146 ceph-mon[103278]: pgmap v11717: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:35 smithi078 ceph-mon[91136]: pgmap v11717: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:35 smithi078 ceph-mon[99959]: pgmap v11717: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[91136]: pgmap v11718: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:37 smithi078 ceph-mon[99959]: pgmap v11718: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:37 smithi146 ceph-mon[103278]: pgmap v11718: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:39 smithi146 ceph-mon[103278]: pgmap v11719: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:39 smithi078 ceph-mon[91136]: pgmap v11719: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:39 smithi078 ceph-mon[99959]: pgmap v11719: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:40.902 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:50:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:50:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:41.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:41 smithi146 ceph-mon[103278]: pgmap v11720: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[91136]: pgmap v11720: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[99959]: pgmap v11720: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:41.807Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:41.807Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:41.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:43 smithi146 ceph-mon[103278]: pgmap v11721: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:43 smithi078 ceph-mon[91136]: pgmap v11721: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:43 smithi078 ceph-mon[99959]: pgmap v11721: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:46.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:45 smithi146 ceph-mon[103278]: pgmap v11722: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:45 smithi078 ceph-mon[91136]: pgmap v11722: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:45 smithi078 ceph-mon[99959]: pgmap v11722: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:47 smithi146 ceph-mon[103278]: pgmap v11723: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:47 smithi078 ceph-mon[91136]: pgmap v11723: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:47 smithi078 ceph-mon[99959]: pgmap v11723: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:49 smithi146 ceph-mon[103278]: pgmap v11724: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:49 smithi078 ceph-mon[91136]: pgmap v11724: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:49 smithi078 ceph-mon[99959]: pgmap v11724: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:50.921 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:50:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:50:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:50:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:51 smithi078 ceph-mon[99959]: pgmap v11725: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:51 smithi078 ceph-mon[91136]: pgmap v11725: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:51.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:51.808Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:50:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:50:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:50:51.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:50:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:51 smithi146 ceph-mon[103278]: pgmap v11725: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:53 smithi146 ceph-mon[103278]: pgmap v11726: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:53 smithi078 ceph-mon[91136]: pgmap v11726: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:53 smithi078 ceph-mon[99959]: pgmap v11726: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:50:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:55 smithi146 ceph-mon[103278]: pgmap v11727: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:55 smithi078 ceph-mon[91136]: pgmap v11727: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:55 smithi078 ceph-mon[99959]: pgmap v11727: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:50:57.894 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[99959]: pgmap v11728: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:57.895 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:57 smithi078 ceph-mon[91136]: pgmap v11728: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:50:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:50:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:57 smithi146 ceph-mon[103278]: pgmap v11728: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T02:50:58.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:58.904 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:59.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:50:59.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:50:59 smithi078 ceph-mon[91136]: pgmap v11729: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:50:59.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:50:59 smithi078 ceph-mon[99959]: pgmap v11729: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:51:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:50:59 smithi146 ceph-mon[103278]: pgmap v11729: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T02:51:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:51:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:51:01.431 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 6h - - 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 6h - - 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 5s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 0s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 5s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 0s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 5s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 5s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 0s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 5s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:51:01.976 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 5s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 0s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 5s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 5s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 5s ago 6h 83.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 0s ago 6h 116M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 0s ago 6h 91.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 0s ago 6h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 0s ago 6h 93.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 0s ago 6h 78.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 5s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 5s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:51:01.977 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[99959]: pgmap v11730: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[91136]: pgmap v11730: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:01.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:01.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:01.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:01.808Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:01.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:01.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:01.979 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:01.809Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:01 smithi146 ceph-mon[103278]: pgmap v11730: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:51:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:02.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:51:02.286 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:51:02.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:51:02.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:51:02.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:51:02.555 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:51:02.555 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:51:02.556 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:51:02.867 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:51:02.868 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:51:02.868 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:51:02.868 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:51:02.868 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:51:02.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[99959]: from='client.90566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:02.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[99959]: from='client.90572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:02.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[99959]: from='client.70659 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:02.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4132072879' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:02.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[91136]: from='client.90566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:02.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[91136]: from='client.90572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:02.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[91136]: from='client.70659 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:02.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4132072879' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:03.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:02 smithi146 ceph-mon[103278]: from='client.90566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:02 smithi146 ceph-mon[103278]: from='client.90572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:02 smithi146 ceph-mon[103278]: from='client.70659 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4132072879' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:03 smithi146 ceph-mon[103278]: from='client.90590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:03 smithi146 ceph-mon[103278]: pgmap v11731: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:51:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2665726273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:03 smithi078 ceph-mon[91136]: from='client.90590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:03 smithi078 ceph-mon[91136]: pgmap v11731: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:51:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2665726273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:04.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:03 smithi078 ceph-mon[99959]: from='client.90590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:03 smithi078 ceph-mon[99959]: pgmap v11731: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T02:51:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2665726273' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:05 smithi146 ceph-mon[103278]: pgmap v11732: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:51:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:05 smithi078 ceph-mon[91136]: pgmap v11732: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:51:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:05 smithi078 ceph-mon[99959]: pgmap v11732: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T02:51:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:07 smithi146 ceph-mon[103278]: pgmap v11733: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:51:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:07 smithi078 ceph-mon[91136]: pgmap v11733: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:51:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:07 smithi078 ceph-mon[99959]: pgmap v11733: 161 pgs: 161 active+clean; 585 KiB data, 362 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:51:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:09 smithi146 ceph-mon[103278]: pgmap v11734: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:51:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:09 smithi078 ceph-mon[91136]: pgmap v11734: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:51:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:09 smithi078 ceph-mon[99959]: pgmap v11734: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T02:51:10.913 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:51:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:51:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:11 smithi146 ceph-mon[103278]: pgmap v11735: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T02:51:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:11 smithi078 ceph-mon[91136]: pgmap v11735: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T02:51:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:11 smithi078 ceph-mon[99959]: pgmap v11735: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-09-21T02:51:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:11.809Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:11.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:11.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:13 smithi146 ceph-mon[103278]: pgmap v11736: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T02:51:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:13 smithi078 ceph-mon[91136]: pgmap v11736: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T02:51:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:13 smithi078 ceph-mon[99959]: pgmap v11736: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T02:51:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:15 smithi146 ceph-mon[103278]: pgmap v11737: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T02:51:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:15 smithi078 ceph-mon[91136]: pgmap v11737: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T02:51:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:15 smithi078 ceph-mon[99959]: pgmap v11737: 161 pgs: 161 active+clean; 585 KiB data, 349 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-09-21T02:51:18.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:17 smithi146 ceph-mon[103278]: pgmap v11738: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[91136]: pgmap v11738: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:18.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:17 smithi078 ceph-mon[99959]: pgmap v11738: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T02:51:20.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:19 smithi078 ceph-mon[91136]: pgmap v11739: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T02:51:20.051 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:19 smithi078 ceph-mon[99959]: pgmap v11739: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T02:51:20.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:19 smithi146 ceph-mon[103278]: pgmap v11739: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-09-21T02:51:20.865 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:51:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:51:21.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:21.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:21 smithi146 ceph-mon[103278]: pgmap v11740: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:51:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[99959]: pgmap v11740: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:51:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[91136]: pgmap v11740: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s 2024-09-21T02:51:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:21.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:21.810Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:21.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:23 smithi146 ceph-mon[103278]: pgmap v11741: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-21T02:51:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:23 smithi078 ceph-mon[91136]: pgmap v11741: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-21T02:51:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:23 smithi078 ceph-mon[99959]: pgmap v11741: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 0 B/s wr, 86 op/s 2024-09-21T02:51:26.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:25 smithi146 ceph-mon[103278]: pgmap v11742: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:51:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:25 smithi078 ceph-mon[91136]: pgmap v11742: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:51:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:25 smithi078 ceph-mon[99959]: pgmap v11742: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-09-21T02:51:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:27 smithi146 ceph-mon[103278]: pgmap v11743: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:51:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:27 smithi078 ceph-mon[91136]: pgmap v11743: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:51:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:27 smithi078 ceph-mon[99959]: pgmap v11743: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T02:51:29.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:29 smithi146 ceph-mon[103278]: pgmap v11744: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T02:51:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:29 smithi078 ceph-mon[91136]: pgmap v11744: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T02:51:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:29 smithi078 ceph-mon[99959]: pgmap v11744: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s 2024-09-21T02:51:30.937 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:51:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:51:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:31 smithi146 ceph-mon[103278]: pgmap v11745: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T02:51:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:31 smithi078 ceph-mon[91136]: pgmap v11745: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T02:51:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:31 smithi078 ceph-mon[99959]: pgmap v11745: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-09-21T02:51:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:31.810Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:31.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:31.811Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:33.165 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:51:33.689 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 6h - - 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 6h - - 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 16s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 11s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 16s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 11s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 16s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 16s ago 6h 551M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 11s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 16s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 16s ago 6h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 11s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 16s ago 6h 96.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 16s ago 6h 101M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:51:33.690 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 16s ago 6h 84.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 16s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 11s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 11s ago 6h 92.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 11s ago 6h 98.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 11s ago 6h 93.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 11s ago 6h 78.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 16s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 16s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:51:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:51:34.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:51:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:51:34.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:51:34.003 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:51:34.004 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:51:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:33 smithi146 ceph-mon[103278]: pgmap v11746: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:51:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:33 smithi146 ceph-mon[103278]: from='client.90602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:34.269 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:51:34.270 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:51:34.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:51:34.271 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:51:34.271 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:51:34.271 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:51:34.271 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:51:34.271 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:51:34.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:33 smithi078 ceph-mon[99959]: pgmap v11746: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:51:34.272 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:33 smithi078 ceph-mon[99959]: from='client.90602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:34.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:33 smithi078 ceph-mon[91136]: pgmap v11746: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T02:51:34.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:33 smithi078 ceph-mon[91136]: from='client.90602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:34.578 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:51:34.579 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:51:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:51:34.579 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:51:34.579 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:51:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:34 smithi146 ceph-mon[103278]: from='client.70689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:34 smithi146 ceph-mon[103278]: from='client.70695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1092899729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:34 smithi146 ceph-mon[103278]: from='client.90626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2508278528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[91136]: from='client.70689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[91136]: from='client.70695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1092899729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[91136]: from='client.90626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2508278528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[99959]: from='client.70689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[99959]: from='client.70695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1092899729' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:51:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[99959]: from='client.90626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:51:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2508278528' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:51:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:35 smithi146 ceph-mon[103278]: pgmap v11747: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:51:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:35 smithi078 ceph-mon[91136]: pgmap v11747: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:51:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:35 smithi078 ceph-mon[99959]: pgmap v11747: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T02:51:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:37 smithi146 ceph-mon[103278]: pgmap v11748: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:51:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[91136]: pgmap v11748: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[99959]: pgmap v11748: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:39 smithi078 ceph-mon[91136]: pgmap v11749: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:39 smithi078 ceph-mon[99959]: pgmap v11749: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:39 smithi146 ceph-mon[103278]: pgmap v11749: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:51:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:51:41.249 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:41.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:41 smithi146 ceph-mon[103278]: pgmap v11750: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[91136]: pgmap v11750: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[99959]: pgmap v11750: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:41.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:41.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:41.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:43 smithi146 ceph-mon[103278]: pgmap v11751: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:43 smithi078 ceph-mon[91136]: pgmap v11751: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:43 smithi078 ceph-mon[99959]: pgmap v11751: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:46.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:45 smithi146 ceph-mon[103278]: pgmap v11752: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:45 smithi078 ceph-mon[91136]: pgmap v11752: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:45 smithi078 ceph-mon[99959]: pgmap v11752: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:47 smithi146 ceph-mon[103278]: pgmap v11753: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:47 smithi078 ceph-mon[99959]: pgmap v11753: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:47 smithi078 ceph-mon[91136]: pgmap v11753: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:49 smithi146 ceph-mon[103278]: pgmap v11754: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:49 smithi078 ceph-mon[91136]: pgmap v11754: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:49 smithi078 ceph-mon[99959]: pgmap v11754: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:50.951 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:51:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:51:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:51:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:51 smithi078 ceph-mon[99959]: pgmap v11755: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:51 smithi078 ceph-mon[91136]: pgmap v11755: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:51.811Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:51.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:51:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:51:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:51:51.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:51:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:51 smithi146 ceph-mon[103278]: pgmap v11755: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:53 smithi146 ceph-mon[103278]: pgmap v11756: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:53 smithi078 ceph-mon[91136]: pgmap v11756: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:53 smithi078 ceph-mon[99959]: pgmap v11756: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:56.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:55 smithi146 ceph-mon[103278]: pgmap v11757: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:55 smithi078 ceph-mon[99959]: pgmap v11757: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:55 smithi078 ceph-mon[91136]: pgmap v11757: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:51:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:58 smithi146 ceph-mon[103278]: pgmap v11758: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[91136]: pgmap v11758: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[99959]: pgmap v11758: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:51:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:51:59.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:51:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:59.468 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:51:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:51:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:51:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:00 smithi078 ceph-mon[99959]: pgmap v11759: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:00 smithi078 ceph-mon[91136]: pgmap v11759: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:00 smithi146 ceph-mon[103278]: pgmap v11759: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:00.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:52:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:52:01.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:01.127 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:02.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[99959]: pgmap v11760: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:02.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.273 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.274 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[91136]: pgmap v11760: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:02.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.274 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:01.812Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:02.274 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:01.812Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:02.275 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:01.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:02 smithi146 ceph-mon[103278]: pgmap v11760: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:04 smithi146 ceph-mon[103278]: pgmap v11761: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:04 smithi078 ceph-mon[91136]: pgmap v11761: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:04 smithi078 ceph-mon[99959]: pgmap v11761: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:04.876 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 6h - - 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 6h - - 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 8s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 3s ago 6h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 8s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 3s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 8s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 8s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:52:05.406 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 3s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 8s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 8s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 3s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 8s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 8s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 8s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 3s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 3s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 3s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 3s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 3s ago 6h 78.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 8s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:52:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 8s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:52:05.408 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:52:05.717 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:52:05.717 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:52:05.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:52:05.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:52:05.719 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:52:05.988 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:52:05.989 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:52:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:52:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:52:05.990 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:52:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[99959]: from='client.70719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[99959]: pgmap v11762: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[99959]: from='client.70725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2058079752' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[91136]: from='client.70719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[91136]: pgmap v11762: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[91136]: from='client.70725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2058079752' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:06.300 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:52:06.300 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:06.300 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:06.300 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:06.301 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:06.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:06 smithi146 ceph-mon[103278]: from='client.70719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:06.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:06 smithi146 ceph-mon[103278]: pgmap v11762: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:06.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:06 smithi146 ceph-mon[103278]: from='client.70725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:06.319 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2058079752' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:07.392 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:07 smithi146 ceph-mon[103278]: from='client.70731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:07.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:07 smithi146 ceph-mon[103278]: from='client.70743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:07.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/157561248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:07 smithi078 ceph-mon[99959]: from='client.70731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:07 smithi078 ceph-mon[99959]: from='client.70743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/157561248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:07 smithi078 ceph-mon[91136]: from='client.70731 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:07 smithi078 ceph-mon[91136]: from='client.70743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/157561248' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:08 smithi146 ceph-mon[103278]: pgmap v11763: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:08 smithi078 ceph-mon[91136]: pgmap v11763: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:08 smithi078 ceph-mon[99959]: pgmap v11763: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:10 smithi146 ceph-mon[103278]: pgmap v11764: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:10 smithi078 ceph-mon[91136]: pgmap v11764: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:10 smithi078 ceph-mon[99959]: pgmap v11764: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:52:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:52:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:12.082 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:11.813Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:12.082 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:11.813Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:12.082 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:11.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:12 smithi146 ceph-mon[103278]: pgmap v11765: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:12 smithi078 ceph-mon[91136]: pgmap v11765: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:12 smithi078 ceph-mon[99959]: pgmap v11765: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:14 smithi146 ceph-mon[103278]: pgmap v11766: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:14 smithi078 ceph-mon[99959]: pgmap v11766: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:14 smithi078 ceph-mon[91136]: pgmap v11766: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:16.399 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:16 smithi078 ceph-mon[91136]: pgmap v11767: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:16.399 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:16 smithi078 ceph-mon[99959]: pgmap v11767: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:16 smithi146 ceph-mon[103278]: pgmap v11767: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:18 smithi146 ceph-mon[103278]: pgmap v11768: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[91136]: pgmap v11768: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[99959]: pgmap v11768: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:20 smithi146 ceph-mon[103278]: pgmap v11769: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:20.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:20 smithi078 ceph-mon[91136]: pgmap v11769: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:20.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:20 smithi078 ceph-mon[99959]: pgmap v11769: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:52:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:52:21.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:22.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:21.814Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:22.151 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:21.814Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:22.152 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:21.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[99959]: pgmap v11770: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[91136]: pgmap v11770: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:22 smithi146 ceph-mon[103278]: pgmap v11770: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:24 smithi146 ceph-mon[103278]: pgmap v11771: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:24 smithi078 ceph-mon[91136]: pgmap v11771: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:24 smithi078 ceph-mon[99959]: pgmap v11771: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:26 smithi146 ceph-mon[103278]: pgmap v11772: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:26 smithi078 ceph-mon[91136]: pgmap v11772: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:26 smithi078 ceph-mon[99959]: pgmap v11772: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:28.374 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:28 smithi146 ceph-mon[103278]: pgmap v11773: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:28 smithi078 ceph-mon[99959]: pgmap v11773: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:28 smithi078 ceph-mon[91136]: pgmap v11773: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:29.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:29.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:30 smithi146 ceph-mon[103278]: pgmap v11774: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:30 smithi078 ceph-mon[91136]: pgmap v11774: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:30 smithi078 ceph-mon[99959]: pgmap v11774: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:52:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:52:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:32.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:31.815Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:32.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:31.815Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:32.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:31.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:32 smithi146 ceph-mon[103278]: pgmap v11775: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:32 smithi078 ceph-mon[91136]: pgmap v11775: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:32 smithi078 ceph-mon[99959]: pgmap v11775: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:34 smithi146 ceph-mon[103278]: pgmap v11776: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:34 smithi078 ceph-mon[91136]: pgmap v11776: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:34 smithi078 ceph-mon[99959]: pgmap v11776: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:36 smithi146 ceph-mon[103278]: pgmap v11777: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:36 smithi078 ceph-mon[99959]: pgmap v11777: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:36 smithi078 ceph-mon[91136]: pgmap v11777: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:36.607 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 6h - - 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 15s ago 6h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 15s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:52:37.155 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 19s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:52:37.156 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 15s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:52:37.156 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 19s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:52:37.156 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:52:37.157 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 15s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:52:37.157 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 19s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:52:37.157 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 19s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:52:37.157 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 19s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 19s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 15s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 15s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 15s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 15s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 15s ago 6h 78.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 19s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 19s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:52:37.158 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:52:37.476 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:52:37.476 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:52:37.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:52:37.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:52:37.477 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:52:37.745 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:52:37.745 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:52:37.745 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:52:37.745 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:52:37.746 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:52:38.051 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:52:38.051 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:38.051 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:52:38.051 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:38.051 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='client.90674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: pgmap v11778: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='client.70761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='client.70767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3678684382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/134958041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='client.90674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: pgmap v11778: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='client.70761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='client.70767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3678684382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/134958041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='client.90674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: pgmap v11778: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='client.70761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='client.70767 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3678684382' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:52:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/134958041' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:52:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:39 smithi146 ceph-mon[103278]: from='client.70779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:39 smithi078 ceph-mon[99959]: from='client.70779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:39 smithi078 ceph-mon[91136]: from='client.70779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:52:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:40 smithi078 ceph-mon[99959]: pgmap v11779: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:40 smithi078 ceph-mon[91136]: pgmap v11779: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:40 smithi146 ceph-mon[103278]: pgmap v11779: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:52:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:52:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:42.121 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:41.815Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:42.122 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:41.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:42.122 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:41.816Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:42.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[99959]: pgmap v11780: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:42.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[91136]: pgmap v11780: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:42.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:42 smithi146 ceph-mon[103278]: pgmap v11780: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:44 smithi146 ceph-mon[103278]: pgmap v11781: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:44 smithi078 ceph-mon[91136]: pgmap v11781: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:44 smithi078 ceph-mon[99959]: pgmap v11781: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:46 smithi146 ceph-mon[103278]: pgmap v11782: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:46 smithi078 ceph-mon[91136]: pgmap v11782: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:46 smithi078 ceph-mon[99959]: pgmap v11782: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:48.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:48 smithi146 ceph-mon[103278]: pgmap v11783: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:48 smithi078 ceph-mon[91136]: pgmap v11783: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:48 smithi078 ceph-mon[99959]: pgmap v11783: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:50.580 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:50 smithi146 ceph-mon[103278]: pgmap v11784: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:50 smithi078 ceph-mon[91136]: pgmap v11784: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:50 smithi078 ceph-mon[99959]: pgmap v11784: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:52:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:52:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:52:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:51.816Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:51.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:52:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:52:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:52:51.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:52:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:52 smithi146 ceph-mon[103278]: pgmap v11785: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:52 smithi078 ceph-mon[91136]: pgmap v11785: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:52 smithi078 ceph-mon[99959]: pgmap v11785: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:54 smithi146 ceph-mon[103278]: pgmap v11786: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:54 smithi078 ceph-mon[99959]: pgmap v11786: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:54 smithi078 ceph-mon[91136]: pgmap v11786: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:56 smithi146 ceph-mon[103278]: pgmap v11787: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:56 smithi078 ceph-mon[91136]: pgmap v11787: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:56 smithi078 ceph-mon[99959]: pgmap v11787: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:52:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:58 smithi146 ceph-mon[103278]: pgmap v11788: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[91136]: pgmap v11788: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[99959]: pgmap v11788: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:52:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:52:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:52:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:52:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:52:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:52:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:00 smithi078 ceph-mon[99959]: pgmap v11789: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:00 smithi078 ceph-mon[91136]: pgmap v11789: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:00 smithi146 ceph-mon[103278]: pgmap v11789: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:53:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:53:01.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:02.080 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:01.817Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:02.080 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:01.817Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:02.080 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:01.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:02 smithi146 ceph-mon[103278]: pgmap v11790: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:02 smithi078 ceph-mon[91136]: pgmap v11790: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:02 smithi078 ceph-mon[99959]: pgmap v11790: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:03 smithi146 ceph-mon[103278]: pgmap v11791: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[91136]: pgmap v11791: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:03 smithi078 ceph-mon[99959]: pgmap v11791: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:05 smithi146 ceph-mon[103278]: pgmap v11792: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:05 smithi078 ceph-mon[91136]: pgmap v11792: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:05 smithi078 ceph-mon[99959]: pgmap v11792: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:08.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:07 smithi146 ceph-mon[103278]: pgmap v11793: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:07 smithi078 ceph-mon[99959]: pgmap v11793: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:07 smithi078 ceph-mon[91136]: pgmap v11793: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:08.346 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:53:08.870 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:53:08.870 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 6h - - 2024-09-21T02:53:08.870 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 6h - - 2024-09-21T02:53:08.870 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 11s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 11s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 6s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 11s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 11s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 6s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 11s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 11s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 11s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 11s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 11s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 6s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:53:08.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 6s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 6s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 6s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 6s ago 6h 79.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 11s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 11s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:53:08.872 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:53:09.184 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:08 smithi078 ceph-mon[99959]: from='client.90710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:09.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:08 smithi078 ceph-mon[91136]: from='client.90710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:09.184 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:53:09.184 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:53:09.184 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:53:09.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:53:09.185 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:53:09.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:08 smithi146 ceph-mon[103278]: from='client.90710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:53:09.453 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:53:09.454 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:53:09.454 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:53:09.454 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:53:09.454 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:53:09.454 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:53:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:53:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:09.763 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[91136]: from='client.70797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[91136]: from='client.70803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[91136]: pgmap v11794: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/30799321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/692169413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[99959]: from='client.70797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[99959]: from='client.70803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[99959]: pgmap v11794: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/30799321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/692169413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:09 smithi146 ceph-mon[103278]: from='client.70797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:09 smithi146 ceph-mon[103278]: from='client.70803 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:09 smithi146 ceph-mon[103278]: pgmap v11794: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/30799321' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/692169413' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:10.960 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:53:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:53:11.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:10 smithi146 ceph-mon[103278]: from='client.90734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:10 smithi078 ceph-mon[91136]: from='client.90734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:10 smithi078 ceph-mon[99959]: from='client.90734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:12.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:11 smithi146 ceph-mon[103278]: pgmap v11795: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:11 smithi078 ceph-mon[91136]: pgmap v11795: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:11 smithi078 ceph-mon[99959]: pgmap v11795: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:11.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:11.818Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:11.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:13 smithi146 ceph-mon[103278]: pgmap v11796: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:13 smithi078 ceph-mon[91136]: pgmap v11796: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:13 smithi078 ceph-mon[99959]: pgmap v11796: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:15 smithi146 ceph-mon[103278]: pgmap v11797: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:15 smithi078 ceph-mon[91136]: pgmap v11797: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:15 smithi078 ceph-mon[99959]: pgmap v11797: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:17 smithi146 ceph-mon[103278]: pgmap v11798: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[91136]: pgmap v11798: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[99959]: pgmap v11798: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:19 smithi146 ceph-mon[103278]: pgmap v11799: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:19 smithi078 ceph-mon[91136]: pgmap v11799: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:19 smithi078 ceph-mon[99959]: pgmap v11799: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:20.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:53:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:53:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:21 smithi078 ceph-mon[99959]: pgmap v11800: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:22.188 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:21 smithi078 ceph-mon[91136]: pgmap v11800: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:22.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:21.818Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:22.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:21.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:22.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:21.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:21 smithi146 ceph-mon[103278]: pgmap v11800: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[91136]: pgmap v11801: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:23 smithi078 ceph-mon[99959]: pgmap v11801: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:24.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:23 smithi146 ceph-mon[103278]: pgmap v11801: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:25 smithi146 ceph-mon[103278]: pgmap v11802: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:25 smithi078 ceph-mon[91136]: pgmap v11802: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:25 smithi078 ceph-mon[99959]: pgmap v11802: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:27 smithi146 ceph-mon[103278]: pgmap v11803: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:27 smithi078 ceph-mon[99959]: pgmap v11803: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:27 smithi078 ceph-mon[91136]: pgmap v11803: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:30.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:29 smithi146 ceph-mon[103278]: pgmap v11804: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:29 smithi078 ceph-mon[91136]: pgmap v11804: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:29 smithi078 ceph-mon[99959]: pgmap v11804: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:30.967 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:53:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:53:31.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:31 smithi146 ceph-mon[103278]: pgmap v11805: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:31 smithi078 ceph-mon[91136]: pgmap v11805: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:31 smithi078 ceph-mon[99959]: pgmap v11805: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:31.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:31.819Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:31.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:33 smithi146 ceph-mon[103278]: pgmap v11806: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:33 smithi078 ceph-mon[91136]: pgmap v11806: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:33 smithi078 ceph-mon[99959]: pgmap v11806: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:35 smithi146 ceph-mon[103278]: pgmap v11807: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:35 smithi078 ceph-mon[99959]: pgmap v11807: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:35 smithi078 ceph-mon[91136]: pgmap v11807: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:37 smithi146 ceph-mon[103278]: pgmap v11808: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[91136]: pgmap v11808: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[99959]: pgmap v11808: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:40.058 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:53:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:39 smithi146 ceph-mon[103278]: pgmap v11809: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:39 smithi078 ceph-mon[91136]: pgmap v11809: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:39 smithi078 ceph-mon[99959]: pgmap v11809: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:40.585 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:53:40.585 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T02:53:40.585 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 6h - - 2024-09-21T02:53:40.585 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:53:40.585 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 18s ago 6h 92.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:53:40.585 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 18s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 2s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 2s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 18s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 2s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 18s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 2s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 2s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 2s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 18s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 18s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:53:40.586 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 18s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:53:40.587 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 18s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:53:40.587 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 18s ago 6h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:53:40.587 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 2s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:53:40.587 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:53:40.587 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 2s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:53:40.587 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:53:40.898 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:53:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:53:40.899 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:53:40.900 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:53:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:53:41.171 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:53:41.172 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[91136]: from='client.90746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[91136]: from='client.90752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1088008637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[99959]: from='client.90746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[99959]: from='client.90752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:41.173 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1088008637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:40 smithi146 ceph-mon[103278]: from='client.90746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:40 smithi146 ceph-mon[103278]: from='client.90752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1088008637' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:53:41.479 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:53:41.479 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:41.479 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:53:41.479 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:41.479 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:53:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:41 smithi146 ceph-mon[103278]: from='client.90758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:41 smithi146 ceph-mon[103278]: pgmap v11810: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:41 smithi146 ceph-mon[103278]: from='client.70851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2939267213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[91136]: from='client.90758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[91136]: pgmap v11810: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[91136]: from='client.70851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2939267213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[99959]: from='client.90758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[99959]: pgmap v11810: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[99959]: from='client.70851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:53:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2939267213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:53:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:41.819Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:41.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:41.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:43.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[99959]: pgmap v11811: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[91136]: pgmap v11811: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:43.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:43 smithi146 ceph-mon[103278]: pgmap v11811: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:45 smithi146 ceph-mon[103278]: pgmap v11812: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:45 smithi078 ceph-mon[91136]: pgmap v11812: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:45 smithi078 ceph-mon[99959]: pgmap v11812: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:47 smithi146 ceph-mon[103278]: pgmap v11813: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:47 smithi078 ceph-mon[91136]: pgmap v11813: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:47 smithi078 ceph-mon[99959]: pgmap v11813: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:49 smithi146 ceph-mon[103278]: pgmap v11814: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:49 smithi078 ceph-mon[91136]: pgmap v11814: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:49 smithi078 ceph-mon[99959]: pgmap v11814: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:50.972 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:53:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:53:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:53:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:51 smithi078 ceph-mon[99959]: pgmap v11815: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:51 smithi078 ceph-mon[91136]: pgmap v11815: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:51.820Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:51.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:53:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:53:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:53:51.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:53:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:51 smithi146 ceph-mon[103278]: pgmap v11815: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:53 smithi146 ceph-mon[103278]: pgmap v11816: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:53 smithi078 ceph-mon[91136]: pgmap v11816: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:53 smithi078 ceph-mon[99959]: pgmap v11816: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:55 smithi146 ceph-mon[103278]: pgmap v11817: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:55 smithi078 ceph-mon[91136]: pgmap v11817: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:55 smithi078 ceph-mon[99959]: pgmap v11817: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:53:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:57 smithi146 ceph-mon[103278]: pgmap v11818: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[91136]: pgmap v11818: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[99959]: pgmap v11818: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:53:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:53:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:53:59 smithi146 ceph-mon[103278]: pgmap v11819: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:53:59 smithi078 ceph-mon[99959]: pgmap v11819: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:53:59 smithi078 ceph-mon[91136]: pgmap v11819: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:00.996 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:54:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:54:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:01.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:01.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:01 smithi146 ceph-mon[103278]: pgmap v11820: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:01 smithi078 ceph-mon[91136]: pgmap v11820: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:01 smithi078 ceph-mon[99959]: pgmap v11820: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:01.821Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:01.821Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:01.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:04.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.219 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[91136]: pgmap v11821: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:04.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.220 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:03 smithi078 ceph-mon[99959]: pgmap v11821: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:03 smithi146 ceph-mon[103278]: pgmap v11821: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:05 smithi146 ceph-mon[103278]: pgmap v11822: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:05 smithi078 ceph-mon[91136]: pgmap v11822: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:05 smithi078 ceph-mon[99959]: pgmap v11822: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:07 smithi146 ceph-mon[103278]: pgmap v11823: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:07 smithi078 ceph-mon[91136]: pgmap v11823: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:07 smithi078 ceph-mon[99959]: pgmap v11823: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:09 smithi146 ceph-mon[103278]: pgmap v11824: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:09 smithi078 ceph-mon[91136]: pgmap v11824: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:09 smithi078 ceph-mon[99959]: pgmap v11824: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:10.978 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:54:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:54:11.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:11.785 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:54:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:11 smithi078 ceph-mon[91136]: pgmap v11825: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:11 smithi078 ceph-mon[99959]: pgmap v11825: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:11.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:11.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:11.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:12.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:11 smithi146 ceph-mon[103278]: pgmap v11825: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:12.311 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:54:12.311 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 6h - - 2024-09-21T02:54:12.311 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 6h - - 2024-09-21T02:54:12.311 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 14s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 9s ago 6h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 14s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 9s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 14s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 14s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 9s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 14s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 14s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 9s ago 6h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 14s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 14s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 14s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 9s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:54:12.312 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 9s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 9s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 9s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 9s ago 6h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 14s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 14s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:54:12.313 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:54:12.621 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:54:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:54:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:54:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:54:12.621 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:54:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:54:12.892 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:54:13.200 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:54:13.200 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:13.200 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[99959]: from='client.90782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[99959]: from='client.70869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[99959]: from='client.90794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3875044195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[91136]: from='client.90782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[91136]: from='client.70869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[91136]: from='client.90794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3875044195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:12 smithi146 ceph-mon[103278]: from='client.90782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:12 smithi146 ceph-mon[103278]: from='client.70869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:12 smithi146 ceph-mon[103278]: from='client.90794 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3875044195' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:13 smithi146 ceph-mon[103278]: from='client.90806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:13 smithi146 ceph-mon[103278]: pgmap v11826: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/926328287' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[91136]: from='client.90806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[91136]: pgmap v11826: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/926328287' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[99959]: from='client.90806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[99959]: pgmap v11826: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/926328287' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:15 smithi146 ceph-mon[103278]: pgmap v11827: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:15 smithi078 ceph-mon[99959]: pgmap v11827: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:15 smithi078 ceph-mon[91136]: pgmap v11827: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:18 smithi146 ceph-mon[103278]: pgmap v11828: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[91136]: pgmap v11828: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[99959]: pgmap v11828: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:20 smithi146 ceph-mon[103278]: pgmap v11829: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:20 smithi078 ceph-mon[91136]: pgmap v11829: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:20 smithi078 ceph-mon[99959]: pgmap v11829: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:21.008 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:54:20] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:54:21.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:21.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:21.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:22.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:22 smithi078 ceph-mon[99959]: pgmap v11830: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:22.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:22 smithi078 ceph-mon[91136]: pgmap v11830: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:22.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:21.822Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:22.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:21.822Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:22.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:21.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:22 smithi146 ceph-mon[103278]: pgmap v11830: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:23 smithi146 ceph-mon[103278]: pgmap v11831: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[91136]: pgmap v11831: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[99959]: pgmap v11831: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:26.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:25 smithi078 ceph-mon[91136]: pgmap v11832: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:26.166 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:25 smithi078 ceph-mon[99959]: pgmap v11832: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:25 smithi146 ceph-mon[103278]: pgmap v11832: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 02:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T02:54:25.849159956Z level=info msg="Completed cleanup jobs" duration=13.191533ms 2024-09-21T02:54:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:27 smithi146 ceph-mon[103278]: pgmap v11833: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:27 smithi078 ceph-mon[91136]: pgmap v11833: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:27 smithi078 ceph-mon[99959]: pgmap v11833: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:29 smithi146 ceph-mon[103278]: pgmap v11834: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:29 smithi078 ceph-mon[91136]: pgmap v11834: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:29 smithi078 ceph-mon[99959]: pgmap v11834: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:30.942 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:54:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:54:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:31 smithi146 ceph-mon[103278]: pgmap v11835: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:31 smithi078 ceph-mon[91136]: pgmap v11835: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:31 smithi078 ceph-mon[99959]: pgmap v11835: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:31.823Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:31.823Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:31.824Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:33 smithi146 ceph-mon[103278]: pgmap v11836: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:33 smithi078 ceph-mon[91136]: pgmap v11836: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:33 smithi078 ceph-mon[99959]: pgmap v11836: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:35 smithi146 ceph-mon[103278]: pgmap v11837: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:35 smithi078 ceph-mon[91136]: pgmap v11837: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:35 smithi078 ceph-mon[99959]: pgmap v11837: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:38.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:37 smithi078 ceph-mon[99959]: pgmap v11838: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:38.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:37 smithi078 ceph-mon[91136]: pgmap v11838: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:38.255 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:37 smithi146 ceph-mon[103278]: pgmap v11838: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:39.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:40 smithi146 ceph-mon[103278]: pgmap v11839: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:40 smithi078 ceph-mon[91136]: pgmap v11839: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:40 smithi078 ceph-mon[99959]: pgmap v11839: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:54:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T02:54:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:42.078 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:41.824Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:42.078 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:41.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:42.078 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:41.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:42 smithi146 ceph-mon[103278]: pgmap v11840: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:42 smithi078 ceph-mon[91136]: pgmap v11840: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:42 smithi078 ceph-mon[99959]: pgmap v11840: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:43.494 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:54:44.020 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:54:44.020 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 6h - - 2024-09-21T02:54:44.020 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 6h - - 2024-09-21T02:54:44.020 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 5s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:54:44.020 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 0s ago 6h 87.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 5s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 0s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 5s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 5s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 0s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 5s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 5s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 0s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 5s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 5s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 5s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 0s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 0s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:54:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 0s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:54:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 0s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:54:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 0s ago 6h 76.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:54:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 5s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:54:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:54:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 5s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:54:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:54:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[91136]: pgmap v11841: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[99959]: pgmap v11841: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:54:44.335 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:44 smithi146 ceph-mon[103278]: pgmap v11841: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:54:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:54:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:54:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:54:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:54:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:54:44.606 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:54:44.914 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:54:44.914 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:54:44.914 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:54:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[99959]: from='client.90818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[99959]: from='client.70905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[99959]: from='client.90830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4038571695' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3035510213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[91136]: from='client.90818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[91136]: from='client.70905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[91136]: from='client.90830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4038571695' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3035510213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:45 smithi146 ceph-mon[103278]: from='client.90818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:45 smithi146 ceph-mon[103278]: from='client.70905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:45 smithi146 ceph-mon[103278]: from='client.90830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4038571695' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:54:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3035510213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:54:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:46 smithi078 ceph-mon[91136]: from='client.90842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:46 smithi078 ceph-mon[91136]: pgmap v11842: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:46 smithi078 ceph-mon[99959]: from='client.90842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:46 smithi078 ceph-mon[99959]: pgmap v11842: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:46 smithi146 ceph-mon[103278]: from='client.90842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:54:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:46 smithi146 ceph-mon[103278]: pgmap v11842: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:48 smithi078 ceph-mon[99959]: pgmap v11843: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:48 smithi078 ceph-mon[91136]: pgmap v11843: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:48.327 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:48 smithi146 ceph-mon[103278]: pgmap v11843: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:50 smithi146 ceph-mon[103278]: pgmap v11844: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:50 smithi078 ceph-mon[91136]: pgmap v11844: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:50 smithi078 ceph-mon[99959]: pgmap v11844: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:54:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:54:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:54:52.077 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:51.825Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:52.078 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:51.825Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:54:52.078 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:54:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:54:51.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:54:52.311 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:52 smithi146 ceph-mon[103278]: pgmap v11845: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:52 smithi078 ceph-mon[91136]: pgmap v11845: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:52 smithi078 ceph-mon[99959]: pgmap v11845: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:54.339 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:54 smithi146 ceph-mon[103278]: pgmap v11846: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:54 smithi078 ceph-mon[91136]: pgmap v11846: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:54 smithi078 ceph-mon[99959]: pgmap v11846: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:56 smithi146 ceph-mon[103278]: pgmap v11847: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:56 smithi078 ceph-mon[91136]: pgmap v11847: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:56 smithi078 ceph-mon[99959]: pgmap v11847: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:54:58.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:58 smithi078 ceph-mon[91136]: pgmap v11848: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:58.491 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:58 smithi078 ceph-mon[99959]: pgmap v11848: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:58 smithi146 ceph-mon[103278]: pgmap v11848: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:54:59.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:54:59.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:00 smithi146 ceph-mon[103278]: pgmap v11849: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:00 smithi078 ceph-mon[91136]: pgmap v11849: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:00 smithi078 ceph-mon[99959]: pgmap v11849: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:55:00] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:55:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:02.126 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:01.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:02.126 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:01.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:02.126 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:01.826Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:02 smithi146 ceph-mon[103278]: pgmap v11850: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:02 smithi078 ceph-mon[91136]: pgmap v11850: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:02 smithi078 ceph-mon[99959]: pgmap v11850: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:04 smithi146 ceph-mon[103278]: pgmap v11851: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[99959]: pgmap v11851: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[91136]: pgmap v11851: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:06 smithi146 ceph-mon[103278]: pgmap v11852: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:06.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:06 smithi078 ceph-mon[91136]: pgmap v11852: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:06.440 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:06 smithi078 ceph-mon[99959]: pgmap v11852: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:08 smithi146 ceph-mon[103278]: pgmap v11853: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:08 smithi078 ceph-mon[91136]: pgmap v11853: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:08 smithi078 ceph-mon[99959]: pgmap v11853: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:10 smithi146 ceph-mon[103278]: pgmap v11854: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:10 smithi078 ceph-mon[91136]: pgmap v11854: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:10 smithi078 ceph-mon[99959]: pgmap v11854: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:55:10] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T02:55:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:12.201 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:11.826Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:12.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:11.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:12.202 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:11.827Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:12.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:12 smithi146 ceph-mon[103278]: pgmap v11855: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:12 smithi078 ceph-mon[91136]: pgmap v11855: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:12 smithi078 ceph-mon[99959]: pgmap v11855: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:14 smithi146 ceph-mon[103278]: pgmap v11856: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:14 smithi078 ceph-mon[91136]: pgmap v11856: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:14 smithi078 ceph-mon[99959]: pgmap v11856: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:15.213 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:55:15.740 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:55:15.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 6h - - 2024-09-21T02:55:15.740 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 6h - - 2024-09-21T02:55:15.740 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 17s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:55:15.740 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 12s ago 6h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 17s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 12s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 17s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 17s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 12s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 17s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 17s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 12s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 17s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 17s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 17s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 17s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 12s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 12s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:55:15.741 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 12s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:55:15.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 12s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:55:15.742 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 12s ago 6h 77.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:55:15.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 17s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:55:15.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:55:15.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 17s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:55:15.742 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:55:16.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:55:16.048 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:55:16.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:55:16.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:55:16.050 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:55:16.321 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:55:16.322 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:55:16.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:16 smithi078 ceph-mon[99959]: pgmap v11857: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:16.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:16 smithi078 ceph-mon[99959]: from='client.70935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:16.323 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3395026454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:16.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:16 smithi078 ceph-mon[91136]: pgmap v11857: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:16.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:16 smithi078 ceph-mon[91136]: from='client.70935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:16.323 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3395026454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:16 smithi146 ceph-mon[103278]: pgmap v11857: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:16 smithi146 ceph-mon[103278]: from='client.70935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3395026454' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:16.633 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:55:16.633 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:16.633 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:16.634 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:16.634 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:17 smithi146 ceph-mon[103278]: from='client.70941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:17 smithi146 ceph-mon[103278]: from='client.90866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:17 smithi146 ceph-mon[103278]: from='client.90878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/141422050' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[99959]: from='client.70941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[99959]: from='client.90866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[99959]: from='client.90878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/141422050' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[91136]: from='client.70941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[91136]: from='client.90866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[91136]: from='client.90878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/141422050' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:18 smithi146 ceph-mon[103278]: pgmap v11858: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:18 smithi078 ceph-mon[91136]: pgmap v11858: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:18 smithi078 ceph-mon[99959]: pgmap v11858: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:19.637 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.637 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.637 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:20.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:20 smithi078 ceph-mon[99959]: pgmap v11859: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:20 smithi078 ceph-mon[91136]: pgmap v11859: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:20.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:20 smithi146 ceph-mon[103278]: pgmap v11859: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:55:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:55:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:21.827Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:21.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:21.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:22 smithi146 ceph-mon[103278]: pgmap v11860: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:22 smithi078 ceph-mon[91136]: pgmap v11860: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:22 smithi078 ceph-mon[99959]: pgmap v11860: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:24 smithi146 ceph-mon[103278]: pgmap v11861: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[99959]: pgmap v11861: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[91136]: pgmap v11861: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:26 smithi146 ceph-mon[103278]: pgmap v11862: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:26 smithi078 ceph-mon[99959]: pgmap v11862: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:26 smithi078 ceph-mon[91136]: pgmap v11862: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:28.650 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:28 smithi078 ceph-mon[99959]: pgmap v11863: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:28.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:28 smithi078 ceph-mon[91136]: pgmap v11863: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:28.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:28 smithi146 ceph-mon[103278]: pgmap v11863: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:29.638 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:29.639 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:30 smithi078 ceph-mon[91136]: pgmap v11864: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:30 smithi078 ceph-mon[99959]: pgmap v11864: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:30 smithi146 ceph-mon[103278]: pgmap v11864: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:55:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:55:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:31.828Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:31.828Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:31.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:32 smithi146 ceph-mon[103278]: pgmap v11865: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:32 smithi078 ceph-mon[99959]: pgmap v11865: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:32 smithi078 ceph-mon[91136]: pgmap v11865: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:34.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:34 smithi146 ceph-mon[103278]: pgmap v11866: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:34 smithi078 ceph-mon[91136]: pgmap v11866: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:34 smithi078 ceph-mon[99959]: pgmap v11866: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T02:55:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:36 smithi146 ceph-mon[103278]: pgmap v11867: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:36 smithi078 ceph-mon[99959]: pgmap v11867: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:36 smithi078 ceph-mon[91136]: pgmap v11867: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:38 smithi146 ceph-mon[103278]: pgmap v11868: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:38 smithi078 ceph-mon[99959]: pgmap v11868: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:38 smithi078 ceph-mon[91136]: pgmap v11868: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:39 smithi146 ceph-mon[103278]: pgmap v11869: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[91136]: pgmap v11869: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:39.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:39 smithi078 ceph-mon[99959]: pgmap v11869: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:55:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:55:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:41 smithi146 ceph-mon[103278]: pgmap v11870: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:41 smithi078 ceph-mon[91136]: pgmap v11870: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:41 smithi078 ceph-mon[99959]: pgmap v11870: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:41.829Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:41.829Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:41.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:44 smithi146 ceph-mon[103278]: pgmap v11871: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[91136]: pgmap v11871: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[99959]: pgmap v11871: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:55:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:55:46.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:46 smithi146 ceph-mon[103278]: pgmap v11872: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:46 smithi078 ceph-mon[91136]: pgmap v11872: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:46 smithi078 ceph-mon[99959]: pgmap v11872: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:46.928 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:55:47.454 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:55:47.454 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:55:47.455 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 6h - - 2024-09-21T02:55:47.455 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:55:47.455 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 4s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:55:47.455 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:55:47.455 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 4s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:55:47.455 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:55:47.456 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:55:47.456 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 4s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:55:47.456 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:55:47.456 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:55:47.456 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 4s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:55:47.456 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 9s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:55:47.457 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:55:47.457 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 9s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:55:47.457 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:55:47.457 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 4s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:55:47.457 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 4s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:55:47.457 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 4s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:55:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 4s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:55:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 4s ago 6h 77.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:55:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:55:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:55:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:55:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:55:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:55:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:55:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:55:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:55:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:55:47.763 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[91136]: from='client.70971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[91136]: pgmap v11873: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[91136]: from='client.70977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2405133215' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[99959]: from='client.70971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[99959]: pgmap v11873: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[99959]: from='client.70977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2405133215' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:48.045 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:55:48.045 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:55:48.045 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:55:48.046 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:55:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:48 smithi146 ceph-mon[103278]: from='client.70971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:48 smithi146 ceph-mon[103278]: pgmap v11873: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:48 smithi146 ceph-mon[103278]: from='client.70977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2405133215' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:55:48.365 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:55:48.365 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:48.365 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:55:48.365 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:48.365 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:55:49.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:49 smithi078 ceph-mon[99959]: from='client.90902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:49.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:49 smithi078 ceph-mon[99959]: from='client.90914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:49.253 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4121495611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:49.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:49 smithi078 ceph-mon[91136]: from='client.90902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:49.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:49 smithi078 ceph-mon[91136]: from='client.90914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:49.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4121495611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:49.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:49 smithi146 ceph-mon[103278]: from='client.90902 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:49 smithi146 ceph-mon[103278]: from='client.90914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:55:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4121495611' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:55:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:50 smithi078 ceph-mon[91136]: pgmap v11874: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:50 smithi078 ceph-mon[99959]: pgmap v11874: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:50 smithi146 ceph-mon[103278]: pgmap v11874: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:55:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:55:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:55:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:52 smithi078 ceph-mon[91136]: pgmap v11875: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:52 smithi078 ceph-mon[99959]: pgmap v11875: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:51.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:51.830Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:55:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:55:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:55:51.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:55:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:52 smithi146 ceph-mon[103278]: pgmap v11875: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:54.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:54 smithi146 ceph-mon[103278]: pgmap v11876: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:54 smithi078 ceph-mon[91136]: pgmap v11876: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:54.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:54 smithi078 ceph-mon[99959]: pgmap v11876: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:56.117 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:56 smithi146 ceph-mon[103278]: pgmap v11877: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:56 smithi078 ceph-mon[91136]: pgmap v11877: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:56 smithi078 ceph-mon[99959]: pgmap v11877: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:55:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:58 smithi078 ceph-mon[91136]: pgmap v11878: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:58 smithi078 ceph-mon[99959]: pgmap v11878: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:55:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:58 smithi146 ceph-mon[103278]: pgmap v11878: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:55:59 smithi146 ceph-mon[103278]: pgmap v11879: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[99959]: pgmap v11879: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:55:59 smithi078 ceph-mon[91136]: pgmap v11879: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:01.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:56:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:56:02.015 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:01 smithi146 ceph-mon[103278]: pgmap v11880: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:01 smithi078 ceph-mon[91136]: pgmap v11880: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:02.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:01 smithi078 ceph-mon[99959]: pgmap v11880: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:01.830Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:01.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:01.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:04 smithi146 ceph-mon[103278]: pgmap v11881: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[99959]: pgmap v11881: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[91136]: pgmap v11881: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:06 smithi146 ceph-mon[103278]: pgmap v11882: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:06 smithi078 ceph-mon[91136]: pgmap v11882: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:06 smithi078 ceph-mon[99959]: pgmap v11882: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:08 smithi078 ceph-mon[91136]: pgmap v11883: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:08 smithi078 ceph-mon[99959]: pgmap v11883: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:08.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:08 smithi146 ceph-mon[103278]: pgmap v11883: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:10 smithi078 ceph-mon[99959]: pgmap v11884: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:10 smithi078 ceph-mon[91136]: pgmap v11884: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:10 smithi146 ceph-mon[103278]: pgmap v11884: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:11.010 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:56:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:56:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:12 smithi078 ceph-mon[91136]: pgmap v11885: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:12 smithi078 ceph-mon[99959]: pgmap v11885: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:11.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:11.831Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:11.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:12 smithi146 ceph-mon[103278]: pgmap v11885: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:14 smithi078 ceph-mon[91136]: pgmap v11886: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:14 smithi078 ceph-mon[99959]: pgmap v11886: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:14.356 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:14 smithi146 ceph-mon[103278]: pgmap v11886: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:14.356 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:16.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:16 smithi146 ceph-mon[103278]: pgmap v11887: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:16 smithi078 ceph-mon[91136]: pgmap v11887: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:16 smithi078 ceph-mon[99959]: pgmap v11887: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:18 smithi146 ceph-mon[103278]: pgmap v11888: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:18 smithi078 ceph-mon[99959]: pgmap v11888: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:18 smithi078 ceph-mon[91136]: pgmap v11888: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:18.661 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 6h - - 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 6h - - 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 0s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 15s ago 6h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 0s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:56:19.192 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 15s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 0s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 0s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 15s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 0s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 0s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 15s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 0s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 0s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 0s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 15s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 15s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 15s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 15s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 15s ago 6h 77.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:56:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 0s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:56:19.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:56:19.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 0s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:56:19.194 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:56:19.515 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:56:19.515 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:56:19.515 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:56:19.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:19.515 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:56:19.516 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='client.71007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='client.71013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: pgmap v11889: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='client.71019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/355157320' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='client.71007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='client.71013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: pgmap v11889: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='client.71019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/355157320' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:56:19.797 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:56:19.798 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:56:19.798 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:56:19.798 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:56:19.798 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:56:19.798 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:56:19.798 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='client.71007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='client.71013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: pgmap v11889: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='client.71019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/355157320' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:20.104 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:56:20.104 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:20.104 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:20.104 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:20.104 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:20.976 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:56:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:56:20.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:20.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-mon[99959]: from='client.90950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:20.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2199676019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:20.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:20.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-mon[91136]: from='client.90950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:20.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2199676019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:20 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:20 smithi146 ceph-mon[103278]: from='client.90950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2199676019' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:21.832 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:21 smithi078 ceph-mon[99959]: pgmap v11890: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:21.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:21 smithi078 ceph-mon[91136]: pgmap v11890: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:21 smithi146 ceph-mon[103278]: pgmap v11890: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:21.831Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:21.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:21.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:24 smithi146 ceph-mon[103278]: pgmap v11891: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[99959]: pgmap v11891: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[91136]: pgmap v11891: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:26 smithi078 ceph-mon[91136]: pgmap v11892: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:26 smithi078 ceph-mon[99959]: pgmap v11892: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:26 smithi146 ceph-mon[103278]: pgmap v11892: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:28 smithi078 ceph-mon[91136]: pgmap v11893: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:28 smithi078 ceph-mon[99959]: pgmap v11893: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:28.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:28 smithi146 ceph-mon[103278]: pgmap v11893: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:30 smithi078 ceph-mon[91136]: pgmap v11894: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:30 smithi078 ceph-mon[99959]: pgmap v11894: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:30 smithi146 ceph-mon[103278]: pgmap v11894: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:30.671 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:56:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:56:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:32 smithi078 ceph-mon[91136]: pgmap v11895: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:32 smithi078 ceph-mon[99959]: pgmap v11895: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:31.832Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:31.832Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:31.833Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:32 smithi146 ceph-mon[103278]: pgmap v11895: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:34 smithi078 ceph-mon[91136]: pgmap v11896: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:34 smithi078 ceph-mon[99959]: pgmap v11896: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:34 smithi146 ceph-mon[103278]: pgmap v11896: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:36 smithi146 ceph-mon[103278]: pgmap v11897: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:36 smithi078 ceph-mon[91136]: pgmap v11897: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:36 smithi078 ceph-mon[99959]: pgmap v11897: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:38 smithi146 ceph-mon[103278]: pgmap v11898: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:38 smithi078 ceph-mon[91136]: pgmap v11898: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:38 smithi078 ceph-mon[99959]: pgmap v11898: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[91136]: pgmap v11899: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:39 smithi078 ceph-mon[99959]: pgmap v11899: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:39 smithi146 ceph-mon[103278]: pgmap v11899: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:40 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:41.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:40 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:56:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:56:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:40 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:41 smithi146 ceph-mon[103278]: pgmap v11900: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:41 smithi078 ceph-mon[99959]: pgmap v11900: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:41 smithi078 ceph-mon[91136]: pgmap v11900: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:41.833Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:41.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:41.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:44 smithi146 ceph-mon[103278]: pgmap v11901: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[91136]: pgmap v11901: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[99959]: pgmap v11901: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:44.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:46 smithi078 ceph-mon[99959]: pgmap v11902: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:46 smithi078 ceph-mon[91136]: pgmap v11902: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:46 smithi146 ceph-mon[103278]: pgmap v11902: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:48 smithi078 ceph-mon[91136]: pgmap v11903: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:48.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:48 smithi078 ceph-mon[99959]: pgmap v11903: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:48 smithi146 ceph-mon[103278]: pgmap v11903: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:50 smithi078 ceph-mon[99959]: pgmap v11904: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:50 smithi078 ceph-mon[91136]: pgmap v11904: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:50.403 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:56:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:50 smithi146 ceph-mon[103278]: pgmap v11904: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:50.934 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:56:50.934 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 6h - - 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 6h - - 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 12s ago 6h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 7s ago 6h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 12s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 7s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 12s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 12s ago 6h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 7s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 12s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 12s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 7s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:56:50.935 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 12s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 12s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 12s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 7s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 7s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 7s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 7s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 7s ago 6h 77.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 12s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 12s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:56:50.936 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:56:50.936 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:56:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:56:51.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:51.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-mon[91136]: from='client.90962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:51.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:51.231 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-mon[99959]: from='client.90962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:51.257 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:56:51.257 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:56:51.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:56:51.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:56:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:56:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:51 smithi146 ceph-mon[103278]: from='client.90962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:56:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:56:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:56:51.540 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:56:51.860 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:56:51.860 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:56:51.860 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:56:52.234 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[99959]: from='client.71049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[99959]: from='client.71055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[99959]: pgmap v11905: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2064349484' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3390490946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[91136]: from='client.71049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[91136]: from='client.71055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[91136]: pgmap v11905: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2064349484' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:52.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3390490946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:52.236 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:51.834Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:52.236 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:51.834Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:56:52.236 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:56:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:56:51.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:56:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:52 smithi146 ceph-mon[103278]: from='client.71049 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:52 smithi146 ceph-mon[103278]: from='client.71055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:52 smithi146 ceph-mon[103278]: pgmap v11905: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2064349484' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:56:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3390490946' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:56:53.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:53 smithi078 ceph-mon[91136]: from='client.71067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:53.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:53 smithi078 ceph-mon[99959]: from='client.71067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:53 smithi146 ceph-mon[103278]: from='client.71067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:56:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:54 smithi146 ceph-mon[103278]: pgmap v11906: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:54 smithi078 ceph-mon[91136]: pgmap v11906: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:54 smithi078 ceph-mon[99959]: pgmap v11906: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:56 smithi146 ceph-mon[103278]: pgmap v11907: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:56 smithi078 ceph-mon[91136]: pgmap v11907: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:56 smithi078 ceph-mon[99959]: pgmap v11907: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:56:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:58 smithi146 ceph-mon[103278]: pgmap v11908: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:58 smithi078 ceph-mon[91136]: pgmap v11908: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:58 smithi078 ceph-mon[99959]: pgmap v11908: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:56:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:59.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:56:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:56:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:56:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:56:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:00 smithi146 ceph-mon[103278]: pgmap v11909: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:00 smithi078 ceph-mon[91136]: pgmap v11909: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:00 smithi078 ceph-mon[99959]: pgmap v11909: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:57:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:57:01.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:02.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:02 smithi078 ceph-mon[91136]: pgmap v11910: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:02 smithi078 ceph-mon[99959]: pgmap v11910: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:01.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:01.835Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:01.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:02.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:02 smithi146 ceph-mon[103278]: pgmap v11910: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:04 smithi146 ceph-mon[103278]: pgmap v11911: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[99959]: pgmap v11911: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[91136]: pgmap v11911: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:06 smithi146 ceph-mon[103278]: pgmap v11912: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:06 smithi078 ceph-mon[91136]: pgmap v11912: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:06 smithi078 ceph-mon[99959]: pgmap v11912: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:08 smithi146 ceph-mon[103278]: pgmap v11913: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:08 smithi078 ceph-mon[99959]: pgmap v11913: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:08 smithi078 ceph-mon[91136]: pgmap v11913: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:10 smithi146 ceph-mon[103278]: pgmap v11914: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:10 smithi078 ceph-mon[91136]: pgmap v11914: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:10 smithi078 ceph-mon[99959]: pgmap v11914: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:10.896 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:57:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T02:57:11.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:11.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:11.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:12.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:11.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:11.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:12 smithi146 ceph-mon[103278]: pgmap v11915: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:12 smithi078 ceph-mon[91136]: pgmap v11915: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:12 smithi078 ceph-mon[99959]: pgmap v11915: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:14 smithi146 ceph-mon[103278]: pgmap v11916: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:14 smithi078 ceph-mon[99959]: pgmap v11916: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:14 smithi078 ceph-mon[91136]: pgmap v11916: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:16.402 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:16 smithi146 ceph-mon[103278]: pgmap v11917: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:16 smithi078 ceph-mon[99959]: pgmap v11917: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:16 smithi078 ceph-mon[91136]: pgmap v11917: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:18 smithi146 ceph-mon[103278]: pgmap v11918: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:18 smithi078 ceph-mon[91136]: pgmap v11918: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:18 smithi078 ceph-mon[99959]: pgmap v11918: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:20 smithi146 ceph-mon[103278]: pgmap v11919: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[91136]: pgmap v11919: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[99959]: pgmap v11919: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:20.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:57:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:57:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:22.086 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:21.835Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:22.087 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:21.836Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:22.087 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:21.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:22.157 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:57:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:22 smithi146 ceph-mon[103278]: pgmap v11920: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:22 smithi078 ceph-mon[99959]: pgmap v11920: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:22 smithi078 ceph-mon[91136]: pgmap v11920: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:22.683 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 6h - - 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 6h - - 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 3s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 18s ago 6h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 3s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 18s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 3s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 3s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 18s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 3s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 3s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 18s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 3s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:57:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 3s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 3s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 3s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 18s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 18s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 18s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 18s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 18s ago 6h 77.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 3s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 3s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:57:22.685 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:57:22.992 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:57:22.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:57:22.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:57:22.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:22.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:57:22.992 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:57:22.993 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:57:23.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:57:23.275 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:57:23.275 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:57:23.276 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:57:23.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:23 smithi078 ceph-mon[91136]: from='client.90998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:23.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1685557109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:23.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:23 smithi078 ceph-mon[99959]: from='client.90998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:23.277 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1685557109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:23.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:23 smithi146 ceph-mon[103278]: from='client.90998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1685557109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:23.599 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:57:23.599 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:23.599 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: from='client.71085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: from='client.71091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: pgmap v11921: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: from='client.71103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/874761177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: from='client.71085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: from='client.71091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: pgmap v11921: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: from='client.71103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/874761177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: from='client.71085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: from='client.71091 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: pgmap v11921: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: from='client.71103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/874761177' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:25.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:26 smithi146 ceph-mon[103278]: pgmap v11922: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:26 smithi078 ceph-mon[91136]: pgmap v11922: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:26 smithi078 ceph-mon[99959]: pgmap v11922: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:28 smithi146 ceph-mon[103278]: pgmap v11923: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:28 smithi078 ceph-mon[91136]: pgmap v11923: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:28 smithi078 ceph-mon[99959]: pgmap v11923: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:30 smithi146 ceph-mon[103278]: pgmap v11924: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:30 smithi078 ceph-mon[91136]: pgmap v11924: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:30 smithi078 ceph-mon[99959]: pgmap v11924: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:57:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:57:31.496 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:31.496 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:32.127 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:31.837Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:32.127 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:31.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:32.127 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:31.837Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:32.435 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:32 smithi078 ceph-mon[99959]: pgmap v11925: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:32.435 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:32 smithi078 ceph-mon[91136]: pgmap v11925: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:32.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:32 smithi146 ceph-mon[103278]: pgmap v11925: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:34 smithi146 ceph-mon[103278]: pgmap v11926: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:34 smithi078 ceph-mon[91136]: pgmap v11926: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:34 smithi078 ceph-mon[99959]: pgmap v11926: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:36 smithi146 ceph-mon[103278]: pgmap v11927: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:36 smithi078 ceph-mon[91136]: pgmap v11927: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:36 smithi078 ceph-mon[99959]: pgmap v11927: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:38.260 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:38 smithi146 ceph-mon[103278]: pgmap v11928: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:38 smithi078 ceph-mon[91136]: pgmap v11928: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:38 smithi078 ceph-mon[99959]: pgmap v11928: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:40 smithi146 ceph-mon[103278]: pgmap v11929: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[91136]: pgmap v11929: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[99959]: pgmap v11929: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:57:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:57:41.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:42.217 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:41.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:42.217 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:41.838Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:42.217 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:41.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:42 smithi146 ceph-mon[103278]: pgmap v11930: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:42 smithi078 ceph-mon[91136]: pgmap v11930: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:42 smithi078 ceph-mon[99959]: pgmap v11930: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:44 smithi146 ceph-mon[103278]: pgmap v11931: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:44 smithi078 ceph-mon[91136]: pgmap v11931: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:44 smithi078 ceph-mon[99959]: pgmap v11931: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:57:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:46 smithi146 ceph-mon[103278]: pgmap v11932: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:46 smithi078 ceph-mon[99959]: pgmap v11932: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:46 smithi078 ceph-mon[91136]: pgmap v11932: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:48 smithi146 ceph-mon[103278]: pgmap v11933: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:48 smithi078 ceph-mon[99959]: pgmap v11933: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:48 smithi078 ceph-mon[91136]: pgmap v11933: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:50 smithi146 ceph-mon[103278]: pgmap v11934: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:50 smithi078 ceph-mon[91136]: pgmap v11934: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:50 smithi078 ceph-mon[99959]: pgmap v11934: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:57:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:57:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:57:52.163 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:51.838Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:52.163 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:51.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:57:52.163 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:57:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:57:51.839Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:57:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:52 smithi146 ceph-mon[103278]: pgmap v11935: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:52 smithi078 ceph-mon[91136]: pgmap v11935: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:52 smithi078 ceph-mon[99959]: pgmap v11935: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:53.902 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:57:54.435 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:57:54.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 6h - - 2024-09-21T02:57:54.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 6h - - 2024-09-21T02:57:54.435 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 15s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 10s ago 6h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 15s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 10s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 15s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 15s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 10s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 15s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 15s ago 6h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 10s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 15s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 15s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:57:54.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 15s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 10s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 10s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 10s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 10s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 10s ago 6h 78.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 15s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 15s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:57:54.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:57:54.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:54 smithi078 ceph-mon[91136]: pgmap v11936: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:54.438 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:54 smithi078 ceph-mon[99959]: pgmap v11936: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:54.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:54 smithi146 ceph-mon[103278]: pgmap v11936: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:57:54.747 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:57:54.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:57:54.749 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:57:54.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:57:54.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:57:54.749 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:57:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:57:55.015 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:57:55.015 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:57:55.015 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:57:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:55 smithi078 ceph-mon[99959]: from='client.71115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:55 smithi078 ceph-mon[99959]: from='client.91040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:55.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2310917852' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:55 smithi078 ceph-mon[91136]: from='client.71115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:55 smithi078 ceph-mon[91136]: from='client.91040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2310917852' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:55.326 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:57:55.327 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:57:55.327 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:55.327 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:57:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:55 smithi146 ceph-mon[103278]: from='client.71115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:55 smithi146 ceph-mon[103278]: from='client.91040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2310917852' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:57:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:56 smithi146 ceph-mon[103278]: from='client.91046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:56 smithi146 ceph-mon[103278]: pgmap v11937: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:56 smithi146 ceph-mon[103278]: from='client.91058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1819700227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[91136]: from='client.91046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[91136]: pgmap v11937: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[91136]: from='client.91058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1819700227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[99959]: from='client.91046 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[99959]: pgmap v11937: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[99959]: from='client.91058 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:57:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1819700227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:57:58.378 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:58 smithi146 ceph-mon[103278]: pgmap v11938: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:58 smithi078 ceph-mon[91136]: pgmap v11938: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:58 smithi078 ceph-mon[99959]: pgmap v11938: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:57:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:57:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:57:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:57:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:57:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[99959]: pgmap v11939: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[91136]: pgmap v11939: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:00 smithi146 ceph-mon[103278]: pgmap v11939: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:58:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:58:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:01.839Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:01.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:01.840Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:02 smithi146 ceph-mon[103278]: pgmap v11940: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:02 smithi078 ceph-mon[91136]: pgmap v11940: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:02 smithi078 ceph-mon[99959]: pgmap v11940: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:04 smithi146 ceph-mon[103278]: pgmap v11941: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:04 smithi078 ceph-mon[91136]: pgmap v11941: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:04 smithi078 ceph-mon[99959]: pgmap v11941: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:06.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:06 smithi146 ceph-mon[103278]: pgmap v11942: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:06 smithi078 ceph-mon[91136]: pgmap v11942: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:06 smithi078 ceph-mon[99959]: pgmap v11942: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:08 smithi146 ceph-mon[103278]: pgmap v11943: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:08 smithi078 ceph-mon[91136]: pgmap v11943: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:08 smithi078 ceph-mon[99959]: pgmap v11943: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:10 smithi146 ceph-mon[103278]: pgmap v11944: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:10 smithi078 ceph-mon[91136]: pgmap v11944: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:10 smithi078 ceph-mon[99959]: pgmap v11944: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:58:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T02:58:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:11.840Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:11.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:11.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:12.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:12 smithi078 ceph-mon[91136]: pgmap v11945: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:12.764 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:12 smithi078 ceph-mon[99959]: pgmap v11945: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:12.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:12 smithi146 ceph-mon[103278]: pgmap v11945: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:14 smithi078 ceph-mon[91136]: pgmap v11946: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:58:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:14 smithi078 ceph-mon[99959]: pgmap v11946: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:58:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:14 smithi146 ceph-mon[103278]: pgmap v11946: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:58:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:16 smithi146 ceph-mon[103278]: pgmap v11947: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:16 smithi078 ceph-mon[99959]: pgmap v11947: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:16 smithi078 ceph-mon[91136]: pgmap v11947: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:18.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:18 smithi146 ceph-mon[103278]: pgmap v11948: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:58:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:18 smithi078 ceph-mon[91136]: pgmap v11948: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:58:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:18 smithi078 ceph-mon[99959]: pgmap v11948: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T02:58:20.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:20 smithi146 ceph-mon[103278]: pgmap v11949: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:58:20.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:58:20] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:58:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[99959]: pgmap v11949: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[91136]: pgmap v11949: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:21.841Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:21.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:21.841Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:22.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:22 smithi146 ceph-mon[103278]: pgmap v11950: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:22 smithi078 ceph-mon[91136]: pgmap v11950: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:22 smithi078 ceph-mon[99959]: pgmap v11950: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:24 smithi146 ceph-mon[103278]: pgmap v11951: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:24 smithi078 ceph-mon[91136]: pgmap v11951: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:24 smithi078 ceph-mon[99959]: pgmap v11951: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:25.626 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:58:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:25 smithi146 ceph-mon[103278]: pgmap v11952: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[99959]: pgmap v11952: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:25 smithi078 ceph-mon[91136]: pgmap v11952: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:26.150 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:58:26.150 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 6h - - 2024-09-21T02:58:26.150 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 6h - - 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 6s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 1s ago 6h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 6s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 1s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 6s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 6s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 1s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 6s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 6s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 1s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 6s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 6s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 6s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 6s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:58:26.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 1s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 1s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 1s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 1s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 1s ago 6h 78.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 6s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 6s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:58:26.152 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:58:26.460 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:58:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:58:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:58:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:58:26.462 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:58:26.733 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:58:26.734 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:58:26.734 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[99959]: from='client.71151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[99959]: from='client.71157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[99959]: from='client.91082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2126042428' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[91136]: from='client.71151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[91136]: from='client.71157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[91136]: from='client.91082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2126042428' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:26 smithi146 ceph-mon[103278]: from='client.71151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:26 smithi146 ceph-mon[103278]: from='client.71157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:26 smithi146 ceph-mon[103278]: from='client.91082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2126042428' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:27.043 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:58:27.043 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:27.043 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:27 smithi146 ceph-mon[103278]: from='client.71175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:27 smithi146 ceph-mon[103278]: pgmap v11953: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4123398575' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:27 smithi078 ceph-mon[91136]: from='client.71175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:27 smithi078 ceph-mon[91136]: pgmap v11953: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4123398575' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:27 smithi078 ceph-mon[99959]: from='client.71175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:27 smithi078 ceph-mon[99959]: pgmap v11953: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T02:58:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4123398575' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:29 smithi146 ceph-mon[103278]: pgmap v11954: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:29 smithi078 ceph-mon[91136]: pgmap v11954: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:29 smithi078 ceph-mon[99959]: pgmap v11954: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:30.952 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:58:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:58:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:31 smithi146 ceph-mon[103278]: pgmap v11955: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:31 smithi078 ceph-mon[91136]: pgmap v11955: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:31 smithi078 ceph-mon[99959]: pgmap v11955: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:32.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:31.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:31.842Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:31.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:34.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:34 smithi078 ceph-mon[91136]: pgmap v11956: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:34.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:34 smithi078 ceph-mon[99959]: pgmap v11956: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:34.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:34 smithi146 ceph-mon[103278]: pgmap v11956: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T02:58:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:36 smithi078 ceph-mon[91136]: pgmap v11957: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:36 smithi078 ceph-mon[99959]: pgmap v11957: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:36.318 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:36 smithi146 ceph-mon[103278]: pgmap v11957: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:38 smithi146 ceph-mon[103278]: pgmap v11958: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:38 smithi078 ceph-mon[91136]: pgmap v11958: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:38 smithi078 ceph-mon[99959]: pgmap v11958: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T02:58:40.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:40 smithi146 ceph-mon[103278]: pgmap v11959: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:40.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[99959]: pgmap v11959: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[91136]: pgmap v11959: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:58:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:58:41.229 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:42.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:42 smithi078 ceph-mon[99959]: pgmap v11960: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:42 smithi078 ceph-mon[91136]: pgmap v11960: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:41.842Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:41.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:41.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:42 smithi146 ceph-mon[103278]: pgmap v11960: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:44 smithi146 ceph-mon[103278]: pgmap v11961: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:44 smithi078 ceph-mon[91136]: pgmap v11961: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:44 smithi078 ceph-mon[99959]: pgmap v11961: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:45 smithi146 ceph-mon[103278]: pgmap v11962: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[99959]: pgmap v11962: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:58:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:45 smithi078 ceph-mon[91136]: pgmap v11962: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:48 smithi146 ceph-mon[103278]: pgmap v11963: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:48 smithi078 ceph-mon[91136]: pgmap v11963: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:48 smithi078 ceph-mon[99959]: pgmap v11963: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:50 smithi146 ceph-mon[103278]: pgmap v11964: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:50 smithi078 ceph-mon[99959]: pgmap v11964: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:50 smithi078 ceph-mon[91136]: pgmap v11964: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:58:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T02:58:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:58:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:52 smithi078 ceph-mon[91136]: pgmap v11965: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:52.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:52 smithi078 ceph-mon[99959]: pgmap v11965: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:51.843Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:51.843Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:58:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:58:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:58:51.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:58:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:52 smithi146 ceph-mon[103278]: pgmap v11965: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:54 smithi078 ceph-mon[91136]: pgmap v11966: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:54 smithi078 ceph-mon[99959]: pgmap v11966: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:54 smithi146 ceph-mon[103278]: pgmap v11966: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:58:56.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:56 smithi078 ceph-mon[99959]: pgmap v11967: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:56.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:56 smithi078 ceph-mon[91136]: pgmap v11967: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:56 smithi146 ceph-mon[103278]: pgmap v11967: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:57.336 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 18s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 13s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:58:57.863 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 18s ago 6h 15.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 13s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 13s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 13s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 13s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 6h 78.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:58:57.864 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:58:57.865 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:58:58.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:58 smithi078 ceph-mon[91136]: pgmap v11968: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:58.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:58 smithi078 ceph-mon[91136]: from='client.91106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:58.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:58 smithi078 ceph-mon[99959]: pgmap v11968: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:58.172 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:58 smithi078 ceph-mon[99959]: from='client.91106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:58.172 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:58:58.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:58:58.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:58:58.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:58:58.173 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:58:58.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:58:58.436 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:58:58.436 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:58:58.437 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:58:58.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:58 smithi146 ceph-mon[103278]: pgmap v11968: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:58:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:58 smithi146 ceph-mon[103278]: from='client.91106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:58.753 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:58:58.753 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:58.753 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:58:58.753 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:58.753 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:59 smithi146 ceph-mon[103278]: from='client.71193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:59 smithi146 ceph-mon[103278]: from='client.71199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3300682065' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:58:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/948777264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[91136]: from='client.71193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[91136]: from='client.71199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3300682065' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/948777264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[99959]: from='client.71193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[99959]: from='client.71199 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:58:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3300682065' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:58:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:58:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:58:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/948777264' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:00 smithi146 ceph-mon[103278]: from='client.91127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:00 smithi146 ceph-mon[103278]: pgmap v11969: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[99959]: from='client.91127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[99959]: pgmap v11969: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[91136]: from='client.91127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[91136]: pgmap v11969: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:59:00] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:59:01.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:02.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:01.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:02.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:01.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:02.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:01.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:02 smithi146 ceph-mon[103278]: pgmap v11970: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:02 smithi078 ceph-mon[91136]: pgmap v11970: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:02 smithi078 ceph-mon[99959]: pgmap v11970: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:04 smithi146 ceph-mon[103278]: pgmap v11971: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:04 smithi078 ceph-mon[91136]: pgmap v11971: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:04 smithi078 ceph-mon[99959]: pgmap v11971: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:05 smithi146 ceph-mon[103278]: pgmap v11972: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[91136]: pgmap v11972: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:05 smithi078 ceph-mon[99959]: pgmap v11972: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:08 smithi146 ceph-mon[103278]: pgmap v11973: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:08 smithi078 ceph-mon[91136]: pgmap v11973: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:08 smithi078 ceph-mon[99959]: pgmap v11973: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:10 smithi146 ceph-mon[103278]: pgmap v11974: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:10 smithi078 ceph-mon[91136]: pgmap v11974: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:10 smithi078 ceph-mon[99959]: pgmap v11974: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:59:10] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:59:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:12 smithi078 ceph-mon[91136]: pgmap v11975: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:12 smithi078 ceph-mon[99959]: pgmap v11975: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:11.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:11.844Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:11.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:12.358 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:12 smithi146 ceph-mon[103278]: pgmap v11975: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:14 smithi146 ceph-mon[103278]: pgmap v11976: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:14 smithi078 ceph-mon[91136]: pgmap v11976: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:14 smithi078 ceph-mon[99959]: pgmap v11976: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:16 smithi146 ceph-mon[103278]: pgmap v11977: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:16 smithi078 ceph-mon[91136]: pgmap v11977: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:16 smithi078 ceph-mon[99959]: pgmap v11977: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:18 smithi146 ceph-mon[103278]: pgmap v11978: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:18 smithi078 ceph-mon[91136]: pgmap v11978: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:18 smithi078 ceph-mon[99959]: pgmap v11978: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:20.398 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:20 smithi146 ceph-mon[103278]: pgmap v11979: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:20.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.399 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[91136]: pgmap v11979: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[99959]: pgmap v11979: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:59:20] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T02:59:21.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:22.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:21.844Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:22.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:21.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:22.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:21.845Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:22.481 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:22 smithi146 ceph-mon[103278]: pgmap v11980: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:22 smithi078 ceph-mon[91136]: pgmap v11980: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:22 smithi078 ceph-mon[99959]: pgmap v11980: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:24 smithi146 ceph-mon[103278]: pgmap v11981: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:24 smithi078 ceph-mon[91136]: pgmap v11981: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:24 smithi078 ceph-mon[99959]: pgmap v11981: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:25 smithi146 ceph-mon[103278]: pgmap v11982: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[91136]: pgmap v11982: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:26.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:25 smithi078 ceph-mon[99959]: pgmap v11982: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:28 smithi146 ceph-mon[103278]: pgmap v11983: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:28 smithi078 ceph-mon[91136]: pgmap v11983: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:28 smithi078 ceph-mon[99959]: pgmap v11983: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:29.045 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T02:59:29.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 6h - - 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 6h - - 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 4s ago 6h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 4s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 4s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T02:59:29.572 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 4s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 9s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 9s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 9s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 4s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 4s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 4s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 4s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 4s ago 6h 78.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T02:59:29.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T02:59:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T02:59:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T02:59:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T02:59:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T02:59:29.887 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T02:59:30.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T02:59:30.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T02:59:30.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[99959]: pgmap v11984: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:30.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[99959]: from='client.91139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:30.157 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[99959]: from='client.91145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:30.158 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3095442693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:30.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[91136]: pgmap v11984: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:30.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[91136]: from='client.91139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:30.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[91136]: from='client.91145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:30.158 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3095442693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:30.462 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T02:59:30.462 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:59:30.462 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T02:59:30.463 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T02:59:30.463 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T02:59:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:30 smithi146 ceph-mon[103278]: pgmap v11984: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:30 smithi146 ceph-mon[103278]: from='client.91139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:30 smithi146 ceph-mon[103278]: from='client.91145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3095442693' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T02:59:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:59:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:59:31.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:31 smithi146 ceph-mon[103278]: from='client.71235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:31 smithi146 ceph-mon[103278]: from='client.71247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3708577113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[99959]: from='client.71235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[99959]: from='client.71247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3708577113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[91136]: from='client.71235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[91136]: from='client.71247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T02:59:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3708577113' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T02:59:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:32 smithi078 ceph-mon[99959]: pgmap v11985: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:32 smithi078 ceph-mon[91136]: pgmap v11985: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:31.845Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:31.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:31.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:32 smithi146 ceph-mon[103278]: pgmap v11985: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:34 smithi146 ceph-mon[103278]: pgmap v11986: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:34 smithi078 ceph-mon[99959]: pgmap v11986: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:34 smithi078 ceph-mon[91136]: pgmap v11986: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:36.487 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:36 smithi078 ceph-mon[99959]: pgmap v11987: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:36.487 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:36 smithi078 ceph-mon[91136]: pgmap v11987: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:36 smithi146 ceph-mon[103278]: pgmap v11987: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:38.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:38 smithi078 ceph-mon[91136]: pgmap v11988: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:38.426 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:38 smithi078 ceph-mon[99959]: pgmap v11988: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:38 smithi146 ceph-mon[103278]: pgmap v11988: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:40 smithi146 ceph-mon[103278]: pgmap v11989: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[91136]: pgmap v11989: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[99959]: pgmap v11989: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:59:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T02:59:41.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:42.125 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:41.846Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:42.125 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:41.846Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:42.125 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:41.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:42.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:42 smithi146 ceph-mon[103278]: pgmap v11990: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:42 smithi078 ceph-mon[91136]: pgmap v11990: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:42 smithi078 ceph-mon[99959]: pgmap v11990: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:44 smithi146 ceph-mon[103278]: pgmap v11991: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:44 smithi078 ceph-mon[99959]: pgmap v11991: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:44 smithi078 ceph-mon[91136]: pgmap v11991: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:46 smithi146 ceph-mon[103278]: pgmap v11992: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[91136]: pgmap v11992: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T02:59:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:46 smithi078 ceph-mon[99959]: pgmap v11992: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:48 smithi078 ceph-mon[91136]: pgmap v11993: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:48 smithi078 ceph-mon[99959]: pgmap v11993: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:48.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:48 smithi146 ceph-mon[103278]: pgmap v11993: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:50 smithi078 ceph-mon[91136]: pgmap v11994: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:50 smithi078 ceph-mon[99959]: pgmap v11994: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:50.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:50 smithi146 ceph-mon[103278]: pgmap v11994: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 02:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:02:59:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T02:59:51.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T02:59:52.099 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:51.846Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:52.099 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:51.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T02:59:52.099 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 02:59:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T02:59:51.847Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T02:59:52.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:52 smithi146 ceph-mon[103278]: pgmap v11995: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:52 smithi078 ceph-mon[91136]: pgmap v11995: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:52 smithi078 ceph-mon[99959]: pgmap v11995: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:54 smithi146 ceph-mon[103278]: pgmap v11996: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:54 smithi078 ceph-mon[91136]: pgmap v11996: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:54 smithi078 ceph-mon[99959]: pgmap v11996: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T02:59:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:56 smithi078 ceph-mon[91136]: pgmap v11997: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:56 smithi078 ceph-mon[99959]: pgmap v11997: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:56 smithi146 ceph-mon[103278]: pgmap v11997: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:58.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:58 smithi078 ceph-mon[91136]: pgmap v11998: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:58.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:58 smithi078 ceph-mon[99959]: pgmap v11998: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:58 smithi146 ceph-mon[103278]: pgmap v11998: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T02:59:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 02:59:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 02:59:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T02:59:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 02:59:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:00.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T02:59:59.998+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:00 smithi146 ceph-mon[103278]: pgmap v11999: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:00 smithi078 ceph-mon[91136]: pgmap v11999: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:00 smithi078 ceph-mon[99959]: pgmap v11999: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:00.747 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:00:01.016 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 21 03:00:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-21T03:00:00.644Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1726876800138 maxt=1726884000000 ulid=01J898B8F212E2HH8CFFR3CY6D duration=34.31704ms 2024-09-21T03:00:01.016 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 21 03:00:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-21T03:00:00.647Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=2.017568ms 2024-09-21T03:00:01.016 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 21 03:00:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-21T03:00:00.647Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=2 to_segment=3 mint=1726884000000 2024-09-21T03:00:01.016 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Sep 21 03:00:00 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-prometheus-a[83885]: ts=2024-09-21T03:00:00.708Z caller=head.go:1241 level=info component=tsdb msg="WAL checkpoint complete" first=2 last=3 duration=61.376544ms 2024-09-21T03:00:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:00:00] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:00:01.278 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:00:01.278 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 6h - - 2024-09-21T03:00:01.278 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T03:00:01.278 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 1s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 1s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 1s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 1s ago 6h 552M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 1s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 1s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 1s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 1s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 1s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 1s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:00:01.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 16s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 16s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 16s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 6h 78.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 1s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 1s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:00:01.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:00:01.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:01.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:00:01.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:00:01.591 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:00:01.861 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:00:02.153 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:01.847Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:02.153 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:01.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:02.153 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:01.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:02.168 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:02.168 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:00:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:00:02.168 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:00:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:00:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:02 smithi146 ceph-mon[103278]: from='client.71259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:02 smithi146 ceph-mon[103278]: from='client.91181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:02 smithi146 ceph-mon[103278]: pgmap v12000: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:02 smithi146 ceph-mon[103278]: from='client.71271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3453520403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[91136]: from='client.71259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[91136]: from='client.91181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[91136]: pgmap v12000: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[91136]: from='client.71271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3453520403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[99959]: from='client.71259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[99959]: from='client.91181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[99959]: pgmap v12000: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[99959]: from='client.71271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:02.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3453520403' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:03 smithi146 ceph-mon[103278]: from='client.91199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3388510912' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:03 smithi078 ceph-mon[91136]: from='client.91199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3388510912' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:03 smithi078 ceph-mon[99959]: from='client.91199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:03.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3388510912' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:04 smithi146 ceph-mon[103278]: pgmap v12001: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:04 smithi078 ceph-mon[99959]: pgmap v12001: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:04 smithi078 ceph-mon[91136]: pgmap v12001: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:06 smithi146 ceph-mon[103278]: pgmap v12002: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[91136]: pgmap v12002: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[99959]: pgmap v12002: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:08 smithi146 ceph-mon[103278]: pgmap v12003: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:08 smithi078 ceph-mon[91136]: pgmap v12003: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:08 smithi078 ceph-mon[99959]: pgmap v12003: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:10 smithi146 ceph-mon[103278]: pgmap v12004: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:10 smithi078 ceph-mon[91136]: pgmap v12004: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:10 smithi078 ceph-mon[99959]: pgmap v12004: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:00:10] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:00:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:12.152 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:11.848Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:12.152 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:11.848Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:12.152 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:11.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:12 smithi146 ceph-mon[103278]: pgmap v12005: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:12 smithi078 ceph-mon[91136]: pgmap v12005: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:12 smithi078 ceph-mon[99959]: pgmap v12005: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:14 smithi146 ceph-mon[103278]: pgmap v12006: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:14 smithi078 ceph-mon[91136]: pgmap v12006: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:14 smithi078 ceph-mon[99959]: pgmap v12006: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:16 smithi146 ceph-mon[103278]: pgmap v12007: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:16 smithi078 ceph-mon[99959]: pgmap v12007: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:16 smithi078 ceph-mon[91136]: pgmap v12007: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:18.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:18 smithi078 ceph-mon[99959]: pgmap v12008: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:18 smithi078 ceph-mon[91136]: pgmap v12008: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:18 smithi146 ceph-mon[103278]: pgmap v12008: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:20 smithi146 ceph-mon[103278]: pgmap v12009: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:20 smithi078 ceph-mon[91136]: pgmap v12009: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:20 smithi078 ceph-mon[99959]: pgmap v12009: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:00:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:00:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:22.277 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:21.849Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:22.277 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:21.849Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:22.277 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:21.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:22 smithi078 ceph-mon[91136]: pgmap v12010: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:22 smithi078 ceph-mon[99959]: pgmap v12010: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:22 smithi146 ceph-mon[103278]: pgmap v12010: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:24 smithi146 ceph-mon[103278]: pgmap v12011: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:24 smithi078 ceph-mon[91136]: pgmap v12011: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:24 smithi078 ceph-mon[99959]: pgmap v12011: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:26 smithi146 ceph-mon[103278]: pgmap v12012: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[99959]: pgmap v12012: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[91136]: pgmap v12012: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:28 smithi078 ceph-mon[91136]: pgmap v12013: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:28 smithi078 ceph-mon[99959]: pgmap v12013: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:28 smithi146 ceph-mon[103278]: pgmap v12013: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:30 smithi078 ceph-mon[99959]: pgmap v12014: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:30 smithi078 ceph-mon[91136]: pgmap v12014: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:30 smithi146 ceph-mon[103278]: pgmap v12014: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:00:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:00:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:31.850Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:31.850Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:31.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:32.464 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:00:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:32 smithi146 ceph-mon[103278]: pgmap v12015: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:32 smithi078 ceph-mon[99959]: pgmap v12015: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:32 smithi078 ceph-mon[91136]: pgmap v12015: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:32.989 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 6h - - 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 6h - - 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 12s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 7s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 12s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 7s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 12s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 12s ago 6h 553M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 7s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 12s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 12s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:00:32.990 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 7s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 12s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 12s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 12s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 12s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 7s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 7s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 7s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 7s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 7s ago 6h 110M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 12s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 12s ago 6h 90.7M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:00:32.991 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:00:33.300 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:00:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:00:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:00:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:00:33.302 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:00:33.574 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:00:33.575 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:00:33.886 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:00:33.886 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:00:33.887 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:00:33.887 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:00:33.887 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:00:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:34 smithi146 ceph-mon[103278]: from='client.71295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:34 smithi146 ceph-mon[103278]: from='client.91217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:34 smithi146 ceph-mon[103278]: from='client.91223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:34 smithi146 ceph-mon[103278]: pgmap v12016: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/412036300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2807569685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[99959]: from='client.71295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[99959]: from='client.91217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[99959]: from='client.91223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[99959]: pgmap v12016: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/412036300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2807569685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[91136]: from='client.71295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[91136]: from='client.91217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[91136]: from='client.91223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[91136]: pgmap v12016: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/412036300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:00:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2807569685' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:00:35.554 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:35 smithi146 ceph-mon[103278]: from='client.71319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:35 smithi078 ceph-mon[99959]: from='client.71319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:35.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:35 smithi078 ceph-mon[91136]: from='client.71319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:00:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:36 smithi146 ceph-mon[103278]: pgmap v12017: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:36 smithi078 ceph-mon[91136]: pgmap v12017: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:36 smithi078 ceph-mon[99959]: pgmap v12017: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:38 smithi078 ceph-mon[91136]: pgmap v12018: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:38 smithi078 ceph-mon[99959]: pgmap v12018: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:38.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:38 smithi146 ceph-mon[103278]: pgmap v12018: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:40 smithi078 ceph-mon[91136]: pgmap v12019: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:40 smithi078 ceph-mon[99959]: pgmap v12019: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:40 smithi146 ceph-mon[103278]: pgmap v12019: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:00:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:00:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:41.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:41.851Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:41.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:42.653 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:42 smithi146 ceph-mon[103278]: pgmap v12020: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:42 smithi078 ceph-mon[99959]: pgmap v12020: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:42 smithi078 ceph-mon[91136]: pgmap v12020: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:44 smithi146 ceph-mon[103278]: pgmap v12021: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:44 smithi078 ceph-mon[91136]: pgmap v12021: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:44 smithi078 ceph-mon[99959]: pgmap v12021: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:46 smithi146 ceph-mon[103278]: pgmap v12022: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[91136]: pgmap v12022: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[99959]: pgmap v12022: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:00:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:48 smithi146 ceph-mon[103278]: pgmap v12023: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:48 smithi078 ceph-mon[91136]: pgmap v12023: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:48 smithi078 ceph-mon[99959]: pgmap v12023: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:50 smithi146 ceph-mon[103278]: pgmap v12024: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:50 smithi078 ceph-mon[91136]: pgmap v12024: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:50 smithi078 ceph-mon[99959]: pgmap v12024: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:00:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:00:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:00:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:51.851Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:51.852Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:00:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:00:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:00:51.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:00:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:52 smithi146 ceph-mon[103278]: pgmap v12025: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:52 smithi078 ceph-mon[91136]: pgmap v12025: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:52 smithi078 ceph-mon[99959]: pgmap v12025: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:54 smithi146 ceph-mon[103278]: pgmap v12026: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:54 smithi078 ceph-mon[99959]: pgmap v12026: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:54 smithi078 ceph-mon[91136]: pgmap v12026: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:00:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:56 smithi146 ceph-mon[103278]: pgmap v12027: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:56 smithi078 ceph-mon[99959]: pgmap v12027: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:56 smithi078 ceph-mon[91136]: pgmap v12027: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:00:58.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:58 smithi146 ceph-mon[103278]: pgmap v12028: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:00:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:58 smithi078 ceph-mon[91136]: pgmap v12028: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:00:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:58 smithi078 ceph-mon[99959]: pgmap v12028: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:00:59.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:59.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:00:59 smithi078 ceph-mon[99959]: pgmap v12029: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:00:59.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:59.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:00:59 smithi078 ceph-mon[91136]: pgmap v12029: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:00:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:00:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:00:59 smithi146 ceph-mon[103278]: pgmap v12029: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:01:00.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:00 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:00 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:00 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:01:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:01:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:01:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[99959]: pgmap v12030: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:01:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-mon[91136]: pgmap v12030: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:01:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:01 smithi146 ceph-mon[103278]: pgmap v12030: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:01:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:01.852Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:01.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:01.853Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:04.187 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:01:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:04 smithi146 ceph-mon[103278]: pgmap v12031: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:01:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:04 smithi078 ceph-mon[99959]: pgmap v12031: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:01:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:04 smithi078 ceph-mon[91136]: pgmap v12031: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T03:01:04.719 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:01:04.719 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 6h - - 2024-09-21T03:01:04.719 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 6h - - 2024-09-21T03:01:04.719 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 4s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:01:04.719 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 19s ago 6h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:01:04.719 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 4s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 19s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 4s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 4s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 19s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 4s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 4s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 19s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 4s ago 6h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 4s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 4s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 4s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 19s ago 6h 117M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 19s ago 6h 92.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:01:04.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 19s ago 6h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:01:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 19s ago 6h 94.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:01:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 19s ago 6h 70.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:01:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 4s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:01:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:01:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 4s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:01:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:01:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:01:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:01:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:01:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:01:05.037 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:01:05.340 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:01:05.341 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:01:05.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:05 smithi078 ceph-mon[99959]: from='client.71331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:05.342 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3742106574' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:05.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:05 smithi078 ceph-mon[91136]: from='client.71331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:05.342 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3742106574' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:05.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:05 smithi146 ceph-mon[103278]: from='client.71331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:05.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3742106574' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:05.661 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:01:05.662 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:05.662 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='client.71337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='client.71343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: pgmap v12032: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='client.71355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1524937071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='client.71337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='client.71343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: pgmap v12032: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:01:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='client.71355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1524937071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='client.71337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='client.71343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: pgmap v12032: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='client.71355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:06.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1524937071' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:08 smithi146 ceph-mon[103278]: pgmap v12033: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:01:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:08 smithi078 ceph-mon[91136]: pgmap v12033: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:01:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:08 smithi078 ceph-mon[99959]: pgmap v12033: 161 pgs: 161 active+clean; 585 KiB data, 354 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:01:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:10 smithi078 ceph-mon[91136]: pgmap v12034: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T03:01:10.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:10 smithi078 ceph-mon[99959]: pgmap v12034: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T03:01:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:10 smithi146 ceph-mon[103278]: pgmap v12034: 161 pgs: 161 active+clean; 585 KiB data, 358 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T03:01:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:01:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:01:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:01:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:12.133 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:11.853Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:12.133 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:11.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:12.133 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:11.854Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:12 smithi146 ceph-mon[103278]: pgmap v12035: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T03:01:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:12 smithi078 ceph-mon[99959]: pgmap v12035: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T03:01:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:12 smithi078 ceph-mon[91136]: pgmap v12035: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T03:01:14.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:14 smithi146 ceph-mon[103278]: pgmap v12036: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:01:14.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:14 smithi078 ceph-mon[99959]: pgmap v12036: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:01:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:14 smithi078 ceph-mon[91136]: pgmap v12036: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-09-21T03:01:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:16.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:16 smithi146 ceph-mon[103278]: pgmap v12037: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T03:01:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:16 smithi078 ceph-mon[99959]: pgmap v12037: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T03:01:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:16 smithi078 ceph-mon[91136]: pgmap v12037: 161 pgs: 161 active+clean; 585 KiB data, 366 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-09-21T03:01:18.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:18 smithi078 ceph-mon[99959]: pgmap v12038: 161 pgs: 161 active+clean; 585 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:01:18.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:18 smithi078 ceph-mon[91136]: pgmap v12038: 161 pgs: 161 active+clean; 585 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:01:18.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:18 smithi146 ceph-mon[103278]: pgmap v12038: 161 pgs: 161 active+clean; 585 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:01:20.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:20 smithi078 ceph-mon[91136]: pgmap v12039: 161 pgs: 161 active+clean; 585 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:01:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:20 smithi078 ceph-mon[99959]: pgmap v12039: 161 pgs: 161 active+clean; 585 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:01:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:20 smithi146 ceph-mon[103278]: pgmap v12039: 161 pgs: 161 active+clean; 585 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:01:20.752 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:01:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:01:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:01:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:22.149 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:21.854Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:22.149 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:21.855Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:22.149 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:21.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:22 smithi146 ceph-mon[103278]: pgmap v12040: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-21T03:01:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:22 smithi078 ceph-mon[99959]: pgmap v12040: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-21T03:01:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:22 smithi078 ceph-mon[91136]: pgmap v12040: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-09-21T03:01:24.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:24 smithi146 ceph-mon[103278]: pgmap v12041: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T03:01:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:24 smithi078 ceph-mon[91136]: pgmap v12041: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T03:01:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:24 smithi078 ceph-mon[99959]: pgmap v12041: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-09-21T03:01:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:26 smithi146 ceph-mon[103278]: pgmap v12042: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T03:01:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[91136]: pgmap v12042: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[99959]: pgmap v12042: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:28 smithi146 ceph-mon[103278]: pgmap v12043: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T03:01:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:28 smithi078 ceph-mon[91136]: pgmap v12043: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T03:01:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:28 smithi078 ceph-mon[99959]: pgmap v12043: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s 2024-09-21T03:01:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:30 smithi078 ceph-mon[91136]: pgmap v12044: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:01:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:30 smithi078 ceph-mon[99959]: pgmap v12044: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:01:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:30 smithi146 ceph-mon[103278]: pgmap v12044: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:01:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:01:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:01:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:01:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:32.174 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:31.856Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:32.174 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:31.856Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:32.174 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:31.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:32 smithi146 ceph-mon[103278]: pgmap v12045: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:01:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:32 smithi078 ceph-mon[91136]: pgmap v12045: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:01:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:32 smithi078 ceph-mon[99959]: pgmap v12045: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:01:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:34 smithi146 ceph-mon[103278]: pgmap v12046: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:01:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:34 smithi078 ceph-mon[91136]: pgmap v12046: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:01:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:34 smithi078 ceph-mon[99959]: pgmap v12046: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:01:35.963 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:01:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:36 smithi078 ceph-mon[91136]: pgmap v12047: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T03:01:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:36 smithi078 ceph-mon[99959]: pgmap v12047: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T03:01:36.493 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:01:36.493 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 6h - - 2024-09-21T03:01:36.493 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 6h - - 2024-09-21T03:01:36.493 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 15s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:01:36.493 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 10s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 15s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 10s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 15s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 15s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 10s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 15s ago 6h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 15s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 10s ago 6h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 15s ago 6h 95.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 15s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 15s ago 6h 83.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 15s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 10s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 10s ago 6h 92.5M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:01:36.494 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 10s ago 6h 99.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:01:36.495 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 10s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:01:36.495 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 10s ago 6h 71.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:01:36.495 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 15s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:01:36.495 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:01:36.495 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 15s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:01:36.495 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:01:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:36 smithi146 ceph-mon[103278]: pgmap v12047: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-09-21T03:01:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:01:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:01:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:01:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:01:36.803 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:01:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:01:37.071 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:01:37.072 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:01:37.386 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:01:37.386 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:37.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:01:37.386 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:37.386 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:01:37.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:37 smithi078 ceph-mon[99959]: from='client.71367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:37.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:37 smithi078 ceph-mon[99959]: from='client.71373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:37.386 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3099072627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:37.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:37 smithi078 ceph-mon[91136]: from='client.71367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:37.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:37 smithi078 ceph-mon[91136]: from='client.71373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:37.387 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3099072627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:37 smithi146 ceph-mon[103278]: from='client.71367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:37 smithi146 ceph-mon[103278]: from='client.71373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3099072627' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:01:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:38 smithi146 ceph-mon[103278]: from='client.91295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:38 smithi146 ceph-mon[103278]: pgmap v12048: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-21T03:01:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:38 smithi146 ceph-mon[103278]: from='client.71391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3261292869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[99959]: from='client.91295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[99959]: pgmap v12048: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-21T03:01:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[99959]: from='client.71391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3261292869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[91136]: from='client.91295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[91136]: pgmap v12048: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-09-21T03:01:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[91136]: from='client.71391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:01:38.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3261292869' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:01:40.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:40 smithi078 ceph-mon[99959]: pgmap v12049: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:40.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:40 smithi078 ceph-mon[91136]: pgmap v12049: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:40 smithi146 ceph-mon[103278]: pgmap v12049: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:01:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:01:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.470 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:42.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:41.857Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:42.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:41.857Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:42.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:41.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:42 smithi146 ceph-mon[103278]: pgmap v12050: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:42 smithi078 ceph-mon[91136]: pgmap v12050: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:42 smithi078 ceph-mon[99959]: pgmap v12050: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:44 smithi146 ceph-mon[103278]: pgmap v12051: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:44 smithi078 ceph-mon[91136]: pgmap v12051: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:44 smithi078 ceph-mon[99959]: pgmap v12051: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:46.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:46 smithi146 ceph-mon[103278]: pgmap v12052: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:46.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[91136]: pgmap v12052: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[99959]: pgmap v12052: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:01:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:48 smithi146 ceph-mon[103278]: pgmap v12053: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:48 smithi078 ceph-mon[91136]: pgmap v12053: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:48 smithi078 ceph-mon[99959]: pgmap v12053: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:50 smithi078 ceph-mon[91136]: pgmap v12054: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:50 smithi078 ceph-mon[99959]: pgmap v12054: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:50 smithi146 ceph-mon[103278]: pgmap v12054: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:01:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:01:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:01:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:01:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:51.858Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:51.858Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:01:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:01:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:01:51.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:01:52.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:52 smithi078 ceph-mon[91136]: pgmap v12055: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:52.291 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:52 smithi078 ceph-mon[99959]: pgmap v12055: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:52 smithi146 ceph-mon[103278]: pgmap v12055: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:54 smithi146 ceph-mon[103278]: pgmap v12056: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:54 smithi078 ceph-mon[91136]: pgmap v12056: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:54 smithi078 ceph-mon[99959]: pgmap v12056: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:01:56.488 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:56 smithi146 ceph-mon[103278]: pgmap v12057: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:56 smithi078 ceph-mon[91136]: pgmap v12057: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:56 smithi078 ceph-mon[99959]: pgmap v12057: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:58 smithi146 ceph-mon[103278]: pgmap v12058: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:58 smithi078 ceph-mon[91136]: pgmap v12058: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:58 smithi078 ceph-mon[99959]: pgmap v12058: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:01:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:01:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:01:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:01:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:01:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:00 smithi146 ceph-mon[103278]: pgmap v12059: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:00 smithi078 ceph-mon[91136]: pgmap v12059: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:00 smithi078 ceph-mon[99959]: pgmap v12059: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:00.914 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:02:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:02:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:02:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:02.242 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:01.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:02.242 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:01.859Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:02:02.243 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:01.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:02 smithi146 ceph-mon[103278]: pgmap v12060: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:02 smithi078 ceph-mon[91136]: pgmap v12060: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:02 smithi078 ceph-mon[99959]: pgmap v12060: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:04 smithi146 ceph-mon[103278]: pgmap v12061: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:04 smithi078 ceph-mon[91136]: pgmap v12061: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:04 smithi078 ceph-mon[99959]: pgmap v12061: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:06 smithi146 ceph-mon[103278]: pgmap v12062: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[91136]: pgmap v12062: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[99959]: pgmap v12062: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:06.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:07.678 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:02:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:02:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 6h - - 2024-09-21T03:02:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T03:02:08.203 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 7s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 2s ago 6h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 7s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 7s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 7s ago 6h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 7s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 7s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 2s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 7s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 7s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 7s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 7s ago 6h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:02:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 6h 71.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 7s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 7s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:02:08.205 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:02:08.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:02:08.512 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:02:08.512 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:08 smithi078 ceph-mon[99959]: pgmap v12063: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:08.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:08 smithi078 ceph-mon[91136]: pgmap v12063: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:08 smithi146 ceph-mon[103278]: pgmap v12063: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:08.778 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:02:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:02:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:02:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:02:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:02:08.778 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:02:08.779 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:02:09.087 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:02:09.087 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:09.088 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:09 smithi146 ceph-mon[103278]: from='client.91319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:09 smithi146 ceph-mon[103278]: from='client.71409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:09 smithi146 ceph-mon[103278]: from='client.71415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/665847144' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1519582258' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[99959]: from='client.91319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[99959]: from='client.71409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[99959]: from='client.71415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/665847144' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1519582258' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[91136]: from='client.91319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[91136]: from='client.71409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[91136]: from='client.71415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/665847144' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1519582258' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:10 smithi146 ceph-mon[103278]: from='client.71427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:10 smithi146 ceph-mon[103278]: pgmap v12064: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:10 smithi078 ceph-mon[99959]: from='client.71427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:10 smithi078 ceph-mon[99959]: pgmap v12064: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:10 smithi078 ceph-mon[91136]: from='client.71427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:10 smithi078 ceph-mon[91136]: pgmap v12064: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:02:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:02:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:02:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:12.264 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:11.859Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:12.264 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:11.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:02:12.264 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:11.860Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:12 smithi146 ceph-mon[103278]: pgmap v12065: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:12 smithi078 ceph-mon[91136]: pgmap v12065: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:12 smithi078 ceph-mon[99959]: pgmap v12065: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:14 smithi078 ceph-mon[91136]: pgmap v12066: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:14 smithi078 ceph-mon[99959]: pgmap v12066: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:14 smithi146 ceph-mon[103278]: pgmap v12066: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:16 smithi078 ceph-mon[91136]: pgmap v12067: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:16 smithi078 ceph-mon[99959]: pgmap v12067: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:16 smithi146 ceph-mon[103278]: pgmap v12067: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:18.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:18 smithi078 ceph-mon[91136]: pgmap v12068: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:18.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:18 smithi078 ceph-mon[99959]: pgmap v12068: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:18 smithi146 ceph-mon[103278]: pgmap v12068: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:20 smithi078 ceph-mon[91136]: pgmap v12069: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:20 smithi078 ceph-mon[99959]: pgmap v12069: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:20 smithi146 ceph-mon[103278]: pgmap v12069: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:20.939 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:02:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:02:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:02:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:22.143 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:21.860Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:22.144 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:21.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:02:22.144 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:21.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:22 smithi078 ceph-mon[99959]: pgmap v12070: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:22 smithi078 ceph-mon[91136]: pgmap v12070: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:22 smithi146 ceph-mon[103278]: pgmap v12070: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:24 smithi078 ceph-mon[91136]: pgmap v12071: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:24 smithi078 ceph-mon[99959]: pgmap v12071: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:24 smithi146 ceph-mon[103278]: pgmap v12071: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:26.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:26 smithi146 ceph-mon[103278]: pgmap v12072: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:26.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[91136]: pgmap v12072: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[99959]: pgmap v12072: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:28.477 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:28 smithi146 ceph-mon[103278]: pgmap v12073: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:28.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:28 smithi078 ceph-mon[91136]: pgmap v12073: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:28.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:28 smithi078 ceph-mon[99959]: pgmap v12073: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:29.765 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:30 smithi078 ceph-mon[91136]: pgmap v12074: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:30 smithi078 ceph-mon[99959]: pgmap v12074: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:30 smithi146 ceph-mon[103278]: pgmap v12074: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:02:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:02:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:02:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:31.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:31.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:02:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:31.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:32 smithi146 ceph-mon[103278]: pgmap v12075: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:32 smithi078 ceph-mon[99959]: pgmap v12075: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:32 smithi078 ceph-mon[91136]: pgmap v12075: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:34 smithi146 ceph-mon[103278]: pgmap v12076: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:34 smithi078 ceph-mon[91136]: pgmap v12076: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:34 smithi078 ceph-mon[99959]: pgmap v12076: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:36 smithi146 ceph-mon[103278]: pgmap v12077: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:36 smithi078 ceph-mon[91136]: pgmap v12077: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:36 smithi078 ceph-mon[99959]: pgmap v12077: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:38.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:38 smithi146 ceph-mon[103278]: pgmap v12078: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:38 smithi078 ceph-mon[91136]: pgmap v12078: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:38 smithi078 ceph-mon[99959]: pgmap v12078: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:39.382 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 6h - - 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 6h - - 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 18s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 13s ago 6h 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:02:39.909 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 6h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 18s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 13s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 13s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 13s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 13s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:02:39.910 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 6h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:02:39.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:02:39.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:02:39.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:02:39.911 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:02:40.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:02:40.226 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:02:40.496 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:02:40.496 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:02:40.496 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:02:40.497 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:02:40.497 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-mon[99959]: pgmap v12079: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:40.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-mon[99959]: from='client.91355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:40.498 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/990161367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:40.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-mon[91136]: pgmap v12079: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:40.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-mon[91136]: from='client.91355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:40.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/990161367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:40 smithi146 ceph-mon[103278]: pgmap v12079: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:40 smithi146 ceph-mon[103278]: from='client.91355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/990161367' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:02:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:02:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:02:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:02:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:02:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:40.808 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:02:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:40.808 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:02:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='client.91361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='client.91367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1335536415' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='client.91361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='client.91367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1335536415' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='client.91361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:41.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='client.91367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:41.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:41.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1335536415' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:02:41.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:41.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:42.226 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:41.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:42.226 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:41.861Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:02:42.226 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:41.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:42 smithi078 ceph-mon[91136]: from='client.91379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:42 smithi078 ceph-mon[91136]: pgmap v12080: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:02:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:42 smithi078 ceph-mon[99959]: from='client.91379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:42 smithi078 ceph-mon[99959]: pgmap v12080: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:02:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:42 smithi146 ceph-mon[103278]: from='client.91379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:02:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:42 smithi146 ceph-mon[103278]: pgmap v12080: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:02:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:44 smithi146 ceph-mon[103278]: pgmap v12081: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:44 smithi078 ceph-mon[91136]: pgmap v12081: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:44 smithi078 ceph-mon[99959]: pgmap v12081: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:46 smithi146 ceph-mon[103278]: pgmap v12082: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[91136]: pgmap v12082: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[99959]: pgmap v12082: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:02:48.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:48 smithi146 ceph-mon[103278]: pgmap v12083: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:48 smithi078 ceph-mon[91136]: pgmap v12083: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:48 smithi078 ceph-mon[99959]: pgmap v12083: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:50.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:50 smithi078 ceph-mon[91136]: pgmap v12084: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:50 smithi078 ceph-mon[99959]: pgmap v12084: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:50 smithi146 ceph-mon[103278]: pgmap v12084: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:02:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:02:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:02:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:02:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:51.861Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:51.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:02:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:02:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:02:51.862Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:02:52.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:52 smithi078 ceph-mon[91136]: pgmap v12085: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:52.539 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:52 smithi078 ceph-mon[99959]: pgmap v12085: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:52 smithi146 ceph-mon[103278]: pgmap v12085: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:54 smithi146 ceph-mon[103278]: pgmap v12086: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:54 smithi078 ceph-mon[91136]: pgmap v12086: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:54 smithi078 ceph-mon[99959]: pgmap v12086: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:02:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:56 smithi146 ceph-mon[103278]: pgmap v12087: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:56 smithi078 ceph-mon[91136]: pgmap v12087: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:56 smithi078 ceph-mon[99959]: pgmap v12087: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:58 smithi146 ceph-mon[103278]: pgmap v12088: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:58 smithi078 ceph-mon[91136]: pgmap v12088: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:58 smithi078 ceph-mon[99959]: pgmap v12088: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:02:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:02:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:02:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:02:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:02:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:00 smithi146 ceph-mon[103278]: pgmap v12089: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:00 smithi078 ceph-mon[91136]: pgmap v12089: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:00 smithi078 ceph-mon[99959]: pgmap v12089: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:03:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:03:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:03:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:02.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:01.862Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:02.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:01.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:02.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:01.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:02.527 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:02 smithi078 ceph-mon[99959]: pgmap v12090: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:02.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:02 smithi078 ceph-mon[91136]: pgmap v12090: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:02 smithi146 ceph-mon[103278]: pgmap v12090: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:04 smithi146 ceph-mon[103278]: pgmap v12091: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:04 smithi078 ceph-mon[91136]: pgmap v12091: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:04 smithi078 ceph-mon[99959]: pgmap v12091: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:06 smithi146 ceph-mon[103278]: pgmap v12092: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[91136]: pgmap v12092: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[99959]: pgmap v12092: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:08.499 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:08 smithi146 ceph-mon[103278]: pgmap v12093: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:08 smithi078 ceph-mon[91136]: pgmap v12093: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:08 smithi078 ceph-mon[99959]: pgmap v12093: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:10 smithi146 ceph-mon[103278]: pgmap v12094: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:10 smithi078 ceph-mon[99959]: pgmap v12094: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:10 smithi078 ceph-mon[91136]: pgmap v12094: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:03:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:03:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:03:11.103 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:03:11.634 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:03:11.634 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 6h - - 2024-09-21T03:03:11.634 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 6h - - 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 10s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 5s ago 6h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 10s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 5s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 10s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 10s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 5s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 10s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 10s ago 6h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 5s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 10s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 10s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:03:11.635 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 10s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 10s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 5s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 5s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 5s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 5s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 5s ago 6h 66.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 10s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 10s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:03:11.636 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 5s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:03:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:11.954 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:03:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:03:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:03:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:03:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:03:11.955 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:03:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:03:12.221 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:03:12.221 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:03:12.222 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:03:12.223 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:03:12.223 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:11.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:12.223 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:11.863Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:12.223 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:11.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:12.533 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:03:12.533 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:12.533 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:12.533 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[99959]: from='client.71475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[99959]: pgmap v12095: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[99959]: from='client.71481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:12.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2878439884' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[91136]: from='client.71475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:12.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[91136]: pgmap v12095: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[91136]: from='client.71481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2878439884' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:12 smithi146 ceph-mon[103278]: from='client.71475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:12 smithi146 ceph-mon[103278]: pgmap v12095: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:12 smithi146 ceph-mon[103278]: from='client.71481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2878439884' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:13 smithi146 ceph-mon[103278]: from='client.71487 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:13 smithi146 ceph-mon[103278]: from='client.71499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/883367886' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:13 smithi078 ceph-mon[91136]: from='client.71487 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:13 smithi078 ceph-mon[91136]: from='client.71499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/883367886' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:13 smithi078 ceph-mon[99959]: from='client.71487 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:13 smithi078 ceph-mon[99959]: from='client.71499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/883367886' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:14.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:14 smithi078 ceph-mon[99959]: pgmap v12096: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:14.735 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:14.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:14 smithi078 ceph-mon[91136]: pgmap v12096: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:14.735 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:14 smithi146 ceph-mon[103278]: pgmap v12096: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:16 smithi146 ceph-mon[103278]: pgmap v12097: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:16 smithi078 ceph-mon[91136]: pgmap v12097: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:16 smithi078 ceph-mon[99959]: pgmap v12097: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:18 smithi146 ceph-mon[103278]: pgmap v12098: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:18 smithi078 ceph-mon[91136]: pgmap v12098: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:18 smithi078 ceph-mon[99959]: pgmap v12098: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:20 smithi146 ceph-mon[103278]: pgmap v12099: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:20 smithi078 ceph-mon[91136]: pgmap v12099: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:20 smithi078 ceph-mon[99959]: pgmap v12099: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:03:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:03:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:03:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:21.863Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:21.864Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:21.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:22 smithi146 ceph-mon[103278]: pgmap v12100: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:22 smithi078 ceph-mon[91136]: pgmap v12100: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:22 smithi078 ceph-mon[99959]: pgmap v12100: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:24.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:24 smithi078 ceph-mon[91136]: pgmap v12101: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:24.480 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:24 smithi078 ceph-mon[99959]: pgmap v12101: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:24 smithi146 ceph-mon[103278]: pgmap v12101: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:26 smithi146 ceph-mon[103278]: pgmap v12102: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:26 smithi078 ceph-mon[91136]: pgmap v12102: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:26 smithi078 ceph-mon[99959]: pgmap v12102: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:28 smithi146 ceph-mon[103278]: pgmap v12103: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:28 smithi078 ceph-mon[91136]: pgmap v12103: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:28 smithi078 ceph-mon[99959]: pgmap v12103: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:29.614 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:30 smithi146 ceph-mon[103278]: pgmap v12104: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:30 smithi078 ceph-mon[99959]: pgmap v12104: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:30 smithi078 ceph-mon[91136]: pgmap v12104: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:03:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:03:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:03:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:31.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:31.865Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:31.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:32 smithi146 ceph-mon[103278]: pgmap v12105: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:32 smithi078 ceph-mon[91136]: pgmap v12105: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:32 smithi078 ceph-mon[99959]: pgmap v12105: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:33 smithi146 ceph-mon[103278]: pgmap v12106: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:33 smithi078 ceph-mon[91136]: pgmap v12106: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:33 smithi078 ceph-mon[99959]: pgmap v12106: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:36.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:36 smithi146 ceph-mon[103278]: pgmap v12107: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:36 smithi078 ceph-mon[91136]: pgmap v12107: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:36 smithi078 ceph-mon[99959]: pgmap v12107: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:38 smithi146 ceph-mon[103278]: pgmap v12108: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:38 smithi078 ceph-mon[91136]: pgmap v12108: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:38 smithi078 ceph-mon[99959]: pgmap v12108: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:40.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:40 smithi146 ceph-mon[103278]: pgmap v12109: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:40 smithi078 ceph-mon[91136]: pgmap v12109: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:40 smithi078 ceph-mon[99959]: pgmap v12109: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:03:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:03:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:03:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:42.180 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:41.865Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:42.180 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:41.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:42.180 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:41.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:42 smithi146 ceph-mon[103278]: pgmap v12110: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[91136]: pgmap v12110: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[99959]: pgmap v12110: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:03:43.355 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:03:43.355 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 6h - - 2024-09-21T03:03:43.355 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 6h - - 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 1s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 1s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 1s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 1s ago 6h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 1s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 1s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 1s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 1s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 1s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 1s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:03:43.356 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 16s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 16s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 16s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 6h 66.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 1s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 1s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:03:43.357 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:03:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:03:43.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:03:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:03:43.949 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:03:44.265 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:03:44.265 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:03:44.265 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:03:44.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[91136]: from='client.71511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[91136]: from='client.71517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[91136]: pgmap v12111: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:44.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[91136]: from='client.91439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:44.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2067760435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:44.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[99959]: from='client.71511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[99959]: from='client.71517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[99959]: pgmap v12111: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:44.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[99959]: from='client.91439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:44.267 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2067760435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:44 smithi146 ceph-mon[103278]: from='client.71511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:44 smithi146 ceph-mon[103278]: from='client.71517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:44 smithi146 ceph-mon[103278]: pgmap v12111: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:44 smithi146 ceph-mon[103278]: from='client.91439 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2067760435' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:03:45.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:45 smithi078 ceph-mon[99959]: from='client.71535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:45.283 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3733226595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:45 smithi078 ceph-mon[91136]: from='client.71535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:45.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3733226595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:45 smithi146 ceph-mon[103278]: from='client.71535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:03:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3733226595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:03:46.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:46 smithi078 ceph-mon[91136]: pgmap v12112: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:46.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:46 smithi078 ceph-mon[99959]: pgmap v12112: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:46 smithi146 ceph-mon[103278]: pgmap v12112: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:47 smithi146 ceph-mon[103278]: pgmap v12113: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[91136]: pgmap v12113: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:03:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:47 smithi078 ceph-mon[99959]: pgmap v12113: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:50 smithi146 ceph-mon[103278]: pgmap v12114: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:50 smithi078 ceph-mon[91136]: pgmap v12114: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:50 smithi078 ceph-mon[99959]: pgmap v12114: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:03:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:03:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:03:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:03:52.175 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:51.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:52.175 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:51.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:03:52.175 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:03:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:03:51.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:03:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:52 smithi146 ceph-mon[103278]: pgmap v12115: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:52 smithi078 ceph-mon[91136]: pgmap v12115: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:52 smithi078 ceph-mon[99959]: pgmap v12115: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:54 smithi146 ceph-mon[103278]: pgmap v12116: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:54 smithi078 ceph-mon[91136]: pgmap v12116: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:54 smithi078 ceph-mon[99959]: pgmap v12116: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:03:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:56 smithi146 ceph-mon[103278]: pgmap v12117: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:56.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:56 smithi078 ceph-mon[91136]: pgmap v12117: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:56.540 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:56 smithi078 ceph-mon[99959]: pgmap v12117: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:58 smithi146 ceph-mon[103278]: pgmap v12118: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:58 smithi078 ceph-mon[91136]: pgmap v12118: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:58 smithi078 ceph-mon[99959]: pgmap v12118: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:03:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:03:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:03:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:03:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:03:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:00 smithi146 ceph-mon[103278]: pgmap v12119: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:00 smithi078 ceph-mon[91136]: pgmap v12119: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:00 smithi078 ceph-mon[99959]: pgmap v12119: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:04:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:04:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:04:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:01.866Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:01.866Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:01.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:02 smithi146 ceph-mon[103278]: pgmap v12120: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[99959]: pgmap v12120: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[91136]: pgmap v12120: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:04.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:04 smithi078 ceph-mon[91136]: pgmap v12121: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:04.508 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:04 smithi078 ceph-mon[99959]: pgmap v12121: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:04.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:04 smithi146 ceph-mon[103278]: pgmap v12121: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:06 smithi078 ceph-mon[91136]: pgmap v12122: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:06.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:06 smithi078 ceph-mon[99959]: pgmap v12122: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:06 smithi146 ceph-mon[103278]: pgmap v12122: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:07 smithi146 ceph-mon[103278]: pgmap v12123: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[91136]: pgmap v12123: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:07 smithi078 ceph-mon[99959]: pgmap v12123: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:10.247 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:10 smithi146 ceph-mon[103278]: pgmap v12124: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:10 smithi078 ceph-mon[91136]: pgmap v12124: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:10 smithi078 ceph-mon[99959]: pgmap v12124: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:04:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:04:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:04:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:12.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:11.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:12.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:11.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:12.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:11.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:12 smithi146 ceph-mon[103278]: pgmap v12125: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:12 smithi078 ceph-mon[91136]: pgmap v12125: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:12 smithi078 ceph-mon[99959]: pgmap v12125: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:14 smithi146 ceph-mon[103278]: pgmap v12126: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:14 smithi078 ceph-mon[91136]: pgmap v12126: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:14 smithi078 ceph-mon[99959]: pgmap v12126: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:14.559 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:04:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:04:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 6h - - 2024-09-21T03:04:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 6h - - 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 13s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 8s ago 6h 91.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 13s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 8s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 13s ago 6h 584M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 13s ago 6h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 8s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 13s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 13s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 8s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 13s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:04:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 13s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 13s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 13s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 8s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 8s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 8s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 8s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 8s ago 6h 68.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 13s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 13s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:04:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 8s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:04:15.390 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:04:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:04:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:04:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:04:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:04:15.392 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:04:15.661 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:04:15.661 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:04:15.661 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:04:15.661 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:04:15.661 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:04:15.662 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:04:15.970 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:04:15.970 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:15.970 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:15.970 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:15.970 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[91136]: from='client.71547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[91136]: from='client.91469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[91136]: from='client.71559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[91136]: pgmap v12127: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2278158229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/215646569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[99959]: from='client.71547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[99959]: from='client.91469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[99959]: from='client.71559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[99959]: pgmap v12127: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2278158229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/215646569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:16 smithi146 ceph-mon[103278]: from='client.71547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:16 smithi146 ceph-mon[103278]: from='client.91469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:16 smithi146 ceph-mon[103278]: from='client.71559 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:16 smithi146 ceph-mon[103278]: pgmap v12127: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2278158229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/215646569' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:17 smithi146 ceph-mon[103278]: from='client.91487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:17 smithi078 ceph-mon[91136]: from='client.91487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:17.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:17 smithi078 ceph-mon[99959]: from='client.91487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:18 smithi146 ceph-mon[103278]: pgmap v12128: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:18 smithi078 ceph-mon[91136]: pgmap v12128: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:18 smithi078 ceph-mon[99959]: pgmap v12128: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:20 smithi146 ceph-mon[103278]: pgmap v12129: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:20 smithi078 ceph-mon[91136]: pgmap v12129: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:20 smithi078 ceph-mon[99959]: pgmap v12129: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:04:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:04:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:04:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:22.208 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:21.867Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:22.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:21.867Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:22.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:21.868Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:22 smithi146 ceph-mon[103278]: pgmap v12130: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[91136]: pgmap v12130: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[99959]: pgmap v12130: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:24 smithi146 ceph-mon[103278]: pgmap v12131: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:24 smithi078 ceph-mon[99959]: pgmap v12131: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:24 smithi078 ceph-mon[91136]: pgmap v12131: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:26.214 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 03:04:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T03:04:25.847836201Z level=info msg="Completed cleanup jobs" duration=12.154331ms 2024-09-21T03:04:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:26 smithi146 ceph-mon[103278]: pgmap v12132: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:26 smithi078 ceph-mon[91136]: pgmap v12132: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:26 smithi078 ceph-mon[99959]: pgmap v12132: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[91136]: pgmap v12133: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:27 smithi078 ceph-mon[99959]: pgmap v12133: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:27 smithi146 ceph-mon[103278]: pgmap v12133: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:29 smithi146 ceph-mon[103278]: pgmap v12134: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:30.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:29 smithi078 ceph-mon[91136]: pgmap v12134: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:30.120 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:29 smithi078 ceph-mon[99959]: pgmap v12134: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:30.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:30 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:30 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:30 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:04:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:04:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:04:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:31 smithi146 ceph-mon[103278]: pgmap v12135: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:31 smithi078 ceph-mon[91136]: pgmap v12135: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:31 smithi078 ceph-mon[99959]: pgmap v12135: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:31.868Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:31.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:31.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:34 smithi146 ceph-mon[103278]: pgmap v12136: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:34 smithi078 ceph-mon[91136]: pgmap v12136: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:34 smithi078 ceph-mon[99959]: pgmap v12136: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:36 smithi146 ceph-mon[103278]: pgmap v12137: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:36 smithi078 ceph-mon[99959]: pgmap v12137: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:36 smithi078 ceph-mon[91136]: pgmap v12137: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:38 smithi146 ceph-mon[103278]: pgmap v12138: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:38 smithi078 ceph-mon[91136]: pgmap v12138: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:38 smithi078 ceph-mon[99959]: pgmap v12138: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:40.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:40 smithi146 ceph-mon[103278]: pgmap v12139: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:40 smithi078 ceph-mon[99959]: pgmap v12139: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:40 smithi078 ceph-mon[91136]: pgmap v12139: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:04:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:04:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:04:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:42.201 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:41.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:42.201 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:41.869Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:42.201 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:41.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:42 smithi146 ceph-mon[103278]: pgmap v12140: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[91136]: pgmap v12140: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[99959]: pgmap v12140: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:42.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:44 smithi146 ceph-mon[103278]: pgmap v12141: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:44 smithi078 ceph-mon[91136]: pgmap v12141: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:44 smithi078 ceph-mon[99959]: pgmap v12141: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:46.280 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:04:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:46 smithi146 ceph-mon[103278]: pgmap v12142: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:46.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:46 smithi078 ceph-mon[99959]: pgmap v12142: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:46.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:46 smithi078 ceph-mon[91136]: pgmap v12142: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:46.822 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 6h - - 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 6h - - 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 4s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 20s ago 6h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 4s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 20s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 4s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 4s ago 6h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 20s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 4s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 4s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 20s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 4s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:04:46.823 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 4s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 4s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 4s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 20s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 20s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 20s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 20s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 20s ago 6h 68.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 4s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 20s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 4s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:04:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 20s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:04:47.145 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:04:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:47 smithi146 ceph-mon[103278]: from='client.91499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/973019958' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:47.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[91136]: from='client.91499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:47.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/973019958' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:47.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[99959]: from='client.91499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:47.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:04:47.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/973019958' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:04:47.430 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:04:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:04:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:04:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:04:47.431 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:04:47.748 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:04:47.748 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:04:47.749 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:04:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:48 smithi146 ceph-mon[103278]: from='client.91505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:48 smithi146 ceph-mon[103278]: from='client.91511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:48 smithi146 ceph-mon[103278]: pgmap v12143: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:48 smithi146 ceph-mon[103278]: from='client.91523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1204509037' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[91136]: from='client.91505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[91136]: from='client.91511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[91136]: pgmap v12143: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[91136]: from='client.91523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1204509037' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[99959]: from='client.91505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[99959]: from='client.91511 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[99959]: pgmap v12143: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[99959]: from='client.91523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:04:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1204509037' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:04:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:50 smithi146 ceph-mon[103278]: pgmap v12144: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:50 smithi078 ceph-mon[91136]: pgmap v12144: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:50 smithi078 ceph-mon[99959]: pgmap v12144: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:04:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:04:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:04:51.340 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:04:52.222 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:51.869Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:52.222 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:51.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:04:52.222 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:04:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:04:51.870Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:04:52.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:52 smithi146 ceph-mon[103278]: pgmap v12145: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:52 smithi078 ceph-mon[91136]: pgmap v12145: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:52 smithi078 ceph-mon[99959]: pgmap v12145: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:54 smithi146 ceph-mon[103278]: pgmap v12146: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:54 smithi078 ceph-mon[91136]: pgmap v12146: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:54 smithi078 ceph-mon[99959]: pgmap v12146: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:04:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:56 smithi146 ceph-mon[103278]: pgmap v12147: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:56 smithi078 ceph-mon[91136]: pgmap v12147: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:56 smithi078 ceph-mon[99959]: pgmap v12147: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:58 smithi146 ceph-mon[103278]: pgmap v12148: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:58 smithi078 ceph-mon[91136]: pgmap v12148: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:58 smithi078 ceph-mon[99959]: pgmap v12148: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:04:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:04:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:04:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:04:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:04:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:00 smithi146 ceph-mon[103278]: pgmap v12149: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:00 smithi078 ceph-mon[91136]: pgmap v12149: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:00 smithi078 ceph-mon[99959]: pgmap v12149: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:05:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:05:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:05:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:02.247 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:01.870Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:02.247 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:01.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:02.247 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:01.871Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:02 smithi146 ceph-mon[103278]: pgmap v12150: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[91136]: pgmap v12150: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[99959]: pgmap v12150: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:04 smithi146 ceph-mon[103278]: pgmap v12151: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:04 smithi078 ceph-mon[91136]: pgmap v12151: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:04 smithi078 ceph-mon[99959]: pgmap v12151: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:06 smithi146 ceph-mon[103278]: pgmap v12152: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:06 smithi078 ceph-mon[91136]: pgmap v12152: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:06 smithi078 ceph-mon[99959]: pgmap v12152: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:08.352 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[91136]: pgmap v12153: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.353 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:08 smithi078 ceph-mon[99959]: pgmap v12153: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:08.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:08.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:08 smithi146 ceph-mon[103278]: pgmap v12153: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:10 smithi146 ceph-mon[103278]: pgmap v12154: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:10 smithi078 ceph-mon[91136]: pgmap v12154: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:10 smithi078 ceph-mon[99959]: pgmap v12154: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:05:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:05:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:05:11.476 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:12.204 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:11.871Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:12.204 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:11.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:12.204 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:11.872Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:12.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:12 smithi146 ceph-mon[103278]: pgmap v12155: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:12 smithi078 ceph-mon[91136]: pgmap v12155: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:12 smithi078 ceph-mon[99959]: pgmap v12155: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:14 smithi146 ceph-mon[103278]: pgmap v12156: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:14 smithi078 ceph-mon[91136]: pgmap v12156: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:14 smithi078 ceph-mon[99959]: pgmap v12156: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:16 smithi146 ceph-mon[103278]: pgmap v12157: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:16 smithi078 ceph-mon[91136]: pgmap v12157: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:16 smithi078 ceph-mon[99959]: pgmap v12157: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.042 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:05:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:18 smithi146 ceph-mon[103278]: pgmap v12158: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:18 smithi078 ceph-mon[91136]: pgmap v12158: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:18 smithi078 ceph-mon[99959]: pgmap v12158: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 6h - - 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 6h - - 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 16s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 11s ago 6h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 16s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:05:18.571 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 11s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 16s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 16s ago 6h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 11s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 16s ago 6h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 16s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 11s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 16s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 16s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 16s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 16s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 11s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 11s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 11s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:05:18.572 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 11s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:05:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 11s ago 6h 69.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:05:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 16s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:05:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:05:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 16s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:05:18.573 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 11s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:05:18.879 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:05:18.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:18.880 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:05:18.880 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:05:18.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:05:18.880 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:05:19.146 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:05:19.147 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:05:19.147 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:05:19.147 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:05:19.147 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:05:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:05:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:19.455 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:19.455 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:19.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:19 smithi078 ceph-mon[99959]: from='client.71619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:19.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:19 smithi078 ceph-mon[99959]: from='client.91541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:19.456 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2020489518' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:19 smithi078 ceph-mon[91136]: from='client.71619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:19 smithi078 ceph-mon[91136]: from='client.91541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2020489518' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:19 smithi146 ceph-mon[103278]: from='client.71619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:19 smithi146 ceph-mon[103278]: from='client.91541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2020489518' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:20 smithi146 ceph-mon[103278]: from='client.91547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:20 smithi146 ceph-mon[103278]: from='client.71643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:20 smithi146 ceph-mon[103278]: pgmap v12159: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1326433621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[91136]: from='client.91547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[91136]: from='client.71643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[91136]: pgmap v12159: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1326433621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[99959]: from='client.91547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[99959]: from='client.71643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[99959]: pgmap v12159: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:20.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1326433621' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:05:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:05:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:05:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:22.200 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:21.872Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:22.200 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:21.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:22.200 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:21.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:22 smithi146 ceph-mon[103278]: pgmap v12160: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[99959]: pgmap v12160: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[91136]: pgmap v12160: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:22.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:24 smithi146 ceph-mon[103278]: pgmap v12161: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:24 smithi078 ceph-mon[91136]: pgmap v12161: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:24 smithi078 ceph-mon[99959]: pgmap v12161: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:26 smithi146 ceph-mon[103278]: pgmap v12162: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:26 smithi078 ceph-mon[91136]: pgmap v12162: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:26 smithi078 ceph-mon[99959]: pgmap v12162: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[91136]: pgmap v12163: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[99959]: pgmap v12163: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.515 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:28 smithi146 ceph-mon[103278]: pgmap v12163: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:29.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:30 smithi146 ceph-mon[103278]: pgmap v12164: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:30 smithi078 ceph-mon[91136]: pgmap v12164: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:30 smithi078 ceph-mon[99959]: pgmap v12164: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:05:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:05:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:05:31.506 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:32.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:31.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:32.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:31.873Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:32.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:31.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:32.463 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:32 smithi146 ceph-mon[103278]: pgmap v12165: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:32.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:32 smithi078 ceph-mon[91136]: pgmap v12165: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:32.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:32 smithi078 ceph-mon[99959]: pgmap v12165: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:34 smithi146 ceph-mon[103278]: pgmap v12166: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:34 smithi078 ceph-mon[91136]: pgmap v12166: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:34.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:34 smithi078 ceph-mon[99959]: pgmap v12166: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:36 smithi146 ceph-mon[103278]: pgmap v12167: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:36 smithi078 ceph-mon[91136]: pgmap v12167: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:36 smithi078 ceph-mon[99959]: pgmap v12167: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:38 smithi146 ceph-mon[103278]: pgmap v12168: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:38 smithi078 ceph-mon[91136]: pgmap v12168: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:38 smithi078 ceph-mon[99959]: pgmap v12168: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:40.502 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:40 smithi146 ceph-mon[103278]: pgmap v12169: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:40 smithi078 ceph-mon[99959]: pgmap v12169: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:40 smithi078 ceph-mon[91136]: pgmap v12169: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:05:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:05:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:05:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:42.244 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:41.873Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:42.244 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:41.874Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:42.244 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:41.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:42 smithi146 ceph-mon[103278]: pgmap v12170: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:42 smithi078 ceph-mon[91136]: pgmap v12170: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:42 smithi078 ceph-mon[99959]: pgmap v12170: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:44 smithi146 ceph-mon[103278]: pgmap v12171: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:44 smithi078 ceph-mon[91136]: pgmap v12171: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:44 smithi078 ceph-mon[99959]: pgmap v12171: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:46 smithi146 ceph-mon[103278]: pgmap v12172: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:46 smithi078 ceph-mon[91136]: pgmap v12172: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:46 smithi078 ceph-mon[99959]: pgmap v12172: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:48.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[99959]: pgmap v12173: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[91136]: pgmap v12173: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:48 smithi146 ceph-mon[103278]: pgmap v12173: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:05:49.772 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:05:50.332 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:05:50.332 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 6h - - 2024-09-21T03:05:50.332 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 6h - - 2024-09-21T03:05:50.332 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 8s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:05:50.332 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 2s ago 6h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 8s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 8s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 8s ago 6h 554M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 6h 373M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 8s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 8s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 2s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 8s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 8s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 8s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:05:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:05:50.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:05:50.334 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 6h 69.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:05:50.334 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 8s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:05:50.334 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:05:50.334 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 8s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:05:50.334 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:05:50.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:50 smithi078 ceph-mon[99959]: pgmap v12174: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:50.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:50 smithi078 ceph-mon[91136]: pgmap v12174: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:50.665 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:05:50.665 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:05:50.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:05:50.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:05:50.667 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:05:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:50 smithi146 ceph-mon[103278]: pgmap v12174: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:50.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:05:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:05:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:05:51.022 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:05:51.023 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:05:51.370 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:05:51.370 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:05:51.370 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[99959]: from='client.91571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[99959]: from='client.91577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[99959]: from='client.71667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1162609175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[91136]: from='client.91571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[91136]: from='client.91577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[91136]: from='client.71667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1162609175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:51.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:51 smithi146 ceph-mon[103278]: from='client.91571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:51 smithi146 ceph-mon[103278]: from='client.91577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:05:51.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:51 smithi146 ceph-mon[103278]: from='client.71667 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:51.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1162609175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:05:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:51.874Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:51.875Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:05:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:05:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:05:51.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:05:52.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:52 smithi146 ceph-mon[103278]: from='client.91595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:52.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:52 smithi146 ceph-mon[103278]: pgmap v12175: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:52.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/553544643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:52 smithi078 ceph-mon[99959]: from='client.91595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:52 smithi078 ceph-mon[99959]: pgmap v12175: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/553544643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:52 smithi078 ceph-mon[91136]: from='client.91595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:05:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:52 smithi078 ceph-mon[91136]: pgmap v12175: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:05:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/553544643' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:05:54.600 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:54 smithi146 ceph-mon[103278]: pgmap v12176: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:54 smithi078 ceph-mon[91136]: pgmap v12176: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:54 smithi078 ceph-mon[99959]: pgmap v12176: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:05:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:56 smithi146 ceph-mon[103278]: pgmap v12177: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:56 smithi078 ceph-mon[91136]: pgmap v12177: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:56 smithi078 ceph-mon[99959]: pgmap v12177: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:58 smithi146 ceph-mon[103278]: pgmap v12178: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:58 smithi078 ceph-mon[91136]: pgmap v12178: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:58 smithi078 ceph-mon[99959]: pgmap v12178: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:05:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:05:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:05:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:05:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:05:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:00 smithi078 ceph-mon[91136]: pgmap v12179: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:00 smithi078 ceph-mon[99959]: pgmap v12179: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:00 smithi146 ceph-mon[103278]: pgmap v12179: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:06:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:06:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:06:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:01.875Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:01.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:01.876Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:02 smithi146 ceph-mon[103278]: pgmap v12180: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:06:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:02 smithi078 ceph-mon[99959]: pgmap v12180: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:06:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:02 smithi078 ceph-mon[91136]: pgmap v12180: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:06:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:03.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:04 smithi146 ceph-mon[103278]: pgmap v12181: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:04 smithi078 ceph-mon[99959]: pgmap v12181: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:04 smithi078 ceph-mon[91136]: pgmap v12181: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:06 smithi146 ceph-mon[103278]: pgmap v12182: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:06 smithi078 ceph-mon[91136]: pgmap v12182: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:06 smithi078 ceph-mon[99959]: pgmap v12182: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:08 smithi146 ceph-mon[103278]: pgmap v12183: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[99959]: pgmap v12183: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[91136]: pgmap v12183: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:10.730 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:10 smithi078 ceph-mon[91136]: pgmap v12184: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:10.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:10 smithi078 ceph-mon[99959]: pgmap v12184: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:10.731 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:06:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:06:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:06:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:10 smithi146 ceph-mon[103278]: pgmap v12184: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:12.096 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:11.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:12.096 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:11.877Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:12.096 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:11.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:12 smithi146 ceph-mon[103278]: pgmap v12185: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:12 smithi078 ceph-mon[99959]: pgmap v12185: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:12 smithi078 ceph-mon[91136]: pgmap v12185: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:13.746 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:13 smithi146 ceph-mon[103278]: pgmap v12186: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:13.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:13 smithi078 ceph-mon[91136]: pgmap v12186: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:13.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:13 smithi078 ceph-mon[99959]: pgmap v12186: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:14.624 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:15 smithi146 ceph-mon[103278]: pgmap v12187: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:15.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:15 smithi078 ceph-mon[91136]: pgmap v12187: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:15 smithi078 ceph-mon[99959]: pgmap v12187: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:18 smithi146 ceph-mon[103278]: pgmap v12188: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:18 smithi078 ceph-mon[99959]: pgmap v12188: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:18 smithi078 ceph-mon[91136]: pgmap v12188: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:20 smithi146 ceph-mon[103278]: pgmap v12189: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:20 smithi078 ceph-mon[91136]: pgmap v12189: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:20 smithi078 ceph-mon[99959]: pgmap v12189: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:06:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:06:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:06:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:21.664 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:06:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:21.877Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:21.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:21.878Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 6h - - 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 6h - - 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 19s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 14s ago 6h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 19s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:06:22.201 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 14s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 19s ago 6h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 14s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 19s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 19s ago 6h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 14s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 19s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 19s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 19s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 19s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 14s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 14s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 14s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:06:22.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 14s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:06:22.203 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 14s ago 6h 69.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:06:22.203 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 19s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:06:22.203 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:06:22.203 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 19s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:06:22.203 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 14s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:06:22.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:22 smithi078 ceph-mon[91136]: pgmap v12190: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:22.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:22 smithi078 ceph-mon[99959]: pgmap v12190: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:22 smithi146 ceph-mon[103278]: pgmap v12190: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:22.530 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:22.531 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:06:22.532 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:06:22.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:06:22.532 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:06:22.825 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:06:22.825 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:06:22.826 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:06:23.139 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:06:23.139 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:23.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:23.139 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:23.140 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='client.71691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='client.71697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='client.71703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/324095052' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2097941590' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='client.71691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='client.71697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='client.71703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/324095052' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2097941590' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='client.71691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='client.71697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='client.71703 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/324095052' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2097941590' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:24 smithi146 ceph-mon[103278]: from='client.71715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:24.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:24 smithi146 ceph-mon[103278]: pgmap v12191: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:24 smithi078 ceph-mon[91136]: from='client.71715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:24 smithi078 ceph-mon[91136]: pgmap v12191: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:24 smithi078 ceph-mon[99959]: from='client.71715 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:24 smithi078 ceph-mon[99959]: pgmap v12191: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:26 smithi146 ceph-mon[103278]: pgmap v12192: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:26.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:26 smithi078 ceph-mon[91136]: pgmap v12192: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:26.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:26 smithi078 ceph-mon[99959]: pgmap v12192: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:28 smithi146 ceph-mon[103278]: pgmap v12193: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[91136]: pgmap v12193: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[99959]: pgmap v12193: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:30.469 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:30 smithi078 ceph-mon[99959]: pgmap v12194: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:30.469 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:30 smithi078 ceph-mon[91136]: pgmap v12194: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:30 smithi146 ceph-mon[103278]: pgmap v12194: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:06:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:06:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:06:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:31.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:31.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:32.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:31.878Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:32.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:31.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:32.103 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:31.879Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:32 smithi146 ceph-mon[103278]: pgmap v12195: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:32.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:32 smithi078 ceph-mon[91136]: pgmap v12195: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:32.524 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:32 smithi078 ceph-mon[99959]: pgmap v12195: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:34 smithi146 ceph-mon[103278]: pgmap v12196: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:34 smithi078 ceph-mon[91136]: pgmap v12196: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:34 smithi078 ceph-mon[99959]: pgmap v12196: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:36 smithi146 ceph-mon[103278]: pgmap v12197: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:36 smithi078 ceph-mon[91136]: pgmap v12197: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:36 smithi078 ceph-mon[99959]: pgmap v12197: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:38 smithi078 ceph-mon[91136]: pgmap v12198: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:38 smithi078 ceph-mon[99959]: pgmap v12198: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:38 smithi146 ceph-mon[103278]: pgmap v12198: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:40.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:40 smithi078 ceph-mon[91136]: pgmap v12199: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:40.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:40 smithi078 ceph-mon[99959]: pgmap v12199: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:40 smithi146 ceph-mon[103278]: pgmap v12199: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:06:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:06:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:06:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:42.177 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:41.879Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:42.177 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:41.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:42.177 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:41.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:42 smithi078 ceph-mon[91136]: pgmap v12200: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:42 smithi078 ceph-mon[99959]: pgmap v12200: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:42 smithi146 ceph-mon[103278]: pgmap v12200: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:43 smithi146 ceph-mon[103278]: pgmap v12201: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[91136]: pgmap v12201: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[99959]: pgmap v12201: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:46 smithi146 ceph-mon[103278]: pgmap v12202: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:46 smithi078 ceph-mon[91136]: pgmap v12202: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:46 smithi078 ceph-mon[99959]: pgmap v12202: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:48 smithi146 ceph-mon[103278]: pgmap v12203: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[91136]: pgmap v12203: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[99959]: pgmap v12203: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:06:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:50 smithi146 ceph-mon[103278]: pgmap v12204: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:50 smithi078 ceph-mon[91136]: pgmap v12204: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:50 smithi078 ceph-mon[99959]: pgmap v12204: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:50.907 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:06:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:06:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:06:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:06:52.164 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:51.880Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:52.164 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:51.880Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:06:52.164 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:06:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:06:51.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:06:52.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:52 smithi078 ceph-mon[99959]: pgmap v12205: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:52.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:52 smithi078 ceph-mon[91136]: pgmap v12205: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:52 smithi146 ceph-mon[103278]: pgmap v12205: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:53.434 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:06:53.970 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:06:53.970 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 6h - - 2024-09-21T03:06:53.970 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 6h - - 2024-09-21T03:06:53.970 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 11s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:06:53.970 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 11s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 6s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 11s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 11s ago 6h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 6s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 11s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 11s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 11s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 11s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 11s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 6s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 6s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 6s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:06:53.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 6s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:06:53.972 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 6s ago 6h 69.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:06:53.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 11s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:06:53.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:06:53.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 11s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:06:53.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:06:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:06:54.298 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:06:54.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:54 smithi146 ceph-mon[103278]: pgmap v12206: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:54.385 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:54 smithi146 ceph-mon[103278]: from='client.91643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:54 smithi078 ceph-mon[99959]: pgmap v12206: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:54 smithi078 ceph-mon[99959]: from='client.91643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:54 smithi078 ceph-mon[91136]: pgmap v12206: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:06:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:54 smithi078 ceph-mon[91136]: from='client.91643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:06:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:06:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:06:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:06:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:06:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:06:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:06:54.594 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:06:54.914 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:06:54.914 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:06:54.915 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:06:55.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:55 smithi146 ceph-mon[103278]: from='client.91649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:55.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:55 smithi146 ceph-mon[103278]: from='client.91655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:55.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2666773042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:55.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/530158699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[91136]: from='client.91649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[91136]: from='client.91655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2666773042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/530158699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[99959]: from='client.91649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[99959]: from='client.91655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2666773042' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:06:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/530158699' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:06:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:56 smithi146 ceph-mon[103278]: from='client.71751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:56 smithi146 ceph-mon[103278]: pgmap v12207: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:56 smithi078 ceph-mon[91136]: from='client.71751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:56 smithi078 ceph-mon[91136]: pgmap v12207: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:56 smithi078 ceph-mon[99959]: from='client.71751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:06:56.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:56 smithi078 ceph-mon[99959]: pgmap v12207: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:58 smithi146 ceph-mon[103278]: pgmap v12208: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:58 smithi078 ceph-mon[91136]: pgmap v12208: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:58 smithi078 ceph-mon[99959]: pgmap v12208: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:06:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:06:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:06:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:06:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:06:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:00 smithi078 ceph-mon[99959]: pgmap v12209: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:00 smithi078 ceph-mon[91136]: pgmap v12209: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:00 smithi146 ceph-mon[103278]: pgmap v12209: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:07:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:07:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:07:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:02.274 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:01.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:02.275 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:01.881Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:02.275 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:01.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:02 smithi078 ceph-mon[91136]: pgmap v12210: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:02 smithi078 ceph-mon[99959]: pgmap v12210: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:02 smithi146 ceph-mon[103278]: pgmap v12210: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:03 smithi146 ceph-mon[103278]: pgmap v12211: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[91136]: pgmap v12211: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:03 smithi078 ceph-mon[99959]: pgmap v12211: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:06 smithi146 ceph-mon[103278]: pgmap v12212: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:06 smithi078 ceph-mon[91136]: pgmap v12212: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:06 smithi078 ceph-mon[99959]: pgmap v12212: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:08 smithi146 ceph-mon[103278]: pgmap v12213: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[99959]: pgmap v12213: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[91136]: pgmap v12213: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:10 smithi146 ceph-mon[103278]: pgmap v12214: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:10 smithi078 ceph-mon[91136]: pgmap v12214: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:10 smithi078 ceph-mon[99959]: pgmap v12214: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:07:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:07:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:07:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:12.250 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:11.881Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:12.250 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:11.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:12.250 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:11.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:12 smithi146 ceph-mon[103278]: pgmap v12215: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:12 smithi078 ceph-mon[91136]: pgmap v12215: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:12 smithi078 ceph-mon[99959]: pgmap v12215: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:14 smithi146 ceph-mon[103278]: pgmap v12216: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:14.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:14 smithi078 ceph-mon[99959]: pgmap v12216: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:14.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:14.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:14 smithi078 ceph-mon[91136]: pgmap v12216: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:14.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:16 smithi146 ceph-mon[103278]: pgmap v12217: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:16 smithi078 ceph-mon[91136]: pgmap v12217: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:16 smithi078 ceph-mon[99959]: pgmap v12217: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:18 smithi146 ceph-mon[103278]: pgmap v12218: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:18 smithi078 ceph-mon[91136]: pgmap v12218: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:18 smithi078 ceph-mon[99959]: pgmap v12218: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:20 smithi146 ceph-mon[103278]: pgmap v12219: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:20 smithi078 ceph-mon[91136]: pgmap v12219: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:20 smithi078 ceph-mon[99959]: pgmap v12219: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:07:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:07:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:07:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:21.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:22.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:21.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:22.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:21.882Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:22.267 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:21.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:22 smithi078 ceph-mon[91136]: pgmap v12220: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:22.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:22 smithi078 ceph-mon[99959]: pgmap v12220: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:22 smithi146 ceph-mon[103278]: pgmap v12220: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:23 smithi146 ceph-mon[103278]: pgmap v12221: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[99959]: pgmap v12221: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:23 smithi078 ceph-mon[91136]: pgmap v12221: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:25.211 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:07:25.741 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:07:25.741 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 6h - - 2024-09-21T03:07:25.741 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 6h - - 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 2s ago 6h 29.8M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 17s ago 6h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 17s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 2s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 2s ago 6h 555M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 17s ago 6h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 2s ago 6h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 2s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 17s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 2s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:07:25.742 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 2s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 2s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 2s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 17s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 17s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 17s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 17s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 17s ago 6h 70.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 2s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 2s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:07:25.743 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 17s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:07:26.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:26.057 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:07:26.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:07:26.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:07:26.057 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:07:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:07:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:07:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:07:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:07:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:07:26.326 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:07:26.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[91136]: pgmap v12222: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:26.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[91136]: from='client.91679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:26.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[91136]: from='client.91685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:26.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1409232905' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:26.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[99959]: pgmap v12222: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:26.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[99959]: from='client.91679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:26.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[99959]: from='client.91685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:26.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1409232905' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:26 smithi146 ceph-mon[103278]: pgmap v12222: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:26 smithi146 ceph-mon[103278]: from='client.91679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:26 smithi146 ceph-mon[103278]: from='client.91685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1409232905' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:26.636 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:07:26.636 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:26.636 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:27 smithi146 ceph-mon[103278]: from='client.71775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:27 smithi146 ceph-mon[103278]: from='client.71787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2887310664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:27 smithi078 ceph-mon[99959]: from='client.71775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:27 smithi078 ceph-mon[99959]: from='client.71787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:27.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2887310664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:27 smithi078 ceph-mon[91136]: from='client.71775 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:27 smithi078 ceph-mon[91136]: from='client.71787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2887310664' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:28 smithi146 ceph-mon[103278]: pgmap v12223: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[91136]: pgmap v12223: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[99959]: pgmap v12223: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:30 smithi146 ceph-mon[103278]: pgmap v12224: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:30.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:30 smithi078 ceph-mon[99959]: pgmap v12224: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:30.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:30 smithi078 ceph-mon[91136]: pgmap v12224: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:07:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:07:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:07:31.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:32.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:31.882Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:32.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:31.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:32.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:31.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:32 smithi078 ceph-mon[91136]: pgmap v12225: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:32.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:32 smithi078 ceph-mon[99959]: pgmap v12225: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:32 smithi146 ceph-mon[103278]: pgmap v12225: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:34 smithi078 ceph-mon[91136]: pgmap v12226: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:34 smithi078 ceph-mon[99959]: pgmap v12226: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:34.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:34 smithi146 ceph-mon[103278]: pgmap v12226: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:36.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:36 smithi078 ceph-mon[99959]: pgmap v12227: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:36 smithi078 ceph-mon[91136]: pgmap v12227: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:36 smithi146 ceph-mon[103278]: pgmap v12227: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:38 smithi078 ceph-mon[91136]: pgmap v12228: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:38 smithi078 ceph-mon[99959]: pgmap v12228: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:38.663 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:38 smithi146 ceph-mon[103278]: pgmap v12228: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:40 smithi078 ceph-mon[91136]: pgmap v12229: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:40 smithi078 ceph-mon[99959]: pgmap v12229: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:40 smithi146 ceph-mon[103278]: pgmap v12229: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:07:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:07:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:07:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:41.883Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:41.883Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:41.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:42 smithi146 ceph-mon[103278]: pgmap v12230: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:07:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:42 smithi078 ceph-mon[91136]: pgmap v12230: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:07:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:42 smithi078 ceph-mon[99959]: pgmap v12230: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:07:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:44 smithi146 ceph-mon[103278]: pgmap v12231: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[99959]: pgmap v12231: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[91136]: pgmap v12231: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:46 smithi146 ceph-mon[103278]: pgmap v12232: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:46.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:46 smithi078 ceph-mon[99959]: pgmap v12232: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:46 smithi078 ceph-mon[91136]: pgmap v12232: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:48 smithi146 ceph-mon[103278]: pgmap v12233: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[99959]: pgmap v12233: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[91136]: pgmap v12233: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:07:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:50 smithi146 ceph-mon[103278]: pgmap v12234: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:50 smithi078 ceph-mon[99959]: pgmap v12234: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:50 smithi078 ceph-mon[91136]: pgmap v12234: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:07:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:07:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:07:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:07:52.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:51.884Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:52.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:51.884Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:07:52.269 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:07:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:07:51.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:07:52.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:52 smithi078 ceph-mon[99959]: pgmap v12235: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:52 smithi078 ceph-mon[91136]: pgmap v12235: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:52 smithi146 ceph-mon[103278]: pgmap v12235: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:54 smithi078 ceph-mon[99959]: pgmap v12236: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:54 smithi078 ceph-mon[91136]: pgmap v12236: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:54 smithi146 ceph-mon[103278]: pgmap v12236: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:07:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:56 smithi146 ceph-mon[103278]: pgmap v12237: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:56 smithi078 ceph-mon[91136]: pgmap v12237: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:56 smithi078 ceph-mon[99959]: pgmap v12237: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:56.937 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:07:57.460 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 6h - - 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 6h - - 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 14s ago 6h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 9s ago 6h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 14s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 9s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 14s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 14s ago 6h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 9s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 14s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 14s ago 6h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 9s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:07:57.461 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 14s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 14s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 14s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 14s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 9s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 9s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 9s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 9s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 9s ago 6h 70.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 14s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 14s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:07:57.462 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:57.768 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:07:57.769 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:07:58.040 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:07:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:07:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:58.350 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:07:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:58.350 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:07:58.350 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[91136]: from='client.71799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[91136]: from='client.91721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[91136]: pgmap v12238: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[91136]: from='client.91727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/852121540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[99959]: from='client.71799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[99959]: from='client.91721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[99959]: pgmap v12238: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[99959]: from='client.91727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/852121540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:58 smithi146 ceph-mon[103278]: from='client.71799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:58 smithi146 ceph-mon[103278]: from='client.91721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:58 smithi146 ceph-mon[103278]: pgmap v12238: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:07:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:58 smithi146 ceph-mon[103278]: from='client.91727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/852121540' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:07:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:59 smithi078 ceph-mon[91136]: from='client.71823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2153586059' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:07:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:59 smithi078 ceph-mon[99959]: from='client.71823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2153586059' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:07:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:07:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:59 smithi146 ceph-mon[103278]: from='client.71823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:07:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2153586059' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:07:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:07:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:00 smithi078 ceph-mon[91136]: pgmap v12239: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:00 smithi078 ceph-mon[99959]: pgmap v12239: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:00 smithi146 ceph-mon[103278]: pgmap v12239: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:08:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:08:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:08:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:01.885Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:01.885Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:01.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:02 smithi146 ceph-mon[103278]: pgmap v12240: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:02 smithi078 ceph-mon[91136]: pgmap v12240: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:02 smithi078 ceph-mon[99959]: pgmap v12240: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:04 smithi146 ceph-mon[103278]: pgmap v12241: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[99959]: pgmap v12241: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[91136]: pgmap v12241: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:04.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:06.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:06 smithi146 ceph-mon[103278]: pgmap v12242: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:06.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:06 smithi078 ceph-mon[91136]: pgmap v12242: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:06.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:06 smithi078 ceph-mon[99959]: pgmap v12242: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:08.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:08 smithi146 ceph-mon[103278]: pgmap v12243: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:08 smithi078 ceph-mon[91136]: pgmap v12243: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:08.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:08 smithi078 ceph-mon[99959]: pgmap v12243: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:10 smithi078 ceph-mon[91136]: pgmap v12244: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:10.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:10 smithi078 ceph-mon[99959]: pgmap v12244: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:10 smithi146 ceph-mon[103278]: pgmap v12244: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:08:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:08:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:08:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:12.276 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:11.886Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:12.276 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:11.886Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:12.276 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:11.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:12 smithi078 ceph-mon[91136]: pgmap v12245: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:12.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:12 smithi078 ceph-mon[99959]: pgmap v12245: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:12 smithi146 ceph-mon[103278]: pgmap v12245: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:14 smithi078 ceph-mon[99959]: pgmap v12246: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:14 smithi078 ceph-mon[91136]: pgmap v12246: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:14 smithi146 ceph-mon[103278]: pgmap v12246: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:16 smithi146 ceph-mon[103278]: pgmap v12247: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:16 smithi078 ceph-mon[91136]: pgmap v12247: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:16 smithi078 ceph-mon[99959]: pgmap v12247: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:18.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:18 smithi146 ceph-mon[103278]: pgmap v12248: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:18 smithi078 ceph-mon[91136]: pgmap v12248: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:18.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:18 smithi078 ceph-mon[99959]: pgmap v12248: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:20.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:20 smithi078 ceph-mon[91136]: pgmap v12249: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:20 smithi078 ceph-mon[99959]: pgmap v12249: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:20 smithi146 ceph-mon[103278]: pgmap v12249: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:08:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:08:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:08:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:21.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:21.887Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:21.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:22.737 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:22 smithi078 ceph-mon[99959]: pgmap v12250: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:22.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:22 smithi078 ceph-mon[91136]: pgmap v12250: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:22 smithi146 ceph-mon[103278]: pgmap v12250: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:24 smithi146 ceph-mon[103278]: pgmap v12251: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[91136]: pgmap v12251: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[99959]: pgmap v12251: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:26 smithi146 ceph-mon[103278]: pgmap v12252: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:26 smithi078 ceph-mon[91136]: pgmap v12252: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:26 smithi078 ceph-mon[99959]: pgmap v12252: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:28.648 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:28 smithi078 ceph-mon[99959]: pgmap v12253: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:28.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:28 smithi078 ceph-mon[91136]: pgmap v12253: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:28.648 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:08:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:28 smithi146 ceph-mon[103278]: pgmap v12253: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:29.173 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:08:29.173 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 6h - - 2024-09-21T03:08:29.173 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 6h - - 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 5s ago 6h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 0s ago 6h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 5s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 0s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 5s ago 6h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 5s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 0s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 5s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 5s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 0s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 5s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 5s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 5s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:08:29.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 5s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 0s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 0s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 0s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 0s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 0s ago 6h 70.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 5s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 5s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:08:29.175 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 0s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:08:29.484 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:08:29.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:08:29.486 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:08:29.753 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:08:29.754 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:08:29.754 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:08:29.754 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:08:29.754 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:08:29.754 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:08:29.754 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:08:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:30.061 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:08:30.061 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:08:30.062 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:08:30.062 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:08:30.062 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:08:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:30 smithi146 ceph-mon[103278]: from='client.71835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:30 smithi146 ceph-mon[103278]: from='client.71841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:30 smithi146 ceph-mon[103278]: from='client.71847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:30 smithi146 ceph-mon[103278]: pgmap v12254: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1532874494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3450627854' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:08:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[99959]: from='client.71835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[99959]: from='client.71841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[99959]: from='client.71847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[99959]: pgmap v12254: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1532874494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3450627854' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[91136]: from='client.71835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[91136]: from='client.71841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[91136]: from='client.71847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[91136]: pgmap v12254: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1532874494' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:08:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3450627854' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:08:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:31 smithi146 ceph-mon[103278]: from='client.71859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-mon[99959]: from='client.71859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-mon[91136]: from='client.71859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:08:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:31.887Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:31.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:31.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:32 smithi146 ceph-mon[103278]: pgmap v12255: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:32 smithi078 ceph-mon[91136]: pgmap v12255: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:32 smithi078 ceph-mon[99959]: pgmap v12255: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:34.759 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:34 smithi078 ceph-mon[91136]: pgmap v12256: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:34.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:34 smithi078 ceph-mon[99959]: pgmap v12256: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:34 smithi146 ceph-mon[103278]: pgmap v12256: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:36 smithi078 ceph-mon[91136]: pgmap v12257: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:36 smithi078 ceph-mon[99959]: pgmap v12257: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:36.678 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:36 smithi146 ceph-mon[103278]: pgmap v12257: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:38.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:38 smithi078 ceph-mon[91136]: pgmap v12258: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:38.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:38 smithi078 ceph-mon[99959]: pgmap v12258: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:38.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:38 smithi146 ceph-mon[103278]: pgmap v12258: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:40 smithi078 ceph-mon[99959]: pgmap v12259: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:40 smithi078 ceph-mon[91136]: pgmap v12259: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:40 smithi146 ceph-mon[103278]: pgmap v12259: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:08:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:08:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:08:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:41.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:41.888Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:41.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:42 smithi146 ceph-mon[103278]: pgmap v12260: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:42 smithi078 ceph-mon[99959]: pgmap v12260: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:42 smithi078 ceph-mon[91136]: pgmap v12260: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:44 smithi146 ceph-mon[103278]: pgmap v12261: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[99959]: pgmap v12261: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[91136]: pgmap v12261: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:46 smithi146 ceph-mon[103278]: pgmap v12262: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:46 smithi078 ceph-mon[91136]: pgmap v12262: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:46 smithi078 ceph-mon[99959]: pgmap v12262: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:48 smithi146 ceph-mon[103278]: pgmap v12263: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:48 smithi078 ceph-mon[91136]: pgmap v12263: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:48 smithi078 ceph-mon[99959]: pgmap v12263: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[91136]: pgmap v12264: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:49.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:49.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:49.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:49.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:49 smithi078 ceph-mon[99959]: pgmap v12264: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:08:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:49 smithi146 ceph-mon[103278]: pgmap v12264: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:08:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:08:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:08:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:08:51.889 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:51 smithi078 ceph-mon[99959]: pgmap v12265: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:51.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:51 smithi078 ceph-mon[91136]: pgmap v12265: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:51 smithi146 ceph-mon[103278]: pgmap v12265: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:51.888Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:51.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:08:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:08:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:08:51.889Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:08:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:54 smithi078 ceph-mon[91136]: pgmap v12266: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:54 smithi078 ceph-mon[99959]: pgmap v12266: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:54 smithi146 ceph-mon[103278]: pgmap v12266: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:08:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:56 smithi078 ceph-mon[91136]: pgmap v12267: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:56 smithi078 ceph-mon[99959]: pgmap v12267: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:56 smithi146 ceph-mon[103278]: pgmap v12267: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:58.524 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:58 smithi146 ceph-mon[103278]: pgmap v12268: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:58 smithi078 ceph-mon[91136]: pgmap v12268: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:58 smithi078 ceph-mon[99959]: pgmap v12268: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:08:59.514 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:08:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:08:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:08:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:08:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:00.354 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:09:00.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:00 smithi078 ceph-mon[99959]: pgmap v12269: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:00 smithi078 ceph-mon[91136]: pgmap v12269: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:00 smithi146 ceph-mon[103278]: pgmap v12269: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:00.883 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 6h - - 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 6h - - 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 17s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 12s ago 6h 91.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 17s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 12s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 17s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 17s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 12s ago 6h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 17s ago 6h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 17s ago 6h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 12s ago 6h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 17s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:09:00.884 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 17s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 17s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 17s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 12s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 12s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 12s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 12s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 12s ago 6h 70.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 17s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 17s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:09:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 12s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:09:00.885 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:09:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:09:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:09:01.194 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:09:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:09:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:09:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:09:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:09:01.195 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:09:01.465 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:09:01.466 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:09:01.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:01.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-mon[91136]: from='client.71871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:01.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/577660129' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:01.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:01.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-mon[99959]: from='client.71871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:01.467 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/577660129' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:01 smithi146 ceph-mon[103278]: from='client.71871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/577660129' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:01.774 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:09:01.774 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:09:01.774 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:09:01.775 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:09:01.775 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:09:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:01.889Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:01.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:01.890Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:02 smithi146 ceph-mon[103278]: from='client.91793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:02 smithi146 ceph-mon[103278]: from='client.71883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:02 smithi146 ceph-mon[103278]: pgmap v12270: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:02 smithi146 ceph-mon[103278]: from='client.71895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/542619213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[91136]: from='client.91793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[91136]: from='client.71883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[91136]: pgmap v12270: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[91136]: from='client.71895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/542619213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[99959]: from='client.91793 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[99959]: from='client.71883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[99959]: pgmap v12270: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[99959]: from='client.71895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/542619213' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:04 smithi146 ceph-mon[103278]: pgmap v12271: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[91136]: pgmap v12271: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[99959]: pgmap v12271: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:06 smithi146 ceph-mon[103278]: pgmap v12272: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:06 smithi078 ceph-mon[91136]: pgmap v12272: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:06 smithi078 ceph-mon[99959]: pgmap v12272: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:08.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:08 smithi146 ceph-mon[103278]: pgmap v12273: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:08 smithi078 ceph-mon[91136]: pgmap v12273: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:08 smithi078 ceph-mon[99959]: pgmap v12273: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:10.013 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.014 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:09 smithi146 ceph-mon[103278]: pgmap v12274: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[91136]: pgmap v12274: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:09 smithi078 ceph-mon[99959]: pgmap v12274: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:10 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:10 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:10 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:09:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:09:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:09:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:11 smithi146 ceph-mon[103278]: pgmap v12275: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:11 smithi078 ceph-mon[91136]: pgmap v12275: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:12.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:11 smithi078 ceph-mon[99959]: pgmap v12275: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:11.890Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:11.891Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:11.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:14 smithi078 ceph-mon[91136]: pgmap v12276: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:14 smithi078 ceph-mon[99959]: pgmap v12276: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:14 smithi146 ceph-mon[103278]: pgmap v12276: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:16.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:16 smithi078 ceph-mon[99959]: pgmap v12277: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:16.495 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:16 smithi078 ceph-mon[91136]: pgmap v12277: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:16 smithi146 ceph-mon[103278]: pgmap v12277: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:18.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:18 smithi146 ceph-mon[103278]: pgmap v12278: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:18.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:18 smithi078 ceph-mon[99959]: pgmap v12278: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:18.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:18 smithi078 ceph-mon[91136]: pgmap v12278: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:20 smithi146 ceph-mon[103278]: pgmap v12279: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:20 smithi078 ceph-mon[99959]: pgmap v12279: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:20.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:20 smithi078 ceph-mon[91136]: pgmap v12279: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:09:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:09:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:09:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:21.891Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:21.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:21.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:22 smithi146 ceph-mon[103278]: pgmap v12280: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:22 smithi078 ceph-mon[91136]: pgmap v12280: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:22 smithi078 ceph-mon[99959]: pgmap v12280: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:24 smithi146 ceph-mon[103278]: pgmap v12281: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[91136]: pgmap v12281: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[99959]: pgmap v12281: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:26 smithi146 ceph-mon[103278]: pgmap v12282: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:26 smithi078 ceph-mon[91136]: pgmap v12282: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:26 smithi078 ceph-mon[99959]: pgmap v12282: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:28 smithi146 ceph-mon[103278]: pgmap v12283: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:28 smithi078 ceph-mon[91136]: pgmap v12283: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:28 smithi078 ceph-mon[99959]: pgmap v12283: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:30 smithi078 ceph-mon[91136]: pgmap v12284: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:30 smithi078 ceph-mon[99959]: pgmap v12284: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:30 smithi146 ceph-mon[103278]: pgmap v12284: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:09:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:09:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:09:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:32.067 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:09:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:31.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:31.892Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:31.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 7h - - 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 6h - - 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 8s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 3s ago 6h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 8s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 3s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 8s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 8s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 3s ago 6h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 8s ago 6h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:09:32.624 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 8s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 3s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 8s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 8s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 8s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 8s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 3s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 3s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 3s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 3s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 3s ago 6h 70.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 8s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:09:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 8s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:09:32.626 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 3s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:09:32.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:32 smithi078 ceph-mon[91136]: pgmap v12285: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:32.626 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:32 smithi078 ceph-mon[99959]: pgmap v12285: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:32 smithi146 ceph-mon[103278]: pgmap v12285: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:32.936 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:09:32.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:09:32.937 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:09:33.206 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:09:33.206 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:09:33.206 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:09:33.206 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:09:33.206 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:09:33.206 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:09:33.207 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:09:33.519 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:09:33.519 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:09:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:09:33.520 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:09:33.520 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:09:33.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:33 smithi078 ceph-mon[91136]: from='client.71907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:33.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:33 smithi078 ceph-mon[91136]: from='client.71913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:33.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/956129671' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:33.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:33 smithi078 ceph-mon[99959]: from='client.71907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:33.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:33 smithi078 ceph-mon[99959]: from='client.71913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:33.520 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/956129671' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:33 smithi146 ceph-mon[103278]: from='client.71907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:33 smithi146 ceph-mon[103278]: from='client.71913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/956129671' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:09:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:34 smithi146 ceph-mon[103278]: from='client.91835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:34 smithi146 ceph-mon[103278]: from='client.71931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:34 smithi146 ceph-mon[103278]: pgmap v12286: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2539130423' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[91136]: from='client.91835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[91136]: from='client.71931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[91136]: pgmap v12286: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2539130423' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[99959]: from='client.91835 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[99959]: from='client.71931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:09:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[99959]: pgmap v12286: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2539130423' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:09:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:36 smithi146 ceph-mon[103278]: pgmap v12287: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:36 smithi078 ceph-mon[91136]: pgmap v12287: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:36 smithi078 ceph-mon[99959]: pgmap v12287: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:38.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:38 smithi078 ceph-mon[91136]: pgmap v12288: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:38.379 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:38 smithi078 ceph-mon[99959]: pgmap v12288: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:38.700 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:38 smithi146 ceph-mon[103278]: pgmap v12288: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:40.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:40 smithi078 ceph-mon[99959]: pgmap v12289: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:40 smithi078 ceph-mon[91136]: pgmap v12289: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:40.761 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:40 smithi146 ceph-mon[103278]: pgmap v12289: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:09:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:09:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:09:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:41.892Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:41.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:41.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:42 smithi146 ceph-mon[103278]: pgmap v12290: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:42 smithi078 ceph-mon[91136]: pgmap v12290: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:42 smithi078 ceph-mon[99959]: pgmap v12290: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:44 smithi146 ceph-mon[103278]: pgmap v12291: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[91136]: pgmap v12291: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[99959]: pgmap v12291: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:46 smithi146 ceph-mon[103278]: pgmap v12292: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:46 smithi078 ceph-mon[91136]: pgmap v12292: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:46 smithi078 ceph-mon[99959]: pgmap v12292: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:48 smithi146 ceph-mon[103278]: pgmap v12293: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:48 smithi078 ceph-mon[91136]: pgmap v12293: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:48 smithi078 ceph-mon[99959]: pgmap v12293: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:49 smithi146 ceph-mon[103278]: pgmap v12294: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[91136]: pgmap v12294: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:09:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:49 smithi078 ceph-mon[99959]: pgmap v12294: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:09:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:09:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:09:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:50 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:50 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:51.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:50 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:09:52.187 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:51 smithi078 ceph-mon[99959]: pgmap v12295: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:52.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:51 smithi078 ceph-mon[91136]: pgmap v12295: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:51.893Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:51.893Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:09:52.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:09:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:09:51.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:09:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:51 smithi146 ceph-mon[103278]: pgmap v12295: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:54 smithi146 ceph-mon[103278]: pgmap v12296: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:54 smithi078 ceph-mon[91136]: pgmap v12296: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:54 smithi078 ceph-mon[99959]: pgmap v12296: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:09:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:56 smithi146 ceph-mon[103278]: pgmap v12297: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:56 smithi078 ceph-mon[91136]: pgmap v12297: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:56 smithi078 ceph-mon[99959]: pgmap v12297: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:58.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:58 smithi078 ceph-mon[91136]: pgmap v12298: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:58.666 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:58 smithi078 ceph-mon[99959]: pgmap v12298: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:58 smithi146 ceph-mon[103278]: pgmap v12298: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:09:59.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:09:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:09:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:09:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:09:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:00.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T03:09:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:00 smithi146 ceph-mon[103278]: pgmap v12299: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:00.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:00 smithi078 ceph-mon[91136]: pgmap v12299: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:00 smithi078 ceph-mon[99959]: pgmap v12299: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:10:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:10:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:10:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:01.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:01.894Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:01.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:02 smithi146 ceph-mon[103278]: pgmap v12300: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:02 smithi078 ceph-mon[91136]: pgmap v12300: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:02 smithi078 ceph-mon[99959]: pgmap v12300: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:03.813 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 7h - - 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 7h - - 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 0s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 15s ago 6h 90.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 0s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 15s ago 6h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 0s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 0s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:10:04.397 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 15s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 0s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 0s ago 6h 14.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 15s ago 6h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 0s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 0s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 0s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 0s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 15s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 15s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 15s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 15s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 15s ago 6h 70.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:10:04.398 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 0s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:10:04.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:10:04.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 0s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:10:04.399 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 15s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:10:04.712 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:10:04.712 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:10:04.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:10:04.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[99959]: pgmap v12301: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[91136]: pgmap v12301: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:04 smithi146 ceph-mon[103278]: pgmap v12301: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:10:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:10:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:10:04.980 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:10:05.288 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:05.288 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:05.288 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:05.288 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:05 smithi146 ceph-mon[103278]: from='client.91859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:05 smithi146 ceph-mon[103278]: from='client.91865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:05 smithi146 ceph-mon[103278]: from='client.91871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3131034885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1137197770' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[91136]: from='client.91859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[91136]: from='client.91865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[91136]: from='client.91871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3131034885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1137197770' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[99959]: from='client.91859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[99959]: from='client.91865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[99959]: from='client.91871 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3131034885' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1137197770' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:06 smithi146 ceph-mon[103278]: from='client.91883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:06 smithi146 ceph-mon[103278]: pgmap v12302: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:06 smithi078 ceph-mon[91136]: from='client.91883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:06 smithi078 ceph-mon[91136]: pgmap v12302: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:06 smithi078 ceph-mon[99959]: from='client.91883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:06 smithi078 ceph-mon[99959]: pgmap v12302: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:08.590 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:08 smithi146 ceph-mon[103278]: pgmap v12303: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:08 smithi078 ceph-mon[91136]: pgmap v12303: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:08 smithi078 ceph-mon[99959]: pgmap v12303: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:10.390 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.390 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.390 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.391 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[99959]: pgmap v12304: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:10.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.391 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-mon[91136]: pgmap v12304: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:10 smithi146 ceph-mon[103278]: pgmap v12304: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:10:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:10:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:10:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:12 smithi078 ceph-mon[91136]: pgmap v12305: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:12 smithi078 ceph-mon[99959]: pgmap v12305: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:11.894Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:11.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:11.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:12 smithi146 ceph-mon[103278]: pgmap v12305: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:14 smithi146 ceph-mon[103278]: pgmap v12306: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:14 smithi078 ceph-mon[99959]: pgmap v12306: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:14 smithi078 ceph-mon[91136]: pgmap v12306: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:16 smithi146 ceph-mon[103278]: pgmap v12307: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:16 smithi078 ceph-mon[91136]: pgmap v12307: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:16 smithi078 ceph-mon[99959]: pgmap v12307: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:18.604 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:18 smithi078 ceph-mon[99959]: pgmap v12308: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:18.604 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:18 smithi078 ceph-mon[91136]: pgmap v12308: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:18.625 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:18 smithi146 ceph-mon[103278]: pgmap v12308: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:20.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:20 smithi078 ceph-mon[91136]: pgmap v12309: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:20.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:20 smithi078 ceph-mon[99959]: pgmap v12309: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:20.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:20 smithi146 ceph-mon[103278]: pgmap v12309: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:21.024 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:10:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:10:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:10:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:21.895Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:21.895Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:21.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:22.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:22 smithi078 ceph-mon[99959]: pgmap v12310: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:22.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:22 smithi078 ceph-mon[91136]: pgmap v12310: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:22 smithi146 ceph-mon[103278]: pgmap v12310: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[91136]: pgmap v12311: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[99959]: pgmap v12311: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:24 smithi146 ceph-mon[103278]: pgmap v12311: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:26 smithi146 ceph-mon[103278]: pgmap v12312: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:26 smithi078 ceph-mon[91136]: pgmap v12312: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:26 smithi078 ceph-mon[99959]: pgmap v12312: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:28 smithi146 ceph-mon[103278]: pgmap v12313: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:28 smithi078 ceph-mon[91136]: pgmap v12313: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:28 smithi078 ceph-mon[99959]: pgmap v12313: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:30.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:30 smithi078 ceph-mon[99959]: pgmap v12314: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:30 smithi078 ceph-mon[91136]: pgmap v12314: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:30 smithi146 ceph-mon[103278]: pgmap v12314: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:10:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:10:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:10:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:31.896Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:31.896Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:31.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:32 smithi146 ceph-mon[103278]: pgmap v12315: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:32 smithi078 ceph-mon[99959]: pgmap v12315: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:32 smithi078 ceph-mon[91136]: pgmap v12315: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:34 smithi146 ceph-mon[103278]: pgmap v12316: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:34 smithi078 ceph-mon[99959]: pgmap v12316: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:34 smithi078 ceph-mon[91136]: pgmap v12316: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:35.581 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:10:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:10:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 7h - - 2024-09-21T03:10:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 7h - - 2024-09-21T03:10:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 11s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:10:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 6s ago 6h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 11s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 6s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 11s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 11s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 6s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 11s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 11s ago 6h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 6s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 11s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 11s ago 6h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 11s ago 6h 84.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 11s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 6s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:10:36.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 6s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 6s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 6s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 6s ago 6h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 11s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 11s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:10:36.112 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 6s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:10:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:10:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:10:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:10:36.421 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:10:36.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:36 smithi078 ceph-mon[91136]: pgmap v12317: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:36.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:36 smithi078 ceph-mon[99959]: pgmap v12317: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:36.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:10:36.687 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:10:36.687 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:10:36.687 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:10:36.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:10:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:36 smithi146 ceph-mon[103278]: pgmap v12317: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:36.997 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:10:36.998 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:10:36.998 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:36.998 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:10:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:37 smithi146 ceph-mon[103278]: from='client.91895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:37 smithi146 ceph-mon[103278]: from='client.91901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:37 smithi146 ceph-mon[103278]: from='client.71991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2136567969' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3854384894' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[91136]: from='client.91895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[91136]: from='client.91901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[91136]: from='client.71991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2136567969' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3854384894' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[99959]: from='client.91895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[99959]: from='client.91901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[99959]: from='client.71991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2136567969' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:10:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3854384894' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:10:38.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:38 smithi146 ceph-mon[103278]: from='client.91919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:38.667 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:38 smithi146 ceph-mon[103278]: pgmap v12318: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:38 smithi078 ceph-mon[91136]: from='client.91919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:38 smithi078 ceph-mon[91136]: pgmap v12318: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:38 smithi078 ceph-mon[99959]: from='client.91919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:10:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:38 smithi078 ceph-mon[99959]: pgmap v12318: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:40.380 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:40 smithi146 ceph-mon[103278]: pgmap v12319: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:40.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:40 smithi078 ceph-mon[99959]: pgmap v12319: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:40.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:40 smithi078 ceph-mon[91136]: pgmap v12319: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:41.002 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:10:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:10:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:10:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:41.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:41.623 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:42.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:41.897Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:42.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:41.897Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:42.188 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:41.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:42.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:42 smithi078 ceph-mon[91136]: pgmap v12320: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:42.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:42 smithi078 ceph-mon[99959]: pgmap v12320: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:42.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:42 smithi146 ceph-mon[103278]: pgmap v12320: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:44 smithi146 ceph-mon[103278]: pgmap v12321: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[91136]: pgmap v12321: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[99959]: pgmap v12321: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:46 smithi146 ceph-mon[103278]: pgmap v12322: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:46 smithi078 ceph-mon[91136]: pgmap v12322: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:46 smithi078 ceph-mon[99959]: pgmap v12322: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:48 smithi146 ceph-mon[103278]: pgmap v12323: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:48 smithi078 ceph-mon[91136]: pgmap v12323: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:48 smithi078 ceph-mon[99959]: pgmap v12323: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[91136]: pgmap v12324: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[99959]: pgmap v12324: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.611 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:50 smithi146 ceph-mon[103278]: pgmap v12324: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:10:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:10:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:10:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:10:51.694 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:51.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:10:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:51.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:51.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:10:52.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:10:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:10:51.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:10:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:52 smithi146 ceph-mon[103278]: pgmap v12325: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:52 smithi078 ceph-mon[91136]: pgmap v12325: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:52 smithi078 ceph-mon[99959]: pgmap v12325: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:10:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:54 smithi146 ceph-mon[103278]: pgmap v12326: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:54 smithi078 ceph-mon[91136]: pgmap v12326: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:54 smithi078 ceph-mon[99959]: pgmap v12326: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:10:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:56 smithi146 ceph-mon[103278]: pgmap v12327: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:10:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:56 smithi078 ceph-mon[91136]: pgmap v12327: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:10:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:56 smithi078 ceph-mon[99959]: pgmap v12327: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T03:10:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:58 smithi146 ceph-mon[103278]: pgmap v12328: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:10:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:58 smithi078 ceph-mon[91136]: pgmap v12328: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:10:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:58 smithi078 ceph-mon[99959]: pgmap v12328: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:10:59.754 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:10:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:59.754 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:10:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:10:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:10:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:00.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:00 smithi078 ceph-mon[99959]: pgmap v12329: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:11:00.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:00 smithi078 ceph-mon[91136]: pgmap v12329: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:11:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:00 smithi146 ceph-mon[103278]: pgmap v12329: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:11:00.946 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:11:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:11:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:11:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:02.220 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:01.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:02.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:01.898Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:02.221 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:01.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:02 smithi146 ceph-mon[103278]: pgmap v12330: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:11:02.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:02 smithi078 ceph-mon[91136]: pgmap v12330: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:11:02.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:02 smithi078 ceph-mon[99959]: pgmap v12330: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:11:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:04 smithi146 ceph-mon[103278]: pgmap v12331: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:11:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:04 smithi078 ceph-mon[91136]: pgmap v12331: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:11:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:04 smithi078 ceph-mon[99959]: pgmap v12331: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:11:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:06 smithi146 ceph-mon[103278]: pgmap v12332: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T03:11:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:06 smithi078 ceph-mon[99959]: pgmap v12332: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T03:11:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:06 smithi078 ceph-mon[91136]: pgmap v12332: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T03:11:07.302 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 7h - - 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 7h - - 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 3s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 18s ago 6h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 3s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 18s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 3s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:11:07.830 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 3s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 18s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 3s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 3s ago 6h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 18s ago 6h 14.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 3s ago 7h 96.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 3s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 3s ago 6h 84.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 3s ago 6h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 18s ago 6h 118M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 18s ago 6h 92.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 18s ago 6h 99.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 18s ago 6h 95.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 18s ago 6h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 3s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:11:07.831 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 3s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:11:07.832 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 18s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:11:08.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:11:08.138 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:11:08.406 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:11:08.407 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:11:08.714 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:11:08.714 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:08.714 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:08.714 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:08.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:08 smithi078 ceph-mon[99959]: pgmap v12333: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T03:11:08.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:08 smithi078 ceph-mon[99959]: from='client.91931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:08.714 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1309183387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:08.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:08 smithi078 ceph-mon[91136]: pgmap v12333: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T03:11:08.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:08 smithi078 ceph-mon[91136]: from='client.91931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:08.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1309183387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:08 smithi146 ceph-mon[103278]: pgmap v12333: 161 pgs: 161 active+clean; 585 KiB data, 385 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T03:11:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:08 smithi146 ceph-mon[103278]: from='client.91931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1309183387' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='client.72021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='client.72027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='client.72039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/382514002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='client.72021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='client.72027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='client.72039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/382514002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='client.72021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='client.72027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='client.72039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/382514002' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:10.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:10 smithi146 ceph-mon[103278]: pgmap v12334: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:11:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:11:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:11:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:11:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:10 smithi078 ceph-mon[99959]: pgmap v12334: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:11:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:10 smithi078 ceph-mon[91136]: pgmap v12334: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:11:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:11.898Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:11.899Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:11.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:12 smithi146 ceph-mon[103278]: pgmap v12335: 161 pgs: 161 active+clean; 585 KiB data, 410 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T03:11:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:12 smithi078 ceph-mon[99959]: pgmap v12335: 161 pgs: 161 active+clean; 585 KiB data, 410 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T03:11:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:12 smithi078 ceph-mon[91136]: pgmap v12335: 161 pgs: 161 active+clean; 585 KiB data, 410 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s 2024-09-21T03:11:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:14 smithi146 ceph-mon[103278]: pgmap v12336: 161 pgs: 161 active+clean; 585 KiB data, 410 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:11:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:14 smithi078 ceph-mon[91136]: pgmap v12336: 161 pgs: 161 active+clean; 585 KiB data, 410 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:11:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:14 smithi078 ceph-mon[99959]: pgmap v12336: 161 pgs: 161 active+clean; 585 KiB data, 410 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:11:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:16 smithi146 ceph-mon[103278]: pgmap v12337: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:11:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:16 smithi078 ceph-mon[91136]: pgmap v12337: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:11:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:16 smithi078 ceph-mon[99959]: pgmap v12337: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 111 op/s 2024-09-21T03:11:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:18 smithi146 ceph-mon[103278]: pgmap v12338: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:11:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:18 smithi078 ceph-mon[91136]: pgmap v12338: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:11:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:18 smithi078 ceph-mon[99959]: pgmap v12338: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:11:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:20 smithi146 ceph-mon[103278]: pgmap v12339: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:11:20.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:20 smithi078 ceph-mon[91136]: pgmap v12339: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:11:20.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:20 smithi078 ceph-mon[99959]: pgmap v12339: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:11:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:11:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:11:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:11:21.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:21.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:21.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:21.899Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:21.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:21.900Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:21.956 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:21.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:22.617 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:22 smithi078 ceph-mon[99959]: pgmap v12340: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:11:22.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:22 smithi078 ceph-mon[91136]: pgmap v12340: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:11:22.764 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:22 smithi146 ceph-mon[103278]: pgmap v12340: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:11:24.754 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:24 smithi146 ceph-mon[103278]: pgmap v12341: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:11:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:24 smithi078 ceph-mon[91136]: pgmap v12341: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:11:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:24 smithi078 ceph-mon[99959]: pgmap v12341: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:11:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:25 smithi146 ceph-mon[103278]: pgmap v12342: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-21T03:11:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[99959]: pgmap v12342: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-21T03:11:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:25 smithi078 ceph-mon[91136]: pgmap v12342: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 135 op/s 2024-09-21T03:11:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:28 smithi146 ceph-mon[103278]: pgmap v12343: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T03:11:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:28 smithi078 ceph-mon[91136]: pgmap v12343: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T03:11:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:28 smithi078 ceph-mon[99959]: pgmap v12343: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-09-21T03:11:29.589 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:30.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[91136]: pgmap v12344: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[99959]: pgmap v12344: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:30 smithi146 ceph-mon[103278]: pgmap v12344: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:11:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:31.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:11:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:11:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:11:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:31.900Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:31.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:31.901Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:32 smithi146 ceph-mon[103278]: pgmap v12345: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:11:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:32 smithi078 ceph-mon[91136]: pgmap v12345: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:11:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:32 smithi078 ceph-mon[99959]: pgmap v12345: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:11:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:34 smithi146 ceph-mon[103278]: pgmap v12346: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:11:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:34 smithi078 ceph-mon[99959]: pgmap v12346: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:11:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:34 smithi078 ceph-mon[91136]: pgmap v12346: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:11:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:36 smithi146 ceph-mon[103278]: pgmap v12347: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T03:11:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:36 smithi078 ceph-mon[99959]: pgmap v12347: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T03:11:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:36 smithi078 ceph-mon[91136]: pgmap v12347: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-09-21T03:11:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:38 smithi146 ceph-mon[103278]: pgmap v12348: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:11:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:38 smithi078 ceph-mon[99959]: pgmap v12348: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:11:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:38 smithi078 ceph-mon[91136]: pgmap v12348: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T03:11:39.007 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:11:39.531 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:11:39.532 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 7h - - 2024-09-21T03:11:39.532 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 7h - - 2024-09-21T03:11:39.532 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 14s ago 6h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:11:39.532 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 9s ago 6h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:11:39.532 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 14s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 9s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 14s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 14s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 9s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 14s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 14s ago 6h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:11:39.533 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 9s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:11:39.534 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 14s ago 7h 97.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:11:39.534 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 14s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:11:39.534 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 14s ago 7h 85.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:11:39.534 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 14s ago 6h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:11:39.534 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 9s ago 6h 119M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 9s ago 6h 94.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 9s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 9s ago 6h 96.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 9s ago 6h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 14s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:11:39.535 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 14s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:11:39.536 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 9s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:11:39.840 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:11:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:11:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:11:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:11:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:11:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:11:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:11:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:11:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:11:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:11:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:11:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:11:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:11:39.843 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:11:40.111 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:11:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:11:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:11:40.112 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:11:40.422 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:11:40.422 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:40.422 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:11:40.422 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:40.422 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:11:40.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[99959]: from='client.72051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.422 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[99959]: from='client.91973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[99959]: pgmap v12349: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[99959]: from='client.72063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/926139408' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[91136]: from='client.72051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[91136]: from='client.91973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[91136]: pgmap v12349: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[91136]: from='client.72063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/926139408' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:40 smithi146 ceph-mon[103278]: from='client.72051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:40 smithi146 ceph-mon[103278]: from='client.91973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:40 smithi146 ceph-mon[103278]: pgmap v12349: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:40 smithi146 ceph-mon[103278]: from='client.72063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/926139408' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:11:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:11:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:11:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:11:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:41 smithi146 ceph-mon[103278]: from='client.72075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3517881239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-mon[99959]: from='client.72075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3517881239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-mon[91136]: from='client.72075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:11:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3517881239' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:11:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:41.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:41.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:41.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:42.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:42 smithi078 ceph-mon[99959]: pgmap v12350: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:42.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:42 smithi078 ceph-mon[91136]: pgmap v12350: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:42.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:42 smithi146 ceph-mon[103278]: pgmap v12350: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:44.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:44 smithi078 ceph-mon[91136]: pgmap v12351: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:44.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:44.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:44 smithi078 ceph-mon[99959]: pgmap v12351: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:44.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:44 smithi146 ceph-mon[103278]: pgmap v12351: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:44.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:45 smithi146 ceph-mon[103278]: pgmap v12352: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[91136]: pgmap v12352: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:45 smithi078 ceph-mon[99959]: pgmap v12352: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:48 smithi146 ceph-mon[103278]: pgmap v12353: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:48 smithi078 ceph-mon[99959]: pgmap v12353: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:48 smithi078 ceph-mon[91136]: pgmap v12353: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[99959]: pgmap v12354: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[91136]: pgmap v12354: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:50 smithi146 ceph-mon[103278]: pgmap v12354: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:11:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:11:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:11:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:11:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:11:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:51.901Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:51.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:11:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:11:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:11:51.902Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:11:52.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:52 smithi078 ceph-mon[91136]: pgmap v12355: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:52.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:52 smithi078 ceph-mon[99959]: pgmap v12355: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:52 smithi146 ceph-mon[103278]: pgmap v12355: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:54 smithi146 ceph-mon[103278]: pgmap v12356: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:54 smithi078 ceph-mon[91136]: pgmap v12356: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:54 smithi078 ceph-mon[99959]: pgmap v12356: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:11:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:56 smithi146 ceph-mon[103278]: pgmap v12357: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:56 smithi078 ceph-mon[91136]: pgmap v12357: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:56 smithi078 ceph-mon[99959]: pgmap v12357: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:58 smithi146 ceph-mon[103278]: pgmap v12358: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:58 smithi078 ceph-mon[91136]: pgmap v12358: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:58 smithi078 ceph-mon[99959]: pgmap v12358: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:11:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:11:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:11:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:11:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:11:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:00 smithi146 ceph-mon[103278]: pgmap v12359: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:00 smithi078 ceph-mon[91136]: pgmap v12359: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:12:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:12:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:12:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:00 smithi078 ceph-mon[99959]: pgmap v12359: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:01.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:01.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:01.902Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:01.903Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:01.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:02.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:02 smithi078 ceph-mon[99959]: pgmap v12360: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:02.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:02 smithi078 ceph-mon[91136]: pgmap v12360: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:02 smithi146 ceph-mon[103278]: pgmap v12360: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:04.512 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:04 smithi146 ceph-mon[103278]: pgmap v12361: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:04 smithi078 ceph-mon[91136]: pgmap v12361: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:04.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:04 smithi078 ceph-mon[99959]: pgmap v12361: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:06.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.201 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:05 smithi146 ceph-mon[103278]: pgmap v12362: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:06.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[91136]: pgmap v12362: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:05 smithi078 ceph-mon[99959]: pgmap v12362: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:08 smithi146 ceph-mon[103278]: pgmap v12363: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:08 smithi078 ceph-mon[91136]: pgmap v12363: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:08 smithi078 ceph-mon[99959]: pgmap v12363: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[99959]: pgmap v12364: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[91136]: pgmap v12364: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:10.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.719 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:12:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:10 smithi146 ceph-mon[103278]: pgmap v12364: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:12:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:12:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:12:11.246 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:12:11.246 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 7h - - 2024-09-21T03:12:11.246 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 7h - - 2024-09-21T03:12:11.246 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 6s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:12:11.246 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 1s ago 6h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 6s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 1s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 6s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 6s ago 7h 556M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 1s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 6s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 6s ago 6h 15.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 1s ago 6h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 6s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 6s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 6s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 6s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 1s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 1s ago 6h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:12:11.247 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 1s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:12:11.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 1s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:12:11.248 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 1s ago 6h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:12:11.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 6s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:12:11.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:12:11.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 6s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:12:11.248 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 1s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:12:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:11.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:11.559 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:12:11.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:12:11.559 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:12:11.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:11.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:12:11.560 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:12:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:12:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:12:11.841 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:12:12.148 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:12:12.148 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:12.148 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:12.149 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:12.149 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:12.149 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:11.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:12.149 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:11.904Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:12.149 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:11.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[91136]: from='client.92003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[91136]: from='client.72093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[91136]: from='client.92015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[91136]: pgmap v12365: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3651587175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:12.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1752276837' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[99959]: from='client.92003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[99959]: from='client.72093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[99959]: from='client.92015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[99959]: pgmap v12365: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3651587175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:12.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1752276837' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:12 smithi146 ceph-mon[103278]: from='client.92003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:12 smithi146 ceph-mon[103278]: from='client.72093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:12 smithi146 ceph-mon[103278]: from='client.92015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:12 smithi146 ceph-mon[103278]: pgmap v12365: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3651587175' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1752276837' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:13.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:13 smithi078 ceph-mon[91136]: from='client.92027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:13.664 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:13 smithi078 ceph-mon[99959]: from='client.92027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:13 smithi146 ceph-mon[103278]: from='client.92027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:14 smithi078 ceph-mon[91136]: pgmap v12366: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:14 smithi078 ceph-mon[99959]: pgmap v12366: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:14 smithi146 ceph-mon[103278]: pgmap v12366: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:16 smithi146 ceph-mon[103278]: pgmap v12367: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:16 smithi078 ceph-mon[91136]: pgmap v12367: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:16 smithi078 ceph-mon[99959]: pgmap v12367: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:18 smithi146 ceph-mon[103278]: pgmap v12368: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:18 smithi078 ceph-mon[91136]: pgmap v12368: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:18 smithi078 ceph-mon[99959]: pgmap v12368: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:20 smithi146 ceph-mon[103278]: pgmap v12369: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:20 smithi078 ceph-mon[99959]: pgmap v12369: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:20 smithi078 ceph-mon[91136]: pgmap v12369: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:12:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:12:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:12:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:21.904Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:21.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:21.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:22.619 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:22 smithi078 ceph-mon[99959]: pgmap v12370: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:22.620 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:22 smithi078 ceph-mon[91136]: pgmap v12370: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:22 smithi146 ceph-mon[103278]: pgmap v12370: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:24.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:24 smithi078 ceph-mon[91136]: pgmap v12371: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:24.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:24 smithi078 ceph-mon[99959]: pgmap v12371: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:24.644 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:24 smithi146 ceph-mon[103278]: pgmap v12371: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:26.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.108 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:25 smithi146 ceph-mon[103278]: pgmap v12372: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[91136]: pgmap v12372: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:25 smithi078 ceph-mon[99959]: pgmap v12372: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:28 smithi146 ceph-mon[103278]: pgmap v12373: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:28 smithi078 ceph-mon[91136]: pgmap v12373: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:28 smithi078 ceph-mon[99959]: pgmap v12373: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[99959]: pgmap v12374: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[91136]: pgmap v12374: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:30 smithi146 ceph-mon[103278]: pgmap v12374: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:12:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:12:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:12:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:31.905Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:31.905Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:31.906Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:32 smithi146 ceph-mon[103278]: pgmap v12375: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:32 smithi078 ceph-mon[91136]: pgmap v12375: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:32 smithi078 ceph-mon[99959]: pgmap v12375: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:34 smithi146 ceph-mon[103278]: pgmap v12376: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:34 smithi078 ceph-mon[91136]: pgmap v12376: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:34 smithi078 ceph-mon[99959]: pgmap v12376: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:36 smithi146 ceph-mon[103278]: pgmap v12377: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:36 smithi078 ceph-mon[91136]: pgmap v12377: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:36 smithi078 ceph-mon[99959]: pgmap v12377: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:38 smithi146 ceph-mon[103278]: pgmap v12378: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:38 smithi078 ceph-mon[91136]: pgmap v12378: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:38 smithi078 ceph-mon[99959]: pgmap v12378: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:40 smithi146 ceph-mon[103278]: pgmap v12379: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:40 smithi078 ceph-mon[91136]: pgmap v12379: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:40 smithi078 ceph-mon[99959]: pgmap v12379: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:12:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:12:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:12:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:41.906Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:41.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:41.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:42.442 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:12:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:42 smithi146 ceph-mon[103278]: pgmap v12380: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:42 smithi078 ceph-mon[91136]: pgmap v12380: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:42 smithi078 ceph-mon[99959]: pgmap v12380: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:42.970 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:12:42.970 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 7h - - 2024-09-21T03:12:42.970 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 7h - - 2024-09-21T03:12:42.970 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 18s ago 6h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:12:42.970 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 13s ago 6h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:12:42.970 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 18s ago 6h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 13s ago 6h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 13s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 13s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 13s ago 6h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:12:42.971 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 6h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:12:42.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:12:42.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:12:42.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:12:42.972 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:12:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:12:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:12:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:12:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:12:43.296 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:12:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:43 smithi078 ceph-mon[99959]: from='client.92039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/588835514' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:43 smithi078 ceph-mon[91136]: from='client.92039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/588835514' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:43.570 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:12:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:12:43.572 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:12:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:43 smithi146 ceph-mon[103278]: from='client.92039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/588835514' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:12:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:12:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:12:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:12:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:44 smithi146 ceph-mon[103278]: from='client.92045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:44 smithi146 ceph-mon[103278]: from='client.72135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:44 smithi146 ceph-mon[103278]: pgmap v12381: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1976473442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[91136]: from='client.92045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[91136]: from='client.72135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[91136]: pgmap v12381: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1976473442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:44.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[99959]: from='client.92045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[99959]: from='client.72135 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[99959]: pgmap v12381: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1976473442' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:12:45.500 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:45 smithi146 ceph-mon[103278]: from='client.92063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:45.500 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.500 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.500 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[99959]: from='client.92063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[91136]: from='client.92063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:45.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:46.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:46 smithi078 ceph-mon[91136]: pgmap v12382: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:46.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:46 smithi078 ceph-mon[99959]: pgmap v12382: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:46 smithi146 ceph-mon[103278]: pgmap v12382: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:48 smithi146 ceph-mon[103278]: pgmap v12383: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:48 smithi078 ceph-mon[91136]: pgmap v12383: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:48 smithi078 ceph-mon[99959]: pgmap v12383: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:50 smithi146 ceph-mon[103278]: pgmap v12384: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:12:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:12:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[99959]: pgmap v12384: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[91136]: pgmap v12384: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:12:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:12:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:51.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:51.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:12:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:12:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:12:51.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:12:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:52 smithi146 ceph-mon[103278]: pgmap v12385: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:52 smithi078 ceph-mon[91136]: pgmap v12385: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:52 smithi078 ceph-mon[99959]: pgmap v12385: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:54 smithi146 ceph-mon[103278]: pgmap v12386: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:54 smithi078 ceph-mon[99959]: pgmap v12386: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:54 smithi078 ceph-mon[91136]: pgmap v12386: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:12:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:56 smithi146 ceph-mon[103278]: pgmap v12387: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:56 smithi078 ceph-mon[91136]: pgmap v12387: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:56 smithi078 ceph-mon[99959]: pgmap v12387: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:58 smithi146 ceph-mon[103278]: pgmap v12388: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:58 smithi078 ceph-mon[91136]: pgmap v12388: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:58 smithi078 ceph-mon[99959]: pgmap v12388: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:12:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:12:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:12:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:12:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:12:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:00 smithi146 ceph-mon[103278]: pgmap v12389: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:00 smithi078 ceph-mon[91136]: pgmap v12389: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:00 smithi078 ceph-mon[99959]: pgmap v12389: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:13:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:13:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:13:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:01.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:01.907Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:01.907Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:01.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:02 smithi146 ceph-mon[103278]: pgmap v12390: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:02 smithi078 ceph-mon[91136]: pgmap v12390: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:02 smithi078 ceph-mon[99959]: pgmap v12390: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:04 smithi146 ceph-mon[103278]: pgmap v12391: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:04 smithi078 ceph-mon[91136]: pgmap v12391: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:04 smithi078 ceph-mon[99959]: pgmap v12391: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:06 smithi146 ceph-mon[103278]: pgmap v12392: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:06.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.309 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[99959]: pgmap v12392: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:06.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:06.309 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:06 smithi078 ceph-mon[91136]: pgmap v12392: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:08.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:08 smithi146 ceph-mon[103278]: pgmap v12393: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:08 smithi078 ceph-mon[91136]: pgmap v12393: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:08 smithi078 ceph-mon[99959]: pgmap v12393: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:10 smithi146 ceph-mon[103278]: pgmap v12394: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:13:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:13:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[99959]: pgmap v12394: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[91136]: pgmap v12394: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:11.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:11.908Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:11.908Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:11.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:12 smithi146 ceph-mon[103278]: pgmap v12395: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:12.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:12 smithi078 ceph-mon[99959]: pgmap v12395: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:12 smithi078 ceph-mon[91136]: pgmap v12395: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:14.184 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:13:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:14 smithi078 ceph-mon[91136]: pgmap v12396: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:14 smithi078 ceph-mon[99959]: pgmap v12396: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:14.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 7h - - 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 7h - - 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 9s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 4s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 9s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 4s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 9s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:13:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 9s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 4s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 9s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 9s ago 6h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 4s ago 6h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 9s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 9s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 9s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 4s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 4s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 4s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:13:14.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 4s ago 6h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:13:14.716 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 4s ago 6h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:13:14.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:13:14.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:13:14.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:13:14.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 4s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:13:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:14 smithi146 ceph-mon[103278]: pgmap v12396: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:15.021 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:13:15.022 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:13:15.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:13:15.023 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:13:15.292 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:13:15.293 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:13:15.601 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:13:15.601 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:15.602 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:15.602 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:15.602 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:15 smithi078 ceph-mon[99959]: from='client.72159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:15.602 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:15 smithi078 ceph-mon[99959]: from='client.72165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:15.602 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2832379862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:15.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:15 smithi078 ceph-mon[91136]: from='client.72159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:15.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:15 smithi078 ceph-mon[91136]: from='client.72165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:15.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2832379862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:15 smithi146 ceph-mon[103278]: from='client.72159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:15 smithi146 ceph-mon[103278]: from='client.72165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2832379862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:16 smithi146 ceph-mon[103278]: from='client.72171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:16 smithi146 ceph-mon[103278]: from='client.72183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:16 smithi146 ceph-mon[103278]: pgmap v12397: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3009790945' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[91136]: from='client.72171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[91136]: from='client.72183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[91136]: pgmap v12397: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3009790945' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[99959]: from='client.72171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[99959]: from='client.72183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[99959]: pgmap v12397: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3009790945' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:18 smithi146 ceph-mon[103278]: pgmap v12398: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:18 smithi078 ceph-mon[91136]: pgmap v12398: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:18 smithi078 ceph-mon[99959]: pgmap v12398: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:20 smithi146 ceph-mon[103278]: pgmap v12399: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:20 smithi078 ceph-mon[91136]: pgmap v12399: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:20 smithi078 ceph-mon[99959]: pgmap v12399: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:13:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:13:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:13:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:21.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:21.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:21.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:21.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:21.909Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:22 smithi146 ceph-mon[103278]: pgmap v12400: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:22 smithi078 ceph-mon[99959]: pgmap v12400: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:22 smithi078 ceph-mon[91136]: pgmap v12400: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:24.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:24 smithi078 ceph-mon[91136]: pgmap v12401: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:24.689 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:24 smithi078 ceph-mon[99959]: pgmap v12401: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:24 smithi146 ceph-mon[103278]: pgmap v12401: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:26 smithi146 ceph-mon[103278]: pgmap v12402: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[91136]: pgmap v12402: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:26 smithi078 ceph-mon[99959]: pgmap v12402: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:28 smithi146 ceph-mon[103278]: pgmap v12403: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:28 smithi078 ceph-mon[91136]: pgmap v12403: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:28.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:28 smithi078 ceph-mon[99959]: pgmap v12403: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:30 smithi146 ceph-mon[103278]: pgmap v12404: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:13:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:13:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[91136]: pgmap v12404: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[99959]: pgmap v12404: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:31.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:31.909Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:31.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:31.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:32 smithi146 ceph-mon[103278]: pgmap v12405: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:32 smithi078 ceph-mon[91136]: pgmap v12405: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:32 smithi078 ceph-mon[99959]: pgmap v12405: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:34 smithi146 ceph-mon[103278]: pgmap v12406: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:34 smithi078 ceph-mon[91136]: pgmap v12406: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:34 smithi078 ceph-mon[99959]: pgmap v12406: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:36 smithi146 ceph-mon[103278]: pgmap v12407: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:36 smithi078 ceph-mon[91136]: pgmap v12407: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:36 smithi078 ceph-mon[99959]: pgmap v12407: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:38.552 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:38 smithi146 ceph-mon[103278]: pgmap v12408: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:38 smithi078 ceph-mon[99959]: pgmap v12408: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:38 smithi078 ceph-mon[91136]: pgmap v12408: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:40 smithi146 ceph-mon[103278]: pgmap v12409: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:40 smithi078 ceph-mon[99959]: pgmap v12409: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:40 smithi078 ceph-mon[91136]: pgmap v12409: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:13:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:13:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:13:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:41.910Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:41.910Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:41.911Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:42 smithi146 ceph-mon[103278]: pgmap v12410: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:42 smithi078 ceph-mon[99959]: pgmap v12410: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:42 smithi078 ceph-mon[91136]: pgmap v12410: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:44 smithi146 ceph-mon[103278]: pgmap v12411: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:44 smithi078 ceph-mon[99959]: pgmap v12411: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:44 smithi078 ceph-mon[91136]: pgmap v12411: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:45.892 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:13:46.435 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:13:46.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 7h - - 2024-09-21T03:13:46.435 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 7h - - 2024-09-21T03:13:46.435 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 0s ago 6h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 16s ago 6h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 0s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 0s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 0s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 0s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 0s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 16s ago 6h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 0s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 0s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 0s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 0s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:13:46.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 16s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 16s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 16s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 6h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 0s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 0s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:13:46.437 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 16s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:13:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:46 smithi146 ceph-mon[103278]: pgmap v12412: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.757 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:13:46.757 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:13:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:13:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:13:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:13:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:13:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:13:46.759 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:13:46.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[99959]: pgmap v12412: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:46.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.759 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[91136]: pgmap v12412: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:46.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:46.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:13:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:13:47.067 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:13:47.393 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:13:47.394 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:13:47.394 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:13:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:47 smithi146 ceph-mon[103278]: from='client.92111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:47 smithi146 ceph-mon[103278]: from='client.72201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:47 smithi146 ceph-mon[103278]: from='client.92123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1055205980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2366169355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:47.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[91136]: from='client.92111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[91136]: from='client.72201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[91136]: from='client.92123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1055205980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2366169355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[99959]: from='client.92111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[99959]: from='client.72201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[99959]: from='client.92123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1055205980' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:13:47.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2366169355' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:13:48.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:48 smithi146 ceph-mon[103278]: from='client.72219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:48.638 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:48 smithi146 ceph-mon[103278]: pgmap v12413: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:48 smithi078 ceph-mon[91136]: from='client.72219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:48 smithi078 ceph-mon[91136]: pgmap v12413: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:48 smithi078 ceph-mon[99959]: from='client.72219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:13:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:48 smithi078 ceph-mon[99959]: pgmap v12413: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:50 smithi146 ceph-mon[103278]: pgmap v12414: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:50 smithi078 ceph-mon[91136]: pgmap v12414: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:50 smithi078 ceph-mon[99959]: pgmap v12414: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:13:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:13:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:13:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:13:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:13:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:51.911Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:51.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:13:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:13:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:13:51.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:13:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:52 smithi146 ceph-mon[103278]: pgmap v12415: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:52 smithi078 ceph-mon[91136]: pgmap v12415: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:52 smithi078 ceph-mon[99959]: pgmap v12415: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:54 smithi146 ceph-mon[103278]: pgmap v12416: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:54 smithi078 ceph-mon[91136]: pgmap v12416: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:54 smithi078 ceph-mon[99959]: pgmap v12416: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:13:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:56 smithi146 ceph-mon[103278]: pgmap v12417: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:56.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:56 smithi078 ceph-mon[91136]: pgmap v12417: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:56.819 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:56 smithi078 ceph-mon[99959]: pgmap v12417: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:58 smithi146 ceph-mon[103278]: pgmap v12418: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:58 smithi078 ceph-mon[99959]: pgmap v12418: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:58 smithi078 ceph-mon[91136]: pgmap v12418: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:13:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:13:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:13:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:13:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:13:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:00 smithi078 ceph-mon[91136]: pgmap v12419: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:14:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:14:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:14:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:00 smithi078 ceph-mon[99959]: pgmap v12419: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:00 smithi146 ceph-mon[103278]: pgmap v12419: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:01.912Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:01.912Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:01.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:02 smithi078 ceph-mon[99959]: pgmap v12420: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:02 smithi078 ceph-mon[91136]: pgmap v12420: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:02 smithi146 ceph-mon[103278]: pgmap v12420: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:04 smithi078 ceph-mon[91136]: pgmap v12421: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:04 smithi078 ceph-mon[99959]: pgmap v12421: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:04 smithi146 ceph-mon[103278]: pgmap v12421: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[99959]: pgmap v12422: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[91136]: pgmap v12422: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.829 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:06 smithi146 ceph-mon[103278]: pgmap v12422: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:06.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:06.830 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:08.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:08 smithi078 ceph-mon[91136]: pgmap v12423: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:08 smithi078 ceph-mon[99959]: pgmap v12423: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:08.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:08 smithi146 ceph-mon[103278]: pgmap v12423: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:10 smithi078 ceph-mon[91136]: pgmap v12424: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:10.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:10 smithi078 ceph-mon[99959]: pgmap v12424: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:14:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:14:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:14:10.839 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:10 smithi146 ceph-mon[103278]: pgmap v12424: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:11.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:11.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:11.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:11.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:11.914 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[99959]: pgmap v12425: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:11.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:11.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:11.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:11.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:11.915 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-mon[91136]: pgmap v12425: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:11 smithi146 ceph-mon[103278]: pgmap v12425: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:11.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:11.913Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:11.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:14 smithi146 ceph-mon[103278]: pgmap v12426: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:14 smithi078 ceph-mon[91136]: pgmap v12426: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:14 smithi078 ceph-mon[99959]: pgmap v12426: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:16 smithi146 ceph-mon[103278]: pgmap v12427: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:16 smithi078 ceph-mon[91136]: pgmap v12427: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:16 smithi078 ceph-mon[99959]: pgmap v12427: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:17.687 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 7h - - 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 7h - - 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 12s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (6h) 7s ago 6h 89.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 12s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 7s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 12s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 12s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:14:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 7s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 12s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (6h) 12s ago 6h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (6h) 7s ago 6h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 12s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 12s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 12s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 12s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 7s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 7s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 7s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 7s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 7s ago 7h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 12s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:14:18.214 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:14:18.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 12s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:14:18.215 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 7s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:14:18.520 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:14:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:14:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:14:18.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:14:18.521 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:14:18.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:18 smithi078 ceph-mon[91136]: pgmap v12428: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:18.522 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:18 smithi078 ceph-mon[99959]: pgmap v12428: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:18 smithi146 ceph-mon[103278]: pgmap v12428: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:18.791 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:14:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:14:18.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:14:19.098 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:14:19.098 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:19.098 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:19.098 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:19 smithi146 ceph-mon[103278]: from='client.92147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:19 smithi146 ceph-mon[103278]: from='client.92153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:19 smithi146 ceph-mon[103278]: from='client.92159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4155904241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/106391227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[91136]: from='client.92147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[91136]: from='client.92153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[91136]: from='client.92159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4155904241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:19.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/106391227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[99959]: from='client.92147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[99959]: from='client.92153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[99959]: from='client.92159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4155904241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/106391227' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:20 smithi146 ceph-mon[103278]: from='client.72255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:20 smithi146 ceph-mon[103278]: pgmap v12429: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:20 smithi078 ceph-mon[91136]: from='client.72255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:20 smithi078 ceph-mon[91136]: pgmap v12429: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:14:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:14:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:14:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:20 smithi078 ceph-mon[99959]: from='client.72255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:20 smithi078 ceph-mon[99959]: pgmap v12429: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:21.913Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:21.914Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:21.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:22.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:22.746 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:22 smithi078 ceph-mon[91136]: pgmap v12430: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:22.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:22.747 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:22 smithi078 ceph-mon[99959]: pgmap v12430: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:22 smithi146 ceph-mon[103278]: pgmap v12430: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:24 smithi146 ceph-mon[103278]: pgmap v12431: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:24 smithi078 ceph-mon[91136]: pgmap v12431: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:24 smithi078 ceph-mon[99959]: pgmap v12431: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 03:14:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T03:14:25.849763053Z level=info msg="Completed cleanup jobs" duration=14.248048ms 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[99959]: pgmap v12432: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[91136]: pgmap v12432: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.713 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.714 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:26 smithi146 ceph-mon[103278]: pgmap v12432: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:28.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:28 smithi146 ceph-mon[103278]: pgmap v12433: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:28.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:28 smithi078 ceph-mon[91136]: pgmap v12433: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:28.663 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:28 smithi078 ceph-mon[99959]: pgmap v12433: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:30.756 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:30 smithi146 ceph-mon[103278]: pgmap v12434: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:30 smithi078 ceph-mon[91136]: pgmap v12434: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:30.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:30 smithi078 ceph-mon[99959]: pgmap v12434: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:14:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:14:30] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:14:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:31 smithi146 ceph-mon[103278]: pgmap v12435: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[99959]: pgmap v12435: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-mon[91136]: pgmap v12435: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:31.914Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:31.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:31.915Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:34 smithi146 ceph-mon[103278]: pgmap v12436: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:34 smithi078 ceph-mon[91136]: pgmap v12436: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:34 smithi078 ceph-mon[99959]: pgmap v12436: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:36 smithi146 ceph-mon[103278]: pgmap v12437: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:36 smithi078 ceph-mon[91136]: pgmap v12437: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:36 smithi078 ceph-mon[99959]: pgmap v12437: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:38 smithi146 ceph-mon[103278]: pgmap v12438: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:38 smithi078 ceph-mon[91136]: pgmap v12438: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:38 smithi078 ceph-mon[99959]: pgmap v12438: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:40.744 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:40 smithi146 ceph-mon[103278]: pgmap v12439: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:40 smithi078 ceph-mon[91136]: pgmap v12439: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:14:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:14:40] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:14:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:40 smithi078 ceph-mon[99959]: pgmap v12439: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:41.915Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:41.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:41.916Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:42 smithi146 ceph-mon[103278]: pgmap v12440: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:42 smithi078 ceph-mon[91136]: pgmap v12440: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:42 smithi078 ceph-mon[99959]: pgmap v12440: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:44 smithi146 ceph-mon[103278]: pgmap v12441: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:44 smithi078 ceph-mon[99959]: pgmap v12441: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:44 smithi078 ceph-mon[91136]: pgmap v12441: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:46 smithi146 ceph-mon[103278]: pgmap v12442: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[99959]: pgmap v12442: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[91136]: pgmap v12442: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:48.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:48 smithi078 ceph-mon[91136]: pgmap v12443: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:48 smithi146 ceph-mon[103278]: pgmap v12443: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:48.670 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:48 smithi078 ceph-mon[99959]: pgmap v12443: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:49.388 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:14:49.919 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 7h - - 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 7h - - 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (6h) 4s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 19s ago 7h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 4s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 19s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 4s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 4s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 19s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 4s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 4s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 19s ago 7h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 4s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:14:49.920 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 4s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 4s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 4s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 19s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 19s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 19s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 19s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 19s ago 7h 71.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 4s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 4s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:14:49.921 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 19s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:14:50.242 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:14:50.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:14:50.244 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:14:50.244 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:14:50.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:14:50.244 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:50 smithi146 ceph-mon[103278]: pgmap v12444: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:50 smithi146 ceph-mon[103278]: from='client.92183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:50.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3751446502' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-mon[91136]: pgmap v12444: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-mon[91136]: from='client.92183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3751446502' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-mon[99959]: pgmap v12444: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:50.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-mon[99959]: from='client.92183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3751446502' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:14:50.537 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:14:50.538 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:14:50.538 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:14:50.865 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:14:50.866 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:50.866 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:14:50.866 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:50.866 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:14:50.866 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:14:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:14:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:14:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='client.72273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='client.92195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='client.92207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3206402221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='client.72273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='client.92195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='client.92207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3206402221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='client.72273 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='client.92195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='client.92207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3206402221' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:14:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:51.916Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:51.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:14:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:14:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:14:51.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:14:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:52 smithi146 ceph-mon[103278]: pgmap v12445: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:52 smithi078 ceph-mon[91136]: pgmap v12445: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:14:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:52 smithi078 ceph-mon[99959]: pgmap v12445: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:54 smithi146 ceph-mon[103278]: pgmap v12446: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:54 smithi078 ceph-mon[91136]: pgmap v12446: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:54 smithi078 ceph-mon[99959]: pgmap v12446: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:14:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:56 smithi146 ceph-mon[103278]: pgmap v12447: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:56 smithi078 ceph-mon[99959]: pgmap v12447: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:56 smithi078 ceph-mon[91136]: pgmap v12447: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:58 smithi146 ceph-mon[103278]: pgmap v12448: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:58 smithi078 ceph-mon[91136]: pgmap v12448: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:58 smithi078 ceph-mon[99959]: pgmap v12448: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:14:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:14:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:14:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:14:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:14:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:00 smithi146 ceph-mon[103278]: pgmap v12449: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:00 smithi078 ceph-mon[91136]: pgmap v12449: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:00 smithi078 ceph-mon[99959]: pgmap v12449: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:15:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:15:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:15:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:01.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:01.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:01.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:02 smithi146 ceph-mon[103278]: pgmap v12450: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:02 smithi078 ceph-mon[99959]: pgmap v12450: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:02 smithi078 ceph-mon[91136]: pgmap v12450: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:04 smithi146 ceph-mon[103278]: pgmap v12451: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:04 smithi078 ceph-mon[99959]: pgmap v12451: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:04 smithi078 ceph-mon[91136]: pgmap v12451: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:06 smithi146 ceph-mon[103278]: pgmap v12452: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[91136]: pgmap v12452: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[99959]: pgmap v12452: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:08 smithi146 ceph-mon[103278]: pgmap v12453: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:08 smithi078 ceph-mon[91136]: pgmap v12453: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:08 smithi078 ceph-mon[99959]: pgmap v12453: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:10.609 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:10 smithi078 ceph-mon[91136]: pgmap v12454: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:10.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:10 smithi078 ceph-mon[99959]: pgmap v12454: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:10 smithi146 ceph-mon[103278]: pgmap v12454: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:15:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:15:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:15:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:11 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:11 smithi146 ceph-mon[103278]: pgmap v12455: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[91136]: pgmap v12455: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-mon[99959]: pgmap v12455: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:11.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:11.917Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:11.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:14 smithi146 ceph-mon[103278]: pgmap v12456: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:14 smithi078 ceph-mon[91136]: pgmap v12456: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:14 smithi078 ceph-mon[99959]: pgmap v12456: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:16 smithi146 ceph-mon[103278]: pgmap v12457: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:16 smithi078 ceph-mon[91136]: pgmap v12457: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:16 smithi078 ceph-mon[99959]: pgmap v12457: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:18 smithi146 ceph-mon[103278]: pgmap v12458: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:18 smithi078 ceph-mon[91136]: pgmap v12458: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:18 smithi078 ceph-mon[99959]: pgmap v12458: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:20 smithi146 ceph-mon[103278]: pgmap v12459: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:20.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:20 smithi078 ceph-mon[91136]: pgmap v12459: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:20.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:20 smithi078 ceph-mon[99959]: pgmap v12459: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:15:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:15:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:15:21.170 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:15:21.698 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:15:21.698 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 7h - - 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 7h - - 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 15s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 10s ago 7h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 15s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 10s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 15s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 15s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 10s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 15s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 15s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 10s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 15s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 15s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 15s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:15:21.699 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 15s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 10s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 10s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 10s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 10s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 10s ago 7h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 15s ago 6h 89.4M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 15s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:15:21.700 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 10s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:15:22.008 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:15:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:15:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:15:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:15:22.010 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:15:22.010 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:21.917Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:22.010 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:21.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:22.010 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:21.918Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:15:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:15:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:15:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:15:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:15:22.276 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[91136]: from='client.72303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[91136]: pgmap v12460: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[91136]: from='client.72309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:22.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3222219549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[99959]: from='client.72303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[99959]: pgmap v12460: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[99959]: from='client.72309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:22.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3222219549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:22.587 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:15:22.588 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:22.588 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:22 smithi146 ceph-mon[103278]: from='client.72303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:22 smithi146 ceph-mon[103278]: pgmap v12460: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:22 smithi146 ceph-mon[103278]: from='client.72309 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3222219549' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:23 smithi146 ceph-mon[103278]: from='client.92231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:23 smithi146 ceph-mon[103278]: from='client.72327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3612855944' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:23 smithi078 ceph-mon[91136]: from='client.92231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:23 smithi078 ceph-mon[91136]: from='client.72327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3612855944' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:23 smithi078 ceph-mon[99959]: from='client.92231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:23 smithi078 ceph-mon[99959]: from='client.72327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3612855944' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:24 smithi146 ceph-mon[103278]: pgmap v12461: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:24 smithi078 ceph-mon[99959]: pgmap v12461: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:24 smithi078 ceph-mon[91136]: pgmap v12461: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:26 smithi146 ceph-mon[103278]: pgmap v12462: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:26 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[91136]: pgmap v12462: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[99959]: pgmap v12462: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:26 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:28.672 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:28 smithi078 ceph-mon[99959]: pgmap v12463: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:28.673 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:28 smithi078 ceph-mon[91136]: pgmap v12463: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:28 smithi146 ceph-mon[103278]: pgmap v12463: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:29.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:29.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:30.579 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:30 smithi078 ceph-mon[99959]: pgmap v12464: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:30.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:30 smithi078 ceph-mon[91136]: pgmap v12464: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:30 smithi146 ceph-mon[103278]: pgmap v12464: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:31.007 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:15:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:15:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:15:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[99959]: pgmap v12465: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:32 smithi078 ceph-mon[91136]: pgmap v12465: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:31.919Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:31.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:31.919Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:32 smithi146 ceph-mon[103278]: pgmap v12465: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:34 smithi146 ceph-mon[103278]: pgmap v12466: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:34 smithi078 ceph-mon[91136]: pgmap v12466: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:34 smithi078 ceph-mon[99959]: pgmap v12466: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:36 smithi146 ceph-mon[103278]: pgmap v12467: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:36 smithi078 ceph-mon[91136]: pgmap v12467: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:36 smithi078 ceph-mon[99959]: pgmap v12467: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:38 smithi146 ceph-mon[103278]: pgmap v12468: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:38 smithi078 ceph-mon[99959]: pgmap v12468: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:38 smithi078 ceph-mon[91136]: pgmap v12468: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:40 smithi146 ceph-mon[103278]: pgmap v12469: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:40 smithi078 ceph-mon[91136]: pgmap v12469: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:40 smithi078 ceph-mon[99959]: pgmap v12469: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:40.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:15:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:15:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:15:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:41.920Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:41.920Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:41.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:42 smithi146 ceph-mon[103278]: pgmap v12470: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:42 smithi078 ceph-mon[99959]: pgmap v12470: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:42 smithi078 ceph-mon[91136]: pgmap v12470: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:44 smithi146 ceph-mon[103278]: pgmap v12471: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:44 smithi078 ceph-mon[99959]: pgmap v12471: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:44 smithi078 ceph-mon[91136]: pgmap v12471: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:46 smithi146 ceph-mon[103278]: pgmap v12472: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:46 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[91136]: pgmap v12472: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[99959]: pgmap v12472: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:46 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:48 smithi146 ceph-mon[103278]: pgmap v12473: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:48 smithi078 ceph-mon[91136]: pgmap v12473: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:48 smithi078 ceph-mon[99959]: pgmap v12473: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:50.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:50 smithi146 ceph-mon[103278]: pgmap v12474: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:50.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:50 smithi078 ceph-mon[91136]: pgmap v12474: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:50.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:50 smithi078 ceph-mon[99959]: pgmap v12474: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:50.693 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:15:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:15:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:15:52.224 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:51.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:52.224 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:51.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:15:52.224 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:15:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:15:51.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:15:52.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:52.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.383 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:52 smithi146 ceph-mon[103278]: pgmap v12475: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[91136]: pgmap v12475: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:15:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:15:52.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:52 smithi078 ceph-mon[99959]: pgmap v12475: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 7h - - 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 7h - - 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 7s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 2s ago 7h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 7s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 7s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 7s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 7s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 7s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 2s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 7s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 7s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 7s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 7s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 7h 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 7s ago 6h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 7s ago 6h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 2s ago 6h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:15:53.741 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:15:53.741 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:15:53.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:15:53.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:15:53.742 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:15:54.022 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:15:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:15:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:15:54.023 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:15:54.340 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:15:54.341 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:15:54.341 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:15:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:54 smithi146 ceph-mon[103278]: from='client.92255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:54 smithi146 ceph-mon[103278]: from='client.72345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:54 smithi146 ceph-mon[103278]: pgmap v12476: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:54.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:54 smithi146 ceph-mon[103278]: from='client.72351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/248842972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:54.772 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/324959923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[91136]: from='client.92255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[91136]: from='client.72345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[91136]: pgmap v12476: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[91136]: from='client.72351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/248842972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:54.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/324959923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[99959]: from='client.92255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[99959]: from='client.72345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[99959]: pgmap v12476: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:15:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[99959]: from='client.72351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/248842972' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:15:54.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/324959923' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:15:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:55 smithi146 ceph-mon[103278]: from='client.72363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:55.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:55 smithi078 ceph-mon[99959]: from='client.72363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:55.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:55 smithi078 ceph-mon[91136]: from='client.72363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:15:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:56 smithi146 ceph-mon[103278]: pgmap v12477: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:56 smithi078 ceph-mon[91136]: pgmap v12477: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:56 smithi078 ceph-mon[99959]: pgmap v12477: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:15:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:58 smithi146 ceph-mon[103278]: pgmap v12478: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:58 smithi078 ceph-mon[91136]: pgmap v12478: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:58 smithi078 ceph-mon[99959]: pgmap v12478: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:15:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:15:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:15:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:15:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:15:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:00 smithi146 ceph-mon[103278]: pgmap v12479: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:00 smithi078 ceph-mon[91136]: pgmap v12479: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:16:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:16:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:16:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:00 smithi078 ceph-mon[99959]: pgmap v12479: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:01.921Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:01.921Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:01.922Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:02 smithi146 ceph-mon[103278]: pgmap v12480: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:02 smithi078 ceph-mon[91136]: pgmap v12480: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:02 smithi078 ceph-mon[99959]: pgmap v12480: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:04 smithi146 ceph-mon[103278]: pgmap v12481: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:04 smithi078 ceph-mon[91136]: pgmap v12481: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:04 smithi078 ceph-mon[99959]: pgmap v12481: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:06 smithi146 ceph-mon[103278]: pgmap v12482: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[91136]: pgmap v12482: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[99959]: pgmap v12482: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:08 smithi146 ceph-mon[103278]: pgmap v12483: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:08 smithi078 ceph-mon[91136]: pgmap v12483: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:08.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:08 smithi078 ceph-mon[99959]: pgmap v12483: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:10 smithi078 ceph-mon[91136]: pgmap v12484: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:10 smithi078 ceph-mon[99959]: pgmap v12484: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:10 smithi146 ceph-mon[103278]: pgmap v12484: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:11.028 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:16:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:16:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:16:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:11.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:11.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:11.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:12 smithi146 ceph-mon[103278]: pgmap v12485: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:12.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:12.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.581 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[99959]: pgmap v12485: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:12.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:12.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:12.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:12 smithi078 ceph-mon[91136]: pgmap v12485: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:14 smithi146 ceph-mon[103278]: pgmap v12486: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:14 smithi078 ceph-mon[91136]: pgmap v12486: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:14.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:14.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:14 smithi078 ceph-mon[99959]: pgmap v12486: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:16 smithi146 ceph-mon[103278]: pgmap v12487: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:16 smithi078 ceph-mon[99959]: pgmap v12487: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:16 smithi078 ceph-mon[91136]: pgmap v12487: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:18 smithi146 ceph-mon[103278]: pgmap v12488: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:18 smithi078 ceph-mon[99959]: pgmap v12488: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:18 smithi078 ceph-mon[91136]: pgmap v12488: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:20 smithi146 ceph-mon[103278]: pgmap v12489: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:20 smithi078 ceph-mon[99959]: pgmap v12489: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:20 smithi078 ceph-mon[91136]: pgmap v12489: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:16:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:16:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:16:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:21.922Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:21.923Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:21.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:22 smithi146 ceph-mon[103278]: pgmap v12490: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:22 smithi078 ceph-mon[91136]: pgmap v12490: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:22 smithi078 ceph-mon[99959]: pgmap v12490: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:24.633 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:16:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:24 smithi146 ceph-mon[103278]: pgmap v12491: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:24 smithi078 ceph-mon[99959]: pgmap v12491: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:24 smithi078 ceph-mon[91136]: pgmap v12491: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 7h - - 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 7h - - 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 18s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 13s ago 7h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 18s ago 6h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 13s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 18s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 18s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 13s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 18s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:16:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 18s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 13s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 18s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 18s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 18s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 13s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 13s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 13s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 13s ago 7h 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:16:25.155 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:16:25.464 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:16:25.464 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:16:25.464 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:16:25.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:25.464 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:16:25.464 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:16:25.465 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:16:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:16:25.731 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:16:25.731 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:16:25.731 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:16:25.731 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:16:25.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:16:26.042 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:16:26.042 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:26.042 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:26 smithi146 ceph-mon[103278]: from='client.72375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:26 smithi146 ceph-mon[103278]: from='client.92297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:26 smithi146 ceph-mon[103278]: from='client.72387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:26 smithi146 ceph-mon[103278]: pgmap v12492: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1204922086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:26.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/256518006' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[91136]: from='client.72375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[91136]: from='client.92297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[91136]: from='client.72387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[91136]: pgmap v12492: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1204922086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:26.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/256518006' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:26.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[99959]: from='client.72375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[99959]: from='client.92297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[99959]: from='client.72387 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:26.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[99959]: pgmap v12492: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:26.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1204922086' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:26.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/256518006' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[99959]: from='client.92315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:27.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[91136]: from='client.92315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:27 smithi146 ceph-mon[103278]: from='client.92315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:28 smithi146 ceph-mon[103278]: pgmap v12493: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:28 smithi078 ceph-mon[99959]: pgmap v12493: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:28 smithi078 ceph-mon[91136]: pgmap v12493: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:30.833 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:30 smithi078 ceph-mon[99959]: pgmap v12494: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:30.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:30 smithi078 ceph-mon[91136]: pgmap v12494: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:30.833 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:16:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:16:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:16:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:30 smithi146 ceph-mon[103278]: pgmap v12494: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:32.015 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:31.923Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:32.015 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:31.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:32.015 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:31.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:32.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:32.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[99959]: pgmap v12495: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:32.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:32.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[91136]: pgmap v12495: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:32.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:32.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:32 smithi146 ceph-mon[103278]: pgmap v12495: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:32.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:32.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:34 smithi146 ceph-mon[103278]: pgmap v12496: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:34 smithi078 ceph-mon[91136]: pgmap v12496: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:34 smithi078 ceph-mon[99959]: pgmap v12496: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:36 smithi146 ceph-mon[103278]: pgmap v12497: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:36 smithi078 ceph-mon[91136]: pgmap v12497: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:36 smithi078 ceph-mon[99959]: pgmap v12497: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:38 smithi146 ceph-mon[103278]: pgmap v12498: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:38 smithi078 ceph-mon[91136]: pgmap v12498: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:38 smithi078 ceph-mon[99959]: pgmap v12498: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:40 smithi146 ceph-mon[103278]: pgmap v12499: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:40 smithi078 ceph-mon[91136]: pgmap v12499: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:40 smithi078 ceph-mon[99959]: pgmap v12499: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:16:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:16:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:16:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:41.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:41.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:41.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:42 smithi146 ceph-mon[103278]: pgmap v12500: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:42 smithi078 ceph-mon[91136]: pgmap v12500: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:42 smithi078 ceph-mon[99959]: pgmap v12500: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:16:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:44 smithi146 ceph-mon[103278]: pgmap v12501: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:44 smithi078 ceph-mon[91136]: pgmap v12501: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:44 smithi078 ceph-mon[99959]: pgmap v12501: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:46 smithi146 ceph-mon[103278]: pgmap v12502: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:46 smithi078 ceph-mon[91136]: pgmap v12502: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:46 smithi078 ceph-mon[99959]: pgmap v12502: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:47 smithi146 ceph-mon[103278]: pgmap v12503: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[91136]: pgmap v12503: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:47 smithi078 ceph-mon[99959]: pgmap v12503: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:50.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:50 smithi078 ceph-mon[91136]: pgmap v12504: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:50.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:50 smithi078 ceph-mon[99959]: pgmap v12504: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:50.699 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:16:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:16:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:16:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:50 smithi146 ceph-mon[103278]: pgmap v12504: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:51.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:51.924Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:16:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:16:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:16:51.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:16:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:52 smithi146 ceph-mon[103278]: pgmap v12505: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[99959]: pgmap v12505: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[91136]: pgmap v12505: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:52.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:16:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:54 smithi078 ceph-mon[91136]: pgmap v12506: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:54 smithi078 ceph-mon[99959]: pgmap v12506: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:54.755 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:54 smithi146 ceph-mon[103278]: pgmap v12506: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:16:56.334 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:16:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:56 smithi146 ceph-mon[103278]: pgmap v12507: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:56 smithi078 ceph-mon[99959]: pgmap v12507: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:56 smithi078 ceph-mon[91136]: pgmap v12507: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:56.860 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 7h - - 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 7h - - 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 10s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 5s ago 7h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 10s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 5s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 10s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 10s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 5s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 10s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 10s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 5s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:16:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 10s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 10s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 10s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 10s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 5s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 5s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 5s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 5s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 5s ago 7h 72.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 10s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 10s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:16:56.862 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:16:57.168 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:16:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:16:57.170 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:16:57.437 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:16:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:16:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:16:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:16:57.437 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:16:57.438 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:16:57.743 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:16:57.743 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:16:57.743 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:16:57.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:57 smithi078 ceph-mon[99959]: from='client.92327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:57.744 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1447904920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:57.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:57 smithi078 ceph-mon[91136]: from='client.92327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:57.744 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1447904920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:57 smithi146 ceph-mon[103278]: from='client.92327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1447904920' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:58 smithi146 ceph-mon[103278]: from='client.72417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:58 smithi146 ceph-mon[103278]: from='client.72423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:58 smithi146 ceph-mon[103278]: pgmap v12508: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:58 smithi146 ceph-mon[103278]: from='client.72435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/359002009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[99959]: from='client.72417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[99959]: from='client.72423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[99959]: pgmap v12508: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[99959]: from='client.72435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/359002009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[91136]: from='client.72417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[91136]: from='client.72423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[91136]: pgmap v12508: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:16:58.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[91136]: from='client.72435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:16:58.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/359002009' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:16:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:16:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:16:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:16:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:16:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:00 smithi146 ceph-mon[103278]: pgmap v12509: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:00 smithi078 ceph-mon[91136]: pgmap v12509: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:00 smithi078 ceph-mon[99959]: pgmap v12509: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:17:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:17:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:17:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:01.924Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:01.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:01.925Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:02 smithi146 ceph-mon[103278]: pgmap v12510: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:02 smithi078 ceph-mon[99959]: pgmap v12510: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:02 smithi078 ceph-mon[91136]: pgmap v12510: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:04 smithi146 ceph-mon[103278]: pgmap v12511: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:04 smithi078 ceph-mon[91136]: pgmap v12511: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:04 smithi078 ceph-mon[99959]: pgmap v12511: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:06 smithi146 ceph-mon[103278]: pgmap v12512: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:06 smithi078 ceph-mon[91136]: pgmap v12512: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:06 smithi078 ceph-mon[99959]: pgmap v12512: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:07 smithi146 ceph-mon[103278]: pgmap v12513: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[99959]: pgmap v12513: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:07 smithi078 ceph-mon[91136]: pgmap v12513: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:10.718 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:10 smithi078 ceph-mon[99959]: pgmap v12514: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:10.718 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:10 smithi078 ceph-mon[91136]: pgmap v12514: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:10.719 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:17:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:17:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:17:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:10 smithi146 ceph-mon[103278]: pgmap v12514: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:11.925Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:11.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:11.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:12.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:12.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:12 smithi146 ceph-mon[103278]: pgmap v12515: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:12.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.654 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:12.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[99959]: pgmap v12515: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[91136]: pgmap v12515: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:12.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:14 smithi078 ceph-mon[91136]: pgmap v12516: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:14 smithi078 ceph-mon[99959]: pgmap v12516: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:14.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:14 smithi146 ceph-mon[103278]: pgmap v12516: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:14.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:16 smithi146 ceph-mon[103278]: pgmap v12517: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:16 smithi078 ceph-mon[91136]: pgmap v12517: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:16 smithi078 ceph-mon[99959]: pgmap v12517: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:18 smithi146 ceph-mon[103278]: pgmap v12518: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:18 smithi078 ceph-mon[91136]: pgmap v12518: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:18 smithi078 ceph-mon[99959]: pgmap v12518: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:20.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:20 smithi078 ceph-mon[91136]: pgmap v12519: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:20.724 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:20 smithi078 ceph-mon[99959]: pgmap v12519: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:20.724 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:17:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:17:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:17:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:20 smithi146 ceph-mon[103278]: pgmap v12519: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:21.926Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:21.926Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:21.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:22 smithi146 ceph-mon[103278]: pgmap v12520: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:22 smithi078 ceph-mon[91136]: pgmap v12520: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:22 smithi078 ceph-mon[99959]: pgmap v12520: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:24 smithi146 ceph-mon[103278]: pgmap v12521: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:24 smithi078 ceph-mon[99959]: pgmap v12521: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:24 smithi078 ceph-mon[91136]: pgmap v12521: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:26 smithi146 ceph-mon[103278]: pgmap v12522: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:26 smithi078 ceph-mon[99959]: pgmap v12522: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:26 smithi078 ceph-mon[91136]: pgmap v12522: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:28.040 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:17:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:27 smithi146 ceph-mon[103278]: pgmap v12523: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[91136]: pgmap v12523: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:27 smithi078 ceph-mon[99959]: pgmap v12523: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:28.564 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 7h - - 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 7h - - 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 1s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 16s ago 7h 89.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 1s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 16s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 1s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 1s ago 7h 557M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 16s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 1s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 1s ago 7h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:17:28.565 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 16s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 1s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 1s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 1s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 1s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 16s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 16s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 16s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 16s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 16s ago 7h 67.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 1s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 1s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:17:28.566 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:17:28.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:17:28.875 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:17:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:17:28.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:17:28.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:17:29.146 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:17:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:17:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:17:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:17:29.147 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:17:29.147 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:17:29.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[99959]: from='client.72447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[99959]: from='client.92369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[99959]: from='client.72459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:29.147 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2878496793' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:29.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[91136]: from='client.72447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[91136]: from='client.92369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[91136]: from='client.72459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:29.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2878496793' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:28 smithi146 ceph-mon[103278]: from='client.72447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:28 smithi146 ceph-mon[103278]: from='client.92369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:28 smithi146 ceph-mon[103278]: from='client.72459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2878496793' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:17:29.459 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:17:29.459 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:17:29.459 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:17:29.459 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:17:29.459 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:17:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:29 smithi146 ceph-mon[103278]: from='client.92387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:29 smithi146 ceph-mon[103278]: pgmap v12524: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/990821602' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:29 smithi078 ceph-mon[91136]: from='client.92387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:29 smithi078 ceph-mon[91136]: pgmap v12524: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/990821602' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:29 smithi078 ceph-mon[99959]: from='client.92387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:17:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:29 smithi078 ceph-mon[99959]: pgmap v12524: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/990821602' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:17:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:17:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:17:30] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:17:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:31.927Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:31.927Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:31.928Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:32 smithi146 ceph-mon[103278]: pgmap v12525: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[91136]: pgmap v12525: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[99959]: pgmap v12525: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:34.677 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:34 smithi078 ceph-mon[99959]: pgmap v12526: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:34.677 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:34 smithi078 ceph-mon[91136]: pgmap v12526: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:34 smithi146 ceph-mon[103278]: pgmap v12526: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:36 smithi146 ceph-mon[103278]: pgmap v12527: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:36 smithi078 ceph-mon[91136]: pgmap v12527: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:36 smithi078 ceph-mon[99959]: pgmap v12527: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:38 smithi146 ceph-mon[103278]: pgmap v12528: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:38 smithi078 ceph-mon[99959]: pgmap v12528: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:38 smithi078 ceph-mon[91136]: pgmap v12528: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:40.745 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:40 smithi078 ceph-mon[91136]: pgmap v12529: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:40.745 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:40 smithi078 ceph-mon[99959]: pgmap v12529: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:40.745 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:17:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:17:40] "GET /metrics HTTP/1.1" 200 39000 "" "Prometheus/2.43.0" 2024-09-21T03:17:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:40 smithi146 ceph-mon[103278]: pgmap v12529: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:41.928Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:41.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:41.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:42 smithi146 ceph-mon[103278]: pgmap v12530: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:42 smithi078 ceph-mon[99959]: pgmap v12530: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:42 smithi078 ceph-mon[91136]: pgmap v12530: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:44 smithi146 ceph-mon[103278]: pgmap v12531: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:44 smithi078 ceph-mon[91136]: pgmap v12531: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:44 smithi078 ceph-mon[99959]: pgmap v12531: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:46.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:46 smithi146 ceph-mon[103278]: pgmap v12532: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:46 smithi078 ceph-mon[91136]: pgmap v12532: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:46.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:46 smithi078 ceph-mon[99959]: pgmap v12532: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:48 smithi146 ceph-mon[103278]: pgmap v12533: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[91136]: pgmap v12533: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:48.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:48.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:48 smithi078 ceph-mon[99959]: pgmap v12533: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:50.748 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:50 smithi078 ceph-mon[99959]: pgmap v12534: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:50.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:50 smithi078 ceph-mon[91136]: pgmap v12534: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:50.749 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:17:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:17:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:17:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:50 smithi146 ceph-mon[103278]: pgmap v12534: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:51.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:51.929Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:17:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:17:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:17:51.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:17:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:52 smithi146 ceph-mon[103278]: pgmap v12535: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[99959]: pgmap v12535: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[91136]: pgmap v12535: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:17:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:54 smithi146 ceph-mon[103278]: pgmap v12536: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:54 smithi078 ceph-mon[91136]: pgmap v12536: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:54 smithi078 ceph-mon[99959]: pgmap v12536: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:17:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:56 smithi146 ceph-mon[103278]: pgmap v12537: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:56 smithi078 ceph-mon[99959]: pgmap v12537: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:56 smithi078 ceph-mon[91136]: pgmap v12537: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:58 smithi146 ceph-mon[103278]: pgmap v12538: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:58 smithi078 ceph-mon[99959]: pgmap v12538: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:58 smithi078 ceph-mon[91136]: pgmap v12538: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:17:59.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:17:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:59.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:17:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:17:59.753 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:17:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:17:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:18:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 7h - - 2024-09-21T03:18:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 7h - - 2024-09-21T03:18:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 13s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 8s ago 7h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 13s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 8s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 13s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 13s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 8s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 13s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 13s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 8s ago 7h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 13s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 13s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 13s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 13s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:18:00.279 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 8s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 8s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 8s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 8s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 8s ago 7h 68.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 13s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 13s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:18:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:18:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:00 smithi078 ceph-mon[99959]: pgmap v12539: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:00 smithi078 ceph-mon[91136]: pgmap v12539: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:18:00.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:18:00.589 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:18:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:00 smithi146 ceph-mon[103278]: pgmap v12539: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:18:00.857 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:18:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:18:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:18:00.858 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:18:00.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:18:00.858 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:18:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:18:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:18:01.165 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:18:01.165 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:01.165 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:01 smithi146 ceph-mon[103278]: from='client.92399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:01 smithi146 ceph-mon[103278]: from='client.92405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:01 smithi146 ceph-mon[103278]: from='client.72495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3641685619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:01 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1411250895' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[91136]: from='client.92399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[91136]: from='client.92405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[91136]: from='client.72495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3641685619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:01.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1411250895' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[99959]: from='client.92399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[99959]: from='client.92405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[99959]: from='client.72495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3641685619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1411250895' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:01.929Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:01.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:01.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:02 smithi146 ceph-mon[103278]: from='client.72507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:02 smithi146 ceph-mon[103278]: pgmap v12540: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:02 smithi078 ceph-mon[99959]: from='client.72507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:02 smithi078 ceph-mon[99959]: pgmap v12540: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:02 smithi078 ceph-mon[91136]: from='client.72507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:02.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:02 smithi078 ceph-mon[91136]: pgmap v12540: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:04 smithi146 ceph-mon[103278]: pgmap v12541: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:04 smithi078 ceph-mon[91136]: pgmap v12541: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:04.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:04 smithi078 ceph-mon[99959]: pgmap v12541: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:06 smithi146 ceph-mon[103278]: pgmap v12542: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:06 smithi078 ceph-mon[91136]: pgmap v12542: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:06 smithi078 ceph-mon[99959]: pgmap v12542: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:08 smithi146 ceph-mon[103278]: pgmap v12543: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[91136]: pgmap v12543: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:08.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:08 smithi078 ceph-mon[99959]: pgmap v12543: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:10.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:10 smithi078 ceph-mon[91136]: pgmap v12544: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:10.761 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:10 smithi078 ceph-mon[99959]: pgmap v12544: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:10.761 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:18:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:18:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:18:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:10 smithi146 ceph-mon[103278]: pgmap v12544: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:11 smithi146 ceph-mon[103278]: pgmap v12545: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-mon[99959]: pgmap v12545: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-mon[91136]: pgmap v12545: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:11.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:11.930Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:11.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:14 smithi146 ceph-mon[103278]: pgmap v12546: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:14.438 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:14.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:14 smithi078 ceph-mon[91136]: pgmap v12546: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:14.478 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:14.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:14 smithi078 ceph-mon[99959]: pgmap v12546: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:14.478 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:16.613 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:16 smithi146 ceph-mon[103278]: pgmap v12547: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:16 smithi078 ceph-mon[91136]: pgmap v12547: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:16 smithi078 ceph-mon[99959]: pgmap v12547: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:18 smithi146 ceph-mon[103278]: pgmap v12548: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:18 smithi078 ceph-mon[91136]: pgmap v12548: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:18 smithi078 ceph-mon[99959]: pgmap v12548: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:20 smithi146 ceph-mon[103278]: pgmap v12549: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:20.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:20 smithi078 ceph-mon[91136]: pgmap v12549: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:20.770 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:20 smithi078 ceph-mon[99959]: pgmap v12549: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:20.771 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:18:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:18:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:18:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:21.930Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:21.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:21.931Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:22 smithi146 ceph-mon[103278]: pgmap v12550: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:22 smithi078 ceph-mon[91136]: pgmap v12550: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:22.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:22 smithi078 ceph-mon[99959]: pgmap v12550: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:24 smithi146 ceph-mon[103278]: pgmap v12551: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:24 smithi078 ceph-mon[91136]: pgmap v12551: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:24 smithi078 ceph-mon[99959]: pgmap v12551: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:26 smithi146 ceph-mon[103278]: pgmap v12552: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:26 smithi078 ceph-mon[91136]: pgmap v12552: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:26.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:26 smithi078 ceph-mon[99959]: pgmap v12552: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:28 smithi146 ceph-mon[103278]: pgmap v12553: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[91136]: pgmap v12553: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:28 smithi078 ceph-mon[99959]: pgmap v12553: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:29.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:30.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:30 smithi078 ceph-mon[99959]: pgmap v12554: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:30.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:30 smithi078 ceph-mon[91136]: pgmap v12554: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:30 smithi146 ceph-mon[103278]: pgmap v12554: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:18:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:18:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:18:31.458 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 7h - - 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 7h - - 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 4s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 19s ago 7h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 4s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 19s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 4s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 4s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 19s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 4s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:18:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 4s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 19s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 4s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 4s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 4s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 4s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 19s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 19s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 19s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 19s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 19s ago 7h 68.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 4s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 4s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:18:31.990 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:18:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:31.931Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:31.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:31.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:32.305 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:18:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:18:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:18:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:18:32.306 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:18:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: pgmap v12555: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: from='client.72519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3174516564' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:18:32.575 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:18:32.576 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:18:32.576 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: pgmap v12555: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: from='client.72519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3174516564' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:32.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:32.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: pgmap v12555: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:32.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: from='client.72519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:32.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:32.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3174516564' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:18:32.889 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:18:32.889 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:18:32.889 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:32.889 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:33 smithi146 ceph-mon[103278]: from='client.92441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:33 smithi146 ceph-mon[103278]: from='client.92447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:33 smithi146 ceph-mon[103278]: from='client.72543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1708199871' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[99959]: from='client.92441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[99959]: from='client.92447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[99959]: from='client.72543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1708199871' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[91136]: from='client.92441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[91136]: from='client.92447 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[91136]: from='client.72543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:18:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1708199871' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:18:34.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:34 smithi146 ceph-mon[103278]: pgmap v12556: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:34 smithi078 ceph-mon[91136]: pgmap v12556: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:34 smithi078 ceph-mon[99959]: pgmap v12556: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:36.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:36 smithi078 ceph-mon[91136]: pgmap v12557: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:36.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:36 smithi078 ceph-mon[99959]: pgmap v12557: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:36 smithi146 ceph-mon[103278]: pgmap v12557: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:38 smithi146 ceph-mon[103278]: pgmap v12558: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:38 smithi078 ceph-mon[91136]: pgmap v12558: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:38 smithi078 ceph-mon[99959]: pgmap v12558: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:40 smithi146 ceph-mon[103278]: pgmap v12559: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:40 smithi078 ceph-mon[91136]: pgmap v12559: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:40.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:40 smithi078 ceph-mon[99959]: pgmap v12559: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:18:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:18:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:18:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:41.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:41.932Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:41.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:42 smithi146 ceph-mon[103278]: pgmap v12560: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:42 smithi078 ceph-mon[91136]: pgmap v12560: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:42 smithi078 ceph-mon[99959]: pgmap v12560: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:44 smithi146 ceph-mon[103278]: pgmap v12561: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:44 smithi078 ceph-mon[91136]: pgmap v12561: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:44 smithi078 ceph-mon[99959]: pgmap v12561: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:46.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:46 smithi146 ceph-mon[103278]: pgmap v12562: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:46.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:46 smithi078 ceph-mon[91136]: pgmap v12562: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:46 smithi078 ceph-mon[99959]: pgmap v12562: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:48 smithi146 ceph-mon[103278]: pgmap v12563: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[99959]: pgmap v12563: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[91136]: pgmap v12563: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:50 smithi146 ceph-mon[103278]: pgmap v12564: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:50.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:50 smithi078 ceph-mon[99959]: pgmap v12564: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:50.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:50 smithi078 ceph-mon[91136]: pgmap v12564: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:18:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:18:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:18:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:51.932Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:51.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:18:52.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:18:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:18:51.933Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:18:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:52 smithi146 ceph-mon[103278]: pgmap v12565: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[91136]: pgmap v12565: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[99959]: pgmap v12565: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:18:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:54 smithi146 ceph-mon[103278]: pgmap v12566: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:54 smithi078 ceph-mon[91136]: pgmap v12566: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:54 smithi078 ceph-mon[99959]: pgmap v12566: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:18:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:56 smithi078 ceph-mon[91136]: pgmap v12567: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:56 smithi078 ceph-mon[99959]: pgmap v12567: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:56.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:56 smithi146 ceph-mon[103278]: pgmap v12567: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:58.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:58 smithi078 ceph-mon[99959]: pgmap v12568: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:58.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:58 smithi078 ceph-mon[91136]: pgmap v12568: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:58 smithi146 ceph-mon[103278]: pgmap v12568: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:18:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:18:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:18:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:18:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:18:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:00 smithi146 ceph-mon[103278]: pgmap v12569: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:00.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:00 smithi078 ceph-mon[99959]: pgmap v12569: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:00 smithi078 ceph-mon[91136]: pgmap v12569: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:19:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:19:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:19:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:01.933Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:01.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:01.934Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:02 smithi146 ceph-mon[103278]: pgmap v12570: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:02 smithi078 ceph-mon[91136]: pgmap v12570: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:02 smithi078 ceph-mon[99959]: pgmap v12570: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:03.186 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 7h - - 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 7h - - 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 16s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 11s ago 7h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 16s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:19:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 11s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 16s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 16s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 11s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 16s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 16s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 11s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 16s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 16s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 16s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 16s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 11s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 11s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 11s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 11s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:19:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 11s ago 7h 68.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:19:03.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 16s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:19:03.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:19:03.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 16s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:19:03.716 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:19:04.024 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:19:04.024 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:19:04.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:19:04.026 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:19:04.290 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:19:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:19:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:19:04.291 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:19:04.602 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:19:04.602 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:19:04.602 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:19:04.602 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:04.602 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[99959]: from='client.92471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[99959]: pgmap v12571: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[99959]: from='client.72561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2888949753' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[91136]: from='client.92471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[91136]: pgmap v12571: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[91136]: from='client.72561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:04.603 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2888949753' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:04 smithi146 ceph-mon[103278]: from='client.92471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:04 smithi146 ceph-mon[103278]: pgmap v12571: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:04 smithi146 ceph-mon[103278]: from='client.72561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2888949753' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:05 smithi146 ceph-mon[103278]: from='client.72567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:05 smithi146 ceph-mon[103278]: from='client.72579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1523126154' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:05 smithi078 ceph-mon[91136]: from='client.72567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:05 smithi078 ceph-mon[91136]: from='client.72579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1523126154' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:05.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:05 smithi078 ceph-mon[99959]: from='client.72567 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:05 smithi078 ceph-mon[99959]: from='client.72579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1523126154' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:06 smithi146 ceph-mon[103278]: pgmap v12572: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:06 smithi078 ceph-mon[91136]: pgmap v12572: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:06 smithi078 ceph-mon[99959]: pgmap v12572: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[99959]: pgmap v12573: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[91136]: pgmap v12573: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:08.935 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:08 smithi146 ceph-mon[103278]: pgmap v12573: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:10 smithi146 ceph-mon[103278]: pgmap v12574: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:11.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:10 smithi078 ceph-mon[91136]: pgmap v12574: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:10 smithi078 ceph-mon[99959]: pgmap v12574: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:19:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:19:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:19:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:11.934Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:11.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:11.935Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:12 smithi146 ceph-mon[103278]: pgmap v12575: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[99959]: pgmap v12575: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[91136]: pgmap v12575: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:14 smithi146 ceph-mon[103278]: pgmap v12576: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:14 smithi078 ceph-mon[91136]: pgmap v12576: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:14 smithi078 ceph-mon[99959]: pgmap v12576: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:16.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:16 smithi146 ceph-mon[103278]: pgmap v12577: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:16.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:16 smithi078 ceph-mon[99959]: pgmap v12577: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:16.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:16 smithi078 ceph-mon[91136]: pgmap v12577: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:18.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:18 smithi146 ceph-mon[103278]: pgmap v12578: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:18 smithi078 ceph-mon[99959]: pgmap v12578: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:18 smithi078 ceph-mon[91136]: pgmap v12578: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:20 smithi146 ceph-mon[103278]: pgmap v12579: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:20 smithi078 ceph-mon[99959]: pgmap v12579: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:20 smithi078 ceph-mon[91136]: pgmap v12579: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:19:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:19:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:19:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:21.935Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:21.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:21.936Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:22 smithi146 ceph-mon[103278]: pgmap v12580: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:22 smithi078 ceph-mon[91136]: pgmap v12580: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:22 smithi078 ceph-mon[99959]: pgmap v12580: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:24 smithi146 ceph-mon[103278]: pgmap v12581: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:24 smithi078 ceph-mon[91136]: pgmap v12581: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:24 smithi078 ceph-mon[99959]: pgmap v12581: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:25 smithi146 ceph-mon[103278]: pgmap v12582: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:25 smithi078 ceph-mon[99959]: pgmap v12582: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:25 smithi078 ceph-mon[91136]: pgmap v12582: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:28 smithi146 ceph-mon[103278]: pgmap v12583: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[91136]: pgmap v12583: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[99959]: pgmap v12583: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:30 smithi146 ceph-mon[103278]: pgmap v12584: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:30.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:30 smithi078 ceph-mon[91136]: pgmap v12584: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:30.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:30 smithi078 ceph-mon[99959]: pgmap v12584: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:19:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:19:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:19:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:31.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:31.937Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:31.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:32 smithi146 ceph-mon[103278]: pgmap v12585: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:32 smithi078 ceph-mon[91136]: pgmap v12585: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:32.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:32 smithi078 ceph-mon[99959]: pgmap v12585: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:33 smithi146 ceph-mon[103278]: pgmap v12586: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[91136]: pgmap v12586: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:34.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:33 smithi078 ceph-mon[99959]: pgmap v12586: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:34.895 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 7h - - 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 7h - - 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 7s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 2s ago 7h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 7s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 2s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 7s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:19:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 7s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 2s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 7s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 7s ago 7h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 2s ago 7h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 7s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 7s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 7s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 7s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 2s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 2s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 2s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 2s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 2s ago 7h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:19:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 7s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:19:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:19:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 7s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:19:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:19:35.730 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:35.731 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:19:35.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:19:35.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:19:35.732 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:19:35.732 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:19:35.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:19:35.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:19:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:19:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:19:36.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:19:36.309 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:19:36.309 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:19:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:19:36.309 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:36.309 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:19:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:36 smithi146 ceph-mon[103278]: from='client.72591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:36 smithi146 ceph-mon[103278]: from='client.92513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:36 smithi146 ceph-mon[103278]: from='client.72603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:36 smithi146 ceph-mon[103278]: pgmap v12587: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4292617859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/302637964' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[91136]: from='client.72591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[91136]: from='client.92513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[91136]: from='client.72603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[91136]: pgmap v12587: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4292617859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/302637964' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[99959]: from='client.72591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[99959]: from='client.92513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[99959]: from='client.72603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[99959]: pgmap v12587: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4292617859' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:19:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/302637964' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:19:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:37 smithi146 ceph-mon[103278]: from='client.72615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:37 smithi078 ceph-mon[91136]: from='client.72615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:37 smithi078 ceph-mon[99959]: from='client.72615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:19:38.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:38 smithi078 ceph-mon[91136]: pgmap v12588: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:38.644 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:38 smithi078 ceph-mon[99959]: pgmap v12588: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:38 smithi146 ceph-mon[103278]: pgmap v12588: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:40 smithi146 ceph-mon[103278]: pgmap v12589: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:40.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:40 smithi078 ceph-mon[91136]: pgmap v12589: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:40.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:19:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:19:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:19:40.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:40 smithi078 ceph-mon[99959]: pgmap v12589: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:41.937Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:41.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:41.938Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:42 smithi078 ceph-mon[99959]: pgmap v12590: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:42 smithi078 ceph-mon[91136]: pgmap v12590: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:42 smithi146 ceph-mon[103278]: pgmap v12590: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:44 smithi078 ceph-mon[91136]: pgmap v12591: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:44 smithi078 ceph-mon[99959]: pgmap v12591: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:44 smithi146 ceph-mon[103278]: pgmap v12591: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:46 smithi078 ceph-mon[91136]: pgmap v12592: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:46 smithi078 ceph-mon[99959]: pgmap v12592: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:46 smithi146 ceph-mon[103278]: pgmap v12592: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[91136]: pgmap v12593: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[99959]: pgmap v12593: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:48 smithi146 ceph-mon[103278]: pgmap v12593: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:48.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:48.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:50.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:50 smithi078 ceph-mon[91136]: pgmap v12594: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:50.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:50 smithi078 ceph-mon[99959]: pgmap v12594: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:50 smithi146 ceph-mon[103278]: pgmap v12594: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:19:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:19:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:19:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:51.938Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:51.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:19:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:19:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:19:51.939Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:19:52.741 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:52.742 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:52 smithi078 ceph-mon[91136]: pgmap v12595: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:52.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:52.742 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:52 smithi078 ceph-mon[99959]: pgmap v12595: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:19:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:52 smithi146 ceph-mon[103278]: pgmap v12595: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:53 smithi146 ceph-mon[103278]: pgmap v12596: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[91136]: pgmap v12596: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:19:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:53 smithi078 ceph-mon[99959]: pgmap v12596: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:19:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:56 smithi146 ceph-mon[103278]: pgmap v12597: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:56.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:56 smithi078 ceph-mon[91136]: pgmap v12597: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:56 smithi078 ceph-mon[99959]: pgmap v12597: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:58.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:58 smithi146 ceph-mon[103278]: pgmap v12598: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:58 smithi078 ceph-mon[91136]: pgmap v12598: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:58.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:58 smithi078 ceph-mon[99959]: pgmap v12598: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:19:59.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:19:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:19:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:19:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:19:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:00.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T03:19:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:00.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:20:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:20:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:20:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:00 smithi078 ceph-mon[99959]: pgmap v12599: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:00 smithi078 ceph-mon[91136]: pgmap v12599: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:00 smithi146 ceph-mon[103278]: pgmap v12599: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:01.939Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:01.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:01.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:02.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:02 smithi078 ceph-mon[91136]: pgmap v12600: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:02.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:02.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:02 smithi078 ceph-mon[99959]: pgmap v12600: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:02 smithi146 ceph-mon[103278]: pgmap v12600: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:04 smithi078 ceph-mon[91136]: pgmap v12601: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:04 smithi078 ceph-mon[99959]: pgmap v12601: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:04 smithi146 ceph-mon[103278]: pgmap v12601: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:06.604 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:20:06.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:06 smithi078 ceph-mon[99959]: pgmap v12602: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:06.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:06 smithi078 ceph-mon[91136]: pgmap v12602: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:06 smithi146 ceph-mon[103278]: pgmap v12602: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:07.129 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:20:07.129 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 7h - - 2024-09-21T03:20:07.129 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 7h - - 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 19s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 14s ago 7h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 19s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 14s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 19s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 19s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 14s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 19s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 19s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 14s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 19s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 19s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 19s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 19s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:20:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 14s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 14s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 14s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 14s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 14s ago 7h 70.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 19s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 19s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:20:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:20:07.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:07.440 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:20:07.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:20:07.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:20:07.440 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:20:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:20:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:20:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:20:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:20:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:20:07.715 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:20:07.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:07 smithi078 ceph-mon[99959]: from='client.92543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:07.715 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3064309655' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:07.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:07 smithi078 ceph-mon[91136]: from='client.92543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:07.715 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3064309655' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:07 smithi146 ceph-mon[103278]: from='client.92543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3064309655' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:08.055 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:08.056 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:08.056 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: from='client.92549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: from='client.92555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: pgmap v12603: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1452724658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: from='client.92549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: from='client.92555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: pgmap v12603: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:08 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1452724658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:08.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: from='client.92549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: from='client.92555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:08.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: pgmap v12603: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:08.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:08.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:08 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1452724658' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:09 smithi146 ceph-mon[103278]: from='client.72651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:09 smithi078 ceph-mon[91136]: from='client.72651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:09 smithi078 ceph-mon[99959]: from='client.72651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:10.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:10 smithi078 ceph-mon[91136]: pgmap v12604: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:10.880 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:10 smithi078 ceph-mon[99959]: pgmap v12604: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:10.880 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:20:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:20:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:20:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:10 smithi146 ceph-mon[103278]: pgmap v12604: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:11.940Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:11.940Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:11.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:12.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:12.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:12 smithi146 ceph-mon[103278]: pgmap v12605: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:12 smithi078 ceph-mon[91136]: pgmap v12605: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:12 smithi078 ceph-mon[99959]: pgmap v12605: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:13 smithi146 ceph-mon[103278]: pgmap v12606: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[91136]: pgmap v12606: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[99959]: pgmap v12606: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:16 smithi146 ceph-mon[103278]: pgmap v12607: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:16 smithi078 ceph-mon[91136]: pgmap v12607: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:16.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:16 smithi078 ceph-mon[99959]: pgmap v12607: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:18.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:18 smithi146 ceph-mon[103278]: pgmap v12608: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:18.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:18 smithi078 ceph-mon[99959]: pgmap v12608: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:18.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:18 smithi078 ceph-mon[91136]: pgmap v12608: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:20.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:20 smithi078 ceph-mon[91136]: pgmap v12609: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:20.753 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:20:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:20:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:20:20.753 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:20 smithi078 ceph-mon[99959]: pgmap v12609: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:20 smithi146 ceph-mon[103278]: pgmap v12609: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:21.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:21.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:21.941Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:22 smithi078 ceph-mon[99959]: pgmap v12610: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:22.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:22 smithi078 ceph-mon[91136]: pgmap v12610: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:22 smithi146 ceph-mon[103278]: pgmap v12610: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:24 smithi078 ceph-mon[91136]: pgmap v12611: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:24.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:24 smithi078 ceph-mon[99959]: pgmap v12611: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:24 smithi146 ceph-mon[103278]: pgmap v12611: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:26.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:26 smithi078 ceph-mon[91136]: pgmap v12612: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:26.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:26 smithi078 ceph-mon[99959]: pgmap v12612: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:26 smithi146 ceph-mon[103278]: pgmap v12612: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[91136]: pgmap v12613: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[99959]: pgmap v12613: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:28 smithi146 ceph-mon[103278]: pgmap v12613: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:30.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:30 smithi078 ceph-mon[91136]: pgmap v12614: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:30.897 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:30 smithi078 ceph-mon[99959]: pgmap v12614: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:30.898 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:20:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:20:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:20:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:30 smithi146 ceph-mon[103278]: pgmap v12614: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:31 smithi146 ceph-mon[103278]: pgmap v12615: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-mon[91136]: pgmap v12615: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-mon[99959]: pgmap v12615: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:31.941Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:31.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:31.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[91136]: pgmap v12616: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:34 smithi078 ceph-mon[99959]: pgmap v12616: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:34 smithi146 ceph-mon[103278]: pgmap v12616: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:36 smithi078 ceph-mon[91136]: pgmap v12617: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:36.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:36 smithi078 ceph-mon[99959]: pgmap v12617: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:36 smithi146 ceph-mon[103278]: pgmap v12617: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:38.351 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:20:38.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:38 smithi078 ceph-mon[91136]: pgmap v12618: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:38 smithi078 ceph-mon[99959]: pgmap v12618: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 7h - - 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 7h - - 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 10s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 5s ago 7h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 10s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 5s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (6h) 10s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 10s ago 7h 558M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 5s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:20:38.879 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 10s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 10s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 5s ago 7h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 10s ago 7h 96.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 10s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 10s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 10s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 5s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 5s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 5s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 5s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 5s ago 7h 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 10s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 10s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:20:38.880 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:20:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:38 smithi146 ceph-mon[103278]: pgmap v12618: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:39.184 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:20:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:20:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:20:39.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:20:39.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:20:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:39 smithi146 ceph-mon[103278]: from='client.92579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:39 smithi146 ceph-mon[103278]: from='client.72669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2649026150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:39 smithi078 ceph-mon[91136]: from='client.92579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:39 smithi078 ceph-mon[91136]: from='client.72669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2649026150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:39 smithi078 ceph-mon[99959]: from='client.92579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:39 smithi078 ceph-mon[99959]: from='client.72669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2649026150' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:20:39.791 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:20:39.791 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:20:39.791 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:20:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[99959]: from='client.72675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[99959]: pgmap v12619: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[99959]: from='client.92603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:40.609 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/90444589' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[91136]: from='client.72675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[91136]: pgmap v12619: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[91136]: from='client.92603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:40.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/90444589' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:40.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:40 smithi146 ceph-mon[103278]: from='client.72675 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:40.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:40 smithi146 ceph-mon[103278]: pgmap v12619: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:40.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:40 smithi146 ceph-mon[103278]: from='client.92603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:20:40.874 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/90444589' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:20:40.899 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:20:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:20:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:20:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:41.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:41.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:41.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:42.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:42 smithi078 ceph-mon[99959]: pgmap v12620: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:42 smithi078 ceph-mon[91136]: pgmap v12620: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:42 smithi146 ceph-mon[103278]: pgmap v12620: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:44 smithi146 ceph-mon[103278]: pgmap v12621: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:44 smithi078 ceph-mon[99959]: pgmap v12621: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:44 smithi078 ceph-mon[91136]: pgmap v12621: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:46 smithi146 ceph-mon[103278]: pgmap v12622: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:46 smithi078 ceph-mon[99959]: pgmap v12622: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:46 smithi078 ceph-mon[91136]: pgmap v12622: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:48 smithi146 ceph-mon[103278]: pgmap v12623: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[91136]: pgmap v12623: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[99959]: pgmap v12623: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:50.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:50 smithi078 ceph-mon[91136]: pgmap v12624: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:50.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:50 smithi078 ceph-mon[99959]: pgmap v12624: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:50.901 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:20:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:20:50] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:20:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:50 smithi146 ceph-mon[103278]: pgmap v12624: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:51.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:51.942Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:20:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:20:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:20:51.943Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:20:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:52 smithi146 ceph-mon[103278]: pgmap v12625: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:52 smithi078 ceph-mon[91136]: pgmap v12625: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:20:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:52 smithi078 ceph-mon[99959]: pgmap v12625: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:54 smithi146 ceph-mon[103278]: pgmap v12626: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[91136]: pgmap v12626: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:20:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:54 smithi078 ceph-mon[99959]: pgmap v12626: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:20:56.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:56 smithi078 ceph-mon[99959]: pgmap v12627: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:20:56.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:56 smithi078 ceph-mon[91136]: pgmap v12627: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:20:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:56 smithi146 ceph-mon[103278]: pgmap v12627: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:20:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:58 smithi078 ceph-mon[91136]: pgmap v12628: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:20:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:58 smithi078 ceph-mon[99959]: pgmap v12628: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:20:59.005 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:58 smithi146 ceph-mon[103278]: pgmap v12628: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:20:59.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:20:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:59.927 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:20:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:20:59.927 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:20:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:00 smithi146 ceph-mon[103278]: pgmap v12629: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:21:00.911 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:00 smithi078 ceph-mon[91136]: pgmap v12629: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:21:00.911 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:00 smithi078 ceph-mon[99959]: pgmap v12629: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:21:00.911 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:21:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:21:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:21:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:01.943Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:02.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:01.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:01.944Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:02 smithi146 ceph-mon[103278]: pgmap v12630: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:21:02.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:02.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:02 smithi078 ceph-mon[91136]: pgmap v12630: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:21:02.823 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:02.823 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:02 smithi078 ceph-mon[99959]: pgmap v12630: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:21:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:04 smithi146 ceph-mon[103278]: pgmap v12631: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:21:05.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:04 smithi078 ceph-mon[99959]: pgmap v12631: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:21:05.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:04 smithi078 ceph-mon[91136]: pgmap v12631: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:21:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:06 smithi146 ceph-mon[103278]: pgmap v12632: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:21:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:06 smithi078 ceph-mon[99959]: pgmap v12632: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:21:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:06 smithi078 ceph-mon[91136]: pgmap v12632: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:21:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[91136]: pgmap v12633: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:21:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[99959]: pgmap v12633: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:21:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:08 smithi146 ceph-mon[103278]: pgmap v12633: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:21:08.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:08.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:10.086 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 7h - - 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 7h - - 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 2s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 17s ago 7h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 2s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 17s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 2s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:21:10.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 2s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 17s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 2s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 2s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 17s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 2s ago 7h 96.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 2s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 2s ago 7h 85.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 2s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:21:10.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 17s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 17s ago 7h 93.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 17s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 17s ago 7h 96.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (6h) 17s ago 7h 72.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 2s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 2s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:21:10.626 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:21:10.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:10 smithi078 ceph-mon[91136]: pgmap v12634: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:21:10.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:10 smithi078 ceph-mon[99959]: pgmap v12634: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:21:10.922 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:21:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:21:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:21:10.935 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:10.936 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:21:10.937 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:21:10.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:21:10.937 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:21:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:10 smithi146 ceph-mon[103278]: pgmap v12634: 161 pgs: 161 active+clean; 585 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:21:11.205 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:21:11.206 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:21:11.206 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:21:11.206 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:21:11.555 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:21:11.555 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:11.555 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:11.556 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:11.556 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:11.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[91136]: from='client.72699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:11.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[91136]: from='client.72705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:11.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[91136]: from='client.92627 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:11.945 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3782289851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:11.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[99959]: from='client.72699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:11.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[99959]: from='client.72705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:11.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[99959]: from='client.92627 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:11.946 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3782289851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:11 smithi146 ceph-mon[103278]: from='client.72699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:11 smithi146 ceph-mon[103278]: from='client.72705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:11 smithi146 ceph-mon[103278]: from='client.92627 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3782289851' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:11.944Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:11.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:11.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:12.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:12.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[91136]: from='client.72723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:12.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[91136]: pgmap v12635: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T03:21:12.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/212947217' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:12.859 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:12.859 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[99959]: from='client.72723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:12.859 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[99959]: pgmap v12635: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T03:21:12.859 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/212947217' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:12 smithi146 ceph-mon[103278]: from='client.72723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:12 smithi146 ceph-mon[103278]: pgmap v12635: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s 2024-09-21T03:21:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/212947217' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[91136]: pgmap v12636: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[99959]: pgmap v12636: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:21:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:14 smithi146 ceph-mon[103278]: pgmap v12636: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:21:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:16 smithi146 ceph-mon[103278]: pgmap v12637: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:21:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:16 smithi078 ceph-mon[91136]: pgmap v12637: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:21:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:16 smithi078 ceph-mon[99959]: pgmap v12637: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:21:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:18 smithi146 ceph-mon[103278]: pgmap v12638: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:21:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:18 smithi078 ceph-mon[91136]: pgmap v12638: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:21:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:18 smithi078 ceph-mon[99959]: pgmap v12638: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:21:20.805 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:21:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:21:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:21:20.805 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:20 smithi078 ceph-mon[99959]: pgmap v12639: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:21:20.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:20 smithi078 ceph-mon[91136]: pgmap v12639: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:21:20.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:20 smithi146 ceph-mon[103278]: pgmap v12639: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:21:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:21.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:21.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:22.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:21.945Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:22.577 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:22.577 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:22 smithi078 ceph-mon[99959]: pgmap v12640: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:21:22.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:22.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:22 smithi078 ceph-mon[91136]: pgmap v12640: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:21:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:22 smithi146 ceph-mon[103278]: pgmap v12640: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:21:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:24 smithi146 ceph-mon[103278]: pgmap v12641: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T03:21:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:24 smithi078 ceph-mon[91136]: pgmap v12641: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T03:21:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:24 smithi078 ceph-mon[99959]: pgmap v12641: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 124 op/s 2024-09-21T03:21:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:26 smithi146 ceph-mon[103278]: pgmap v12642: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T03:21:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:26 smithi078 ceph-mon[91136]: pgmap v12642: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T03:21:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:26 smithi078 ceph-mon[99959]: pgmap v12642: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 140 op/s 2024-09-21T03:21:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[91136]: pgmap v12643: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:21:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:28.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:28.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[99959]: pgmap v12643: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:21:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:28.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:28 smithi146 ceph-mon[103278]: pgmap v12643: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 102 op/s 2024-09-21T03:21:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:29.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:30.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:30 smithi078 ceph-mon[91136]: pgmap v12644: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:21:30.940 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:30 smithi078 ceph-mon[99959]: pgmap v12644: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:21:30.941 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:21:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:21:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:21:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:30 smithi146 ceph-mon[103278]: pgmap v12644: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:21:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:31.945Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:31.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:31.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:32 smithi146 ceph-mon[103278]: pgmap v12645: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:21:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:32 smithi078 ceph-mon[91136]: pgmap v12645: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:21:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:32 smithi078 ceph-mon[99959]: pgmap v12645: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:21:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:34 smithi146 ceph-mon[103278]: pgmap v12646: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:21:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[91136]: pgmap v12646: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:21:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:34 smithi078 ceph-mon[99959]: pgmap v12646: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:21:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:36 smithi146 ceph-mon[103278]: pgmap v12647: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:21:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:36 smithi078 ceph-mon[91136]: pgmap v12647: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:21:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:36 smithi078 ceph-mon[99959]: pgmap v12647: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:21:38.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:38 smithi146 ceph-mon[103278]: pgmap v12648: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:21:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:38 smithi078 ceph-mon[91136]: pgmap v12648: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:21:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:38 smithi078 ceph-mon[99959]: pgmap v12648: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:21:40.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:40 smithi078 ceph-mon[91136]: pgmap v12649: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:21:40.950 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:21:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:21:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:21:40.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:40 smithi078 ceph-mon[99959]: pgmap v12649: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:21:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:40 smithi146 ceph-mon[103278]: pgmap v12649: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:21:41.862 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:21:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:41.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:41.946Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:42.035 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:41.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:42.401 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:21:42.401 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 7h - - 2024-09-21T03:21:42.401 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 7h - - 2024-09-21T03:21:42.401 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 13s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 8s ago 7h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (6h) 13s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 8s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 13s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 13s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 8s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 13s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 13s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 8s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 13s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 13s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 13s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 13s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:21:42.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 8s ago 7h 120M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 8s ago 7h 93.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 8s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 8s ago 7h 96.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 8s ago 7h 72.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 13s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 13s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:21:42.403 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:21:42.672 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:42.672 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:42 smithi078 ceph-mon[91136]: pgmap v12650: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:42.672 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:42.672 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:42 smithi078 ceph-mon[99959]: pgmap v12650: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:21:42.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:21:42.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:21:42.727 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:21:42.727 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:21:42.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:21:42.727 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:21:43.005 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:21:43.006 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:21:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:42 smithi146 ceph-mon[103278]: pgmap v12650: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:43.354 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:21:43.355 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:21:43.355 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:21:43.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[99959]: from='client.72735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:43.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[99959]: from='client.72741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:43.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[99959]: from='client.72747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:43.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1194125088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:43.607 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/549759427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:43.607 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[91136]: from='client.72735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:43.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[91136]: from='client.72741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:43.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[91136]: from='client.72747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:43.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1194125088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:43.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/549759427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:43 smithi146 ceph-mon[103278]: from='client.72735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:43 smithi146 ceph-mon[103278]: from='client.72741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:43 smithi146 ceph-mon[103278]: from='client.72747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1194125088' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:21:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/549759427' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:21:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:44 smithi146 ceph-mon[103278]: from='client.72759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:44 smithi146 ceph-mon[103278]: pgmap v12651: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:44 smithi078 ceph-mon[99959]: from='client.72759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:44 smithi078 ceph-mon[99959]: pgmap v12651: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:44 smithi078 ceph-mon[91136]: from='client.72759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:21:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:44 smithi078 ceph-mon[91136]: pgmap v12651: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:21:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:46 smithi146 ceph-mon[103278]: pgmap v12652: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:46 smithi078 ceph-mon[91136]: pgmap v12652: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:46 smithi078 ceph-mon[99959]: pgmap v12652: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:48 smithi146 ceph-mon[103278]: pgmap v12653: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:48 smithi078 ceph-mon[99959]: pgmap v12653: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:48 smithi078 ceph-mon[91136]: pgmap v12653: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:50.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:50 smithi078 ceph-mon[91136]: pgmap v12654: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:50.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:50 smithi078 ceph-mon[99959]: pgmap v12654: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:50.961 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:21:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:21:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:21:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:50 smithi146 ceph-mon[103278]: pgmap v12654: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:52.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:51.946Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:52.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:51.947Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:21:52.209 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:21:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:21:51.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:21:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:52 smithi146 ceph-mon[103278]: pgmap v12655: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:52 smithi078 ceph-mon[99959]: pgmap v12655: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:21:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:52 smithi078 ceph-mon[91136]: pgmap v12655: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:54.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[99959]: pgmap v12656: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[91136]: pgmap v12656: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:54.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:54 smithi146 ceph-mon[103278]: pgmap v12656: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:21:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:21:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:56 smithi146 ceph-mon[103278]: pgmap v12657: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:56 smithi078 ceph-mon[91136]: pgmap v12657: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:56 smithi078 ceph-mon[99959]: pgmap v12657: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:58 smithi146 ceph-mon[103278]: pgmap v12658: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:58 smithi078 ceph-mon[91136]: pgmap v12658: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:21:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:58 smithi078 ceph-mon[99959]: pgmap v12658: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:21:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:21:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:21:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:00 smithi146 ceph-mon[103278]: pgmap v12659: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:00.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:00 smithi078 ceph-mon[91136]: pgmap v12659: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:00.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:00 smithi078 ceph-mon[99959]: pgmap v12659: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:00.968 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:22:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:22:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:22:02.199 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:01.947Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:02.199 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:01.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:02.199 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:01.948Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:02.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:02.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:02 smithi146 ceph-mon[103278]: pgmap v12660: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:02 smithi078 ceph-mon[99959]: pgmap v12660: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:02 smithi078 ceph-mon[91136]: pgmap v12660: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:22:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:04 smithi078 ceph-mon[91136]: pgmap v12661: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:04 smithi078 ceph-mon[99959]: pgmap v12661: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:04 smithi146 ceph-mon[103278]: pgmap v12661: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:06 smithi146 ceph-mon[103278]: pgmap v12662: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:06 smithi078 ceph-mon[91136]: pgmap v12662: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:06 smithi078 ceph-mon[99959]: pgmap v12662: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:08.818 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:08 smithi146 ceph-mon[103278]: pgmap v12663: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:08 smithi078 ceph-mon[99959]: pgmap v12663: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:08 smithi078 ceph-mon[91136]: pgmap v12663: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:09 smithi146 ceph-mon[103278]: pgmap v12664: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[99959]: pgmap v12664: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:09 smithi078 ceph-mon[91136]: pgmap v12664: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:10.972 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:22:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:22:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:22:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:11.948Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:11.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:11.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:12 smithi146 ceph-mon[103278]: pgmap v12665: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:12 smithi078 ceph-mon[91136]: pgmap v12665: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:12 smithi078 ceph-mon[99959]: pgmap v12665: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:13.653 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:22:14.183 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 7h - - 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 7h - - 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 5s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 0s ago 7h 89.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 5s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 0s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 5s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 5s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:22:14.230 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 0s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 5s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 5s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 0s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 5s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 5s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 5s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 5s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 0s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 0s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 0s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:22:14.231 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 0s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:22:14.232 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 0s ago 7h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:22:14.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 5s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:22:14.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:22:14.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 5s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:22:14.232 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:14.492 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:22:14.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:22:14.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:14.493 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:22:14.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:22:14.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:22:14.493 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:22:14.760 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:22:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:22:14.760 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:22:14.761 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:22:14.761 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[99959]: pgmap v12666: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3884068098' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[91136]: pgmap v12666: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:14.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:14.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:14.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3884068098' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:14 smithi146 ceph-mon[103278]: pgmap v12666: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3884068098' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:15.072 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:22:15.072 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:15.072 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:15.073 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:15.073 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:15 smithi146 ceph-mon[103278]: from='client.72771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:15 smithi146 ceph-mon[103278]: from='client.92693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:15 smithi146 ceph-mon[103278]: from='client.72783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1751406718' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[99959]: from='client.72771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[99959]: from='client.92693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[99959]: from='client.72783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1751406718' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[91136]: from='client.72771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[91136]: from='client.92693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[91136]: from='client.72783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1751406718' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:16 smithi146 ceph-mon[103278]: from='client.92711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:16 smithi146 ceph-mon[103278]: pgmap v12667: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:16 smithi078 ceph-mon[99959]: from='client.92711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:16 smithi078 ceph-mon[99959]: pgmap v12667: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:16 smithi078 ceph-mon[91136]: from='client.92711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:16 smithi078 ceph-mon[91136]: pgmap v12667: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:18 smithi146 ceph-mon[103278]: pgmap v12668: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:18 smithi078 ceph-mon[99959]: pgmap v12668: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:18 smithi078 ceph-mon[91136]: pgmap v12668: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:20.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:20 smithi078 ceph-mon[91136]: pgmap v12669: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:20.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:20 smithi078 ceph-mon[99959]: pgmap v12669: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:20.981 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:22:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:22:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:22:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:20 smithi146 ceph-mon[103278]: pgmap v12669: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:21.949Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:21.949Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:21.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:22 smithi146 ceph-mon[103278]: pgmap v12670: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:22.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:22.921 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:22 smithi078 ceph-mon[91136]: pgmap v12670: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:22.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:22.921 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:22 smithi078 ceph-mon[99959]: pgmap v12670: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:24 smithi146 ceph-mon[103278]: pgmap v12671: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:24.882 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:24 smithi078 ceph-mon[99959]: pgmap v12671: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:24.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:24 smithi078 ceph-mon[91136]: pgmap v12671: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:26 smithi146 ceph-mon[103278]: pgmap v12672: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:26 smithi078 ceph-mon[91136]: pgmap v12672: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:26 smithi078 ceph-mon[99959]: pgmap v12672: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:28.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:28 smithi078 ceph-mon[99959]: pgmap v12673: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:28.896 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:28 smithi078 ceph-mon[91136]: pgmap v12673: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:28 smithi146 ceph-mon[103278]: pgmap v12673: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:30.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:30 smithi078 ceph-mon[91136]: pgmap v12674: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:30.992 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:30 smithi078 ceph-mon[99959]: pgmap v12674: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:30.992 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:22:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:22:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:22:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:30 smithi146 ceph-mon[103278]: pgmap v12674: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:31.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:31.950Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:31.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:32 smithi146 ceph-mon[103278]: pgmap v12675: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:32 smithi078 ceph-mon[91136]: pgmap v12675: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:32 smithi078 ceph-mon[99959]: pgmap v12675: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[91136]: pgmap v12676: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[99959]: pgmap v12676: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:34 smithi146 ceph-mon[103278]: pgmap v12676: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:36 smithi146 ceph-mon[103278]: pgmap v12677: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:36 smithi078 ceph-mon[91136]: pgmap v12677: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:36 smithi078 ceph-mon[99959]: pgmap v12677: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:38 smithi146 ceph-mon[103278]: pgmap v12678: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:38 smithi078 ceph-mon[91136]: pgmap v12678: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:38 smithi078 ceph-mon[99959]: pgmap v12678: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:41.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:40 smithi078 ceph-mon[91136]: pgmap v12679: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:41.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:40 smithi078 ceph-mon[99959]: pgmap v12679: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:41.003 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:22:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:22:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:22:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:40 smithi146 ceph-mon[103278]: pgmap v12679: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:41.950Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:41.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:41.951Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:42.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:42.847 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:42 smithi146 ceph-mon[103278]: pgmap v12680: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:42 smithi078 ceph-mon[91136]: pgmap v12680: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:42 smithi078 ceph-mon[99959]: pgmap v12680: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:44.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:44 smithi078 ceph-mon[99959]: pgmap v12681: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:44.868 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:44.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:44 smithi078 ceph-mon[91136]: pgmap v12681: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:44.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:44 smithi146 ceph-mon[103278]: pgmap v12681: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:22:45.383 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:22:45.922 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:22:45.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 7h - - 2024-09-21T03:22:45.922 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 7h - - 2024-09-21T03:22:45.922 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 17s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:22:45.922 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 12s ago 7h 89.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 17s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (6h) 12s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 17s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (6h) 17s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 12s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 17s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 17s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 12s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 17s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 17s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 17s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 17s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 12s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:22:45.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 12s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 12s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 12s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 12s ago 7h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 17s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 17s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:22:45.924 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:22:46.236 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:22:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:22:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:22:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:22:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:22:46.238 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:22:46.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:22:46.504 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:22:46.504 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:22:46.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:22:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:46 smithi078 ceph-mon[91136]: from='client.72807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:46 smithi078 ceph-mon[91136]: pgmap v12682: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:46.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/869463251' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:46 smithi078 ceph-mon[99959]: from='client.72807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:46 smithi078 ceph-mon[99959]: pgmap v12682: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:46.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/869463251' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:46.815 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:22:46.815 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:22:46.815 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:22:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:46 smithi146 ceph-mon[103278]: from='client.72807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:46 smithi146 ceph-mon[103278]: pgmap v12682: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/869463251' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:22:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:47 smithi146 ceph-mon[103278]: from='client.92729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:47 smithi146 ceph-mon[103278]: from='client.72819 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:47 smithi146 ceph-mon[103278]: from='client.92747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/720534381' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[99959]: from='client.92729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[99959]: from='client.72819 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[99959]: from='client.92747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/720534381' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[91136]: from='client.92729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[91136]: from='client.72819 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[91136]: from='client.92747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:22:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/720534381' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:22:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:48 smithi146 ceph-mon[103278]: pgmap v12683: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:49.018 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:48 smithi078 ceph-mon[99959]: pgmap v12683: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:49.018 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:48 smithi078 ceph-mon[91136]: pgmap v12683: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:50 smithi146 ceph-mon[103278]: pgmap v12684: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[99959]: pgmap v12684: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-mon[91136]: pgmap v12684: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:51.013 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:22:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:22:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:22:52.203 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:51.951Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:52.203 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:51.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:22:52.203 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:22:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:22:51.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:22:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:52 smithi078 ceph-mon[91136]: pgmap v12685: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:52.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:52 smithi078 ceph-mon[99959]: pgmap v12685: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:22:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:52 smithi146 ceph-mon[103278]: pgmap v12685: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:54 smithi146 ceph-mon[103278]: pgmap v12686: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[99959]: pgmap v12686: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[91136]: pgmap v12686: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:22:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:22:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:56 smithi146 ceph-mon[103278]: pgmap v12687: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:56 smithi078 ceph-mon[91136]: pgmap v12687: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:56 smithi078 ceph-mon[99959]: pgmap v12687: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:58 smithi146 ceph-mon[103278]: pgmap v12688: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:58 smithi078 ceph-mon[99959]: pgmap v12688: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:22:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:58 smithi078 ceph-mon[91136]: pgmap v12688: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:22:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:22:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:22:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:01.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:00 smithi146 ceph-mon[103278]: pgmap v12689: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:01.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:00 smithi078 ceph-mon[91136]: pgmap v12689: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:01.022 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:00 smithi078 ceph-mon[99959]: pgmap v12689: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:01.023 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:23:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:23:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:23:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:01.952Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:01.952Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:01.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:02 smithi146 ceph-mon[103278]: pgmap v12690: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:02.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:02.892 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:02 smithi078 ceph-mon[99959]: pgmap v12690: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:02.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:02.892 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:02 smithi078 ceph-mon[91136]: pgmap v12690: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:04.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:04 smithi078 ceph-mon[91136]: pgmap v12691: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:04 smithi146 ceph-mon[103278]: pgmap v12691: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:04 smithi078 ceph-mon[99959]: pgmap v12691: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:06.936 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:06 smithi078 ceph-mon[91136]: pgmap v12692: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:06.936 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:06 smithi078 ceph-mon[99959]: pgmap v12692: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:06 smithi146 ceph-mon[103278]: pgmap v12692: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:08 smithi146 ceph-mon[103278]: pgmap v12693: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:08 smithi078 ceph-mon[91136]: pgmap v12693: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:09.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:08 smithi078 ceph-mon[99959]: pgmap v12693: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:10 smithi146 ceph-mon[103278]: pgmap v12694: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[91136]: pgmap v12694: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:10.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-mon[99959]: pgmap v12694: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:11.033 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:23:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:23:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:23:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:11.953Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:11.953Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:11.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:12 smithi146 ceph-mon[103278]: pgmap v12695: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:12 smithi078 ceph-mon[91136]: pgmap v12695: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:12 smithi078 ceph-mon[99959]: pgmap v12695: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:14 smithi146 ceph-mon[103278]: pgmap v12696: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[91136]: pgmap v12696: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[99959]: pgmap v12696: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:15.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:16 smithi146 ceph-mon[103278]: pgmap v12697: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:16 smithi078 ceph-mon[91136]: pgmap v12697: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:16 smithi078 ceph-mon[99959]: pgmap v12697: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:17.114 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:23:17.646 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 7h - - 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 7h - - 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 8s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 3s ago 7h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 8s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 3s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 8s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 8s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 3s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6h) 8s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 8s ago 7h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 3s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:23:17.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 8s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 8s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 8s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 8s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 3s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 3s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 3s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 3s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 3s ago 7h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 8s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 8s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:23:17.648 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:17.956 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:23:17.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:23:17.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:17.957 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:23:17.957 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:23:17.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:23:17.957 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:23:18.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:23:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:23:18.544 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:23:18.544 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:18.544 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:18.544 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:18.544 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:18 smithi146 ceph-mon[103278]: from='client.92759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:18 smithi146 ceph-mon[103278]: from='client.72849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:18 smithi146 ceph-mon[103278]: pgmap v12698: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:18 smithi146 ceph-mon[103278]: from='client.92771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2552837252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:19.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4120180779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[99959]: from='client.92759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[99959]: from='client.72849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[99959]: pgmap v12698: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[99959]: from='client.92771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2552837252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4120180779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[91136]: from='client.92759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[91136]: from='client.72849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[91136]: pgmap v12698: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[91136]: from='client.92771 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2552837252' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4120180779' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:19 smithi146 ceph-mon[103278]: from='client.72867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:19 smithi078 ceph-mon[91136]: from='client.72867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:19 smithi078 ceph-mon[99959]: from='client.72867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:20 smithi146 ceph-mon[103278]: pgmap v12699: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:20 smithi078 ceph-mon[91136]: pgmap v12699: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:23:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:23:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:23:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:20 smithi078 ceph-mon[99959]: pgmap v12699: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:21.954Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:21.954Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:21.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:22.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:22.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:22 smithi146 ceph-mon[103278]: pgmap v12700: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:22 smithi078 ceph-mon[99959]: pgmap v12700: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:22 smithi078 ceph-mon[91136]: pgmap v12700: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:24 smithi146 ceph-mon[103278]: pgmap v12701: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:24.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:24 smithi078 ceph-mon[99959]: pgmap v12701: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:24.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:24 smithi078 ceph-mon[91136]: pgmap v12701: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:26 smithi146 ceph-mon[103278]: pgmap v12702: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:26 smithi078 ceph-mon[91136]: pgmap v12702: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:26 smithi078 ceph-mon[99959]: pgmap v12702: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:28 smithi146 ceph-mon[103278]: pgmap v12703: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:28 smithi078 ceph-mon[91136]: pgmap v12703: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:28 smithi078 ceph-mon[99959]: pgmap v12703: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:30 smithi146 ceph-mon[103278]: pgmap v12704: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:30 smithi078 ceph-mon[91136]: pgmap v12704: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:31.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:30 smithi078 ceph-mon[99959]: pgmap v12704: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:23:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:23:30] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:23:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:31.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:31.955Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:31.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:32.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:32.986 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:32 smithi078 ceph-mon[99959]: pgmap v12705: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:32.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:32.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:32 smithi078 ceph-mon[91136]: pgmap v12705: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:32 smithi146 ceph-mon[103278]: pgmap v12705: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:34 smithi146 ceph-mon[103278]: pgmap v12706: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[91136]: pgmap v12706: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[99959]: pgmap v12706: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:36 smithi146 ceph-mon[103278]: pgmap v12707: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:36 smithi078 ceph-mon[91136]: pgmap v12707: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:36 smithi078 ceph-mon[99959]: pgmap v12707: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:38 smithi146 ceph-mon[103278]: pgmap v12708: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:38 smithi078 ceph-mon[91136]: pgmap v12708: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:38 smithi078 ceph-mon[99959]: pgmap v12708: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:40 smithi146 ceph-mon[103278]: pgmap v12709: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:40 smithi078 ceph-mon[91136]: pgmap v12709: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:23:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:23:40] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:23:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:40 smithi078 ceph-mon[99959]: pgmap v12709: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:42.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:41.955Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:42.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:41.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:42.289 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:41.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:42 smithi146 ceph-mon[103278]: pgmap v12710: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:42 smithi078 ceph-mon[99959]: pgmap v12710: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:42 smithi078 ceph-mon[91136]: pgmap v12710: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:44 smithi146 ceph-mon[103278]: pgmap v12711: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:44 smithi078 ceph-mon[91136]: pgmap v12711: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:44 smithi078 ceph-mon[99959]: pgmap v12711: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:23:46.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:46 smithi078 ceph-mon[91136]: pgmap v12712: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:46.852 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:46 smithi078 ceph-mon[99959]: pgmap v12712: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:46 smithi146 ceph-mon[103278]: pgmap v12712: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:48.830 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:23:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:48 smithi146 ceph-mon[103278]: pgmap v12713: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:48 smithi078 ceph-mon[99959]: pgmap v12713: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:48 smithi078 ceph-mon[91136]: pgmap v12713: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:49.367 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:23:49.367 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 20s ago 7h - - 2024-09-21T03:23:49.367 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 7h - - 2024-09-21T03:23:49.367 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 20s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 15s ago 7h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 20s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 15s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 20s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 20s ago 7h 559M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (6h) 15s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 20s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 20s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 15s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 20s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 20s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 20s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 20s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:23:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 15s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 15s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 15s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 15s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 15s ago 7h 66.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 20s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 20s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:23:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:23:49.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:23:49.681 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:23:49.952 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:23:49.953 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:23:50.264 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:23:50.264 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:23:50.264 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:23:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='client.72879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='client.72885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='client.72891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: pgmap v12714: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3591629666' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1205256978' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='client.72879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='client.72885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='client.72891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: pgmap v12714: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:50.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3591629666' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:50.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1205256978' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:50.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='client.72879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='client.72885 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='client.72891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: pgmap v12714: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3591629666' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:23:50.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1205256978' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:23:51.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:23:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:23:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:23:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:51 smithi146 ceph-mon[103278]: from='client.92819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:51 smithi078 ceph-mon[91136]: from='client.92819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:51.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:51 smithi078 ceph-mon[99959]: from='client.92819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:23:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:51.956Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:51.956Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:23:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:23:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:23:51.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:23:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:52 smithi146 ceph-mon[103278]: pgmap v12715: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:52.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:52 smithi078 ceph-mon[91136]: pgmap v12715: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:23:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:52 smithi078 ceph-mon[99959]: pgmap v12715: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:54 smithi146 ceph-mon[103278]: pgmap v12716: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[91136]: pgmap v12716: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[99959]: pgmap v12716: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:23:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:56 smithi146 ceph-mon[103278]: pgmap v12717: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:56 smithi078 ceph-mon[91136]: pgmap v12717: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:57.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:56 smithi078 ceph-mon[99959]: pgmap v12717: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:58 smithi146 ceph-mon[103278]: pgmap v12718: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:58 smithi078 ceph-mon[91136]: pgmap v12718: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:23:59.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:58 smithi078 ceph-mon[99959]: pgmap v12718: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:23:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:23:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:23:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:00 smithi146 ceph-mon[103278]: pgmap v12719: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:00 smithi078 ceph-mon[91136]: pgmap v12719: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:24:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:24:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:24:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:00 smithi078 ceph-mon[99959]: pgmap v12719: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:01.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:01.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:01.957Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:02 smithi146 ceph-mon[103278]: pgmap v12720: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:02 smithi078 ceph-mon[91136]: pgmap v12720: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:02 smithi078 ceph-mon[99959]: pgmap v12720: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:04.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:04 smithi146 ceph-mon[103278]: pgmap v12721: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:04 smithi078 ceph-mon[91136]: pgmap v12721: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:04 smithi078 ceph-mon[99959]: pgmap v12721: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:06.939 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:06 smithi078 ceph-mon[99959]: pgmap v12722: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:06.940 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:06 smithi078 ceph-mon[91136]: pgmap v12722: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:06 smithi146 ceph-mon[103278]: pgmap v12722: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:08.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:08 smithi078 ceph-mon[91136]: pgmap v12723: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:08.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:08 smithi078 ceph-mon[99959]: pgmap v12723: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:08 smithi146 ceph-mon[103278]: pgmap v12723: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:10 smithi146 ceph-mon[103278]: pgmap v12724: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:10.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:24:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[91136]: pgmap v12724: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:10 smithi078 ceph-mon[99959]: pgmap v12724: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:11.957Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:11.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:11.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:12.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:12.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:12 smithi146 ceph-mon[103278]: pgmap v12725: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:12 smithi078 ceph-mon[91136]: pgmap v12725: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:12 smithi078 ceph-mon[99959]: pgmap v12725: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:14 smithi146 ceph-mon[103278]: pgmap v12726: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[91136]: pgmap v12726: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[99959]: pgmap v12726: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:16 smithi146 ceph-mon[103278]: pgmap v12727: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:16 smithi078 ceph-mon[91136]: pgmap v12727: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:17.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:16 smithi078 ceph-mon[99959]: pgmap v12727: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:18 smithi146 ceph-mon[103278]: pgmap v12728: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:18 smithi078 ceph-mon[91136]: pgmap v12728: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:18 smithi078 ceph-mon[99959]: pgmap v12728: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:20.557 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:24:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:20 smithi146 ceph-mon[103278]: pgmap v12729: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:20 smithi078 ceph-mon[91136]: pgmap v12729: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:20 smithi078 ceph-mon[99959]: pgmap v12729: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:24:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:24:20] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 7h - - 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 7h - - 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 11s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 6s ago 7h 88.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 11s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:24:21.085 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 6s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 11s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 11s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 6s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 11s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 11s ago 7h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 6s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 11s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 11s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 11s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 11s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 6s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 6s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 6s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:24:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 6s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:24:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 6s ago 7h 68.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:24:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 11s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:24:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:24:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 11s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:24:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:21.393 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:24:21.394 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:24:21.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:21.394 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:24:21.394 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:24:21.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:24:21.394 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:24:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:24:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:24:21.665 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:24:21.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-mon[91136]: from='client.72915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:21.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3200968014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:21.665 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-mon[99959]: from='client.72915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:21.665 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3200968014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:21.976 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:24:21.976 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:21.976 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:21 smithi146 ceph-mon[103278]: from='client.72915 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3200968014' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:21.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:21.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:21.958Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:22 smithi146 ceph-mon[103278]: from='client.92837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:22 smithi146 ceph-mon[103278]: from='client.92843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:22 smithi146 ceph-mon[103278]: pgmap v12730: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:22 smithi146 ceph-mon[103278]: from='client.72939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:22.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:22 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3059560388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[91136]: from='client.92837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[91136]: from='client.92843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[91136]: pgmap v12730: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[91136]: from='client.72939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3059560388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[99959]: from='client.92837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[99959]: from='client.92843 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[99959]: pgmap v12730: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[99959]: from='client.72939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:22 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3059560388' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:24.989 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:24 smithi146 ceph-mon[103278]: pgmap v12731: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:24 smithi078 ceph-mon[91136]: pgmap v12731: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:25.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:24 smithi078 ceph-mon[99959]: pgmap v12731: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:26.016 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 03:24:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T03:24:25.847440434Z level=info msg="Completed cleanup jobs" duration=11.608828ms 2024-09-21T03:24:26.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:26 smithi146 ceph-mon[103278]: pgmap v12732: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:26.968 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:26 smithi078 ceph-mon[99959]: pgmap v12732: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:26.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:26 smithi078 ceph-mon[91136]: pgmap v12732: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:28 smithi146 ceph-mon[103278]: pgmap v12733: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:28 smithi078 ceph-mon[91136]: pgmap v12733: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:28 smithi078 ceph-mon[99959]: pgmap v12733: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:30 smithi146 ceph-mon[103278]: pgmap v12734: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:24:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:24:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[91136]: pgmap v12734: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[99959]: pgmap v12734: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:31 smithi146 ceph-mon[103278]: pgmap v12735: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-mon[91136]: pgmap v12735: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-mon[99959]: pgmap v12735: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:31.958Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:31.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:31.959Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:34 smithi146 ceph-mon[103278]: pgmap v12736: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[91136]: pgmap v12736: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[99959]: pgmap v12736: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:36 smithi146 ceph-mon[103278]: pgmap v12737: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:36 smithi078 ceph-mon[91136]: pgmap v12737: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:36 smithi078 ceph-mon[99959]: pgmap v12737: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:38 smithi146 ceph-mon[103278]: pgmap v12738: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:38 smithi078 ceph-mon[91136]: pgmap v12738: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:39.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:38 smithi078 ceph-mon[99959]: pgmap v12738: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:40 smithi146 ceph-mon[103278]: pgmap v12739: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:40 smithi078 ceph-mon[91136]: pgmap v12739: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:40 smithi078 ceph-mon[99959]: pgmap v12739: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:24:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:24:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:24:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:41.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:41.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:41.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:42 smithi146 ceph-mon[103278]: pgmap v12740: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:42 smithi078 ceph-mon[91136]: pgmap v12740: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:42 smithi078 ceph-mon[99959]: pgmap v12740: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:44.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:44 smithi146 ceph-mon[103278]: pgmap v12741: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:44.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:44 smithi078 ceph-mon[91136]: pgmap v12741: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:44 smithi078 ceph-mon[99959]: pgmap v12741: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:24:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:46 smithi146 ceph-mon[103278]: pgmap v12742: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:46 smithi078 ceph-mon[91136]: pgmap v12742: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:46 smithi078 ceph-mon[99959]: pgmap v12742: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:48.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:48 smithi078 ceph-mon[91136]: pgmap v12743: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:48.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:48 smithi078 ceph-mon[99959]: pgmap v12743: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:48 smithi146 ceph-mon[103278]: pgmap v12743: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:50 smithi146 ceph-mon[103278]: pgmap v12744: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:24:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[99959]: pgmap v12744: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[91136]: pgmap v12744: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:51.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:52.272 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:51.959Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:52.272 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:51.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:24:52.272 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:24:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:24:51.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:24:52.272 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:24:52.799 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 7h - - 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 7h - - 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 3s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 18s ago 7h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 3s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 18s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 3s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 3s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 18s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 3s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 3s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 18s ago 7h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6h) 3s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:24:52.800 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 3s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 3s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 3s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 18s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 18s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 18s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 18s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 18s ago 7h 70.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 3s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 3s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:24:52.801 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:24:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:52 smithi146 ceph-mon[103278]: pgmap v12745: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:52 smithi078 ceph-mon[99959]: pgmap v12745: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:24:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:52 smithi078 ceph-mon[91136]: pgmap v12745: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:24:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:24:53.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:24:53.385 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:24:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:24:53.386 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:24:53.707 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:24:53.707 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:53.707 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:24:53.707 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:53.707 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:24:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:53 smithi146 ceph-mon[103278]: from='client.92867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:53 smithi146 ceph-mon[103278]: from='client.92873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/144516640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:53 smithi078 ceph-mon[99959]: from='client.92867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:53 smithi078 ceph-mon[99959]: from='client.92873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/144516640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:53 smithi078 ceph-mon[91136]: from='client.92867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:53 smithi078 ceph-mon[91136]: from='client.92873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/144516640' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:24:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[91136]: from='client.72963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[91136]: from='client.72975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[91136]: pgmap v12746: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2031745784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[99959]: from='client.72963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[99959]: from='client.72975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[99959]: pgmap v12746: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:54 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2031745784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:54 smithi146 ceph-mon[103278]: from='client.72963 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:54 smithi146 ceph-mon[103278]: from='client.72975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:24:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:54 smithi146 ceph-mon[103278]: pgmap v12746: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:24:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:54 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2031745784' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:24:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:55 smithi146 ceph-mon[103278]: pgmap v12747: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[91136]: pgmap v12747: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:24:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:55 smithi078 ceph-mon[99959]: pgmap v12747: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:59.019 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:58 smithi146 ceph-mon[103278]: pgmap v12748: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:59.120 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:58 smithi078 ceph-mon[91136]: pgmap v12748: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:24:59.121 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:58 smithi078 ceph-mon[99959]: pgmap v12748: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:24:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:00.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:24:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:24:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:00 smithi146 ceph-mon[103278]: pgmap v12749: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:00 smithi078 ceph-mon[91136]: pgmap v12749: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:00 smithi078 ceph-mon[99959]: pgmap v12749: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:25:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:25:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:25:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:01.960Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:01.960Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:01.961Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:02 smithi146 ceph-mon[103278]: pgmap v12750: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:02 smithi078 ceph-mon[91136]: pgmap v12750: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:02 smithi078 ceph-mon[99959]: pgmap v12750: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:04.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:04 smithi078 ceph-mon[91136]: pgmap v12751: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:04 smithi078 ceph-mon[99959]: pgmap v12751: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:04 smithi146 ceph-mon[103278]: pgmap v12751: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:06.853 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:06 smithi146 ceph-mon[103278]: pgmap v12752: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:06 smithi078 ceph-mon[91136]: pgmap v12752: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:07.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:06 smithi078 ceph-mon[99959]: pgmap v12752: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:08 smithi146 ceph-mon[103278]: pgmap v12753: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:08 smithi078 ceph-mon[91136]: pgmap v12753: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:08 smithi078 ceph-mon[99959]: pgmap v12753: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:10.635 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:25:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:25:10.635 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[99959]: pgmap v12754: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:10.635 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:10.635 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:10.635 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:10.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[91136]: pgmap v12754: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:10.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:10.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:10.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:10 smithi146 ceph-mon[103278]: pgmap v12754: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:11.961Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:11.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:11.962Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:12 smithi146 ceph-mon[103278]: pgmap v12755: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:12 smithi078 ceph-mon[91136]: pgmap v12755: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:12 smithi078 ceph-mon[99959]: pgmap v12755: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:14 smithi146 ceph-mon[103278]: pgmap v12756: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:14 smithi078 ceph-mon[91136]: pgmap v12756: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:14 smithi078 ceph-mon[99959]: pgmap v12756: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:15 smithi146 ceph-mon[103278]: pgmap v12757: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[91136]: pgmap v12757: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:15 smithi078 ceph-mon[99959]: pgmap v12757: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:18 smithi146 ceph-mon[103278]: pgmap v12758: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:18 smithi078 ceph-mon[91136]: pgmap v12758: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:18 smithi078 ceph-mon[99959]: pgmap v12758: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:20 smithi146 ceph-mon[103278]: pgmap v12759: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:20 smithi078 ceph-mon[91136]: pgmap v12759: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:20 smithi078 ceph-mon[99959]: pgmap v12759: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:25:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:25:20] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:25:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:21.962Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:21.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:21.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:22 smithi146 ceph-mon[103278]: pgmap v12760: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:22 smithi078 ceph-mon[91136]: pgmap v12760: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:22 smithi078 ceph-mon[99959]: pgmap v12760: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:24.008 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 7h - - 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 7h - - 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 14s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 9s ago 7h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 14s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 9s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:25:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 14s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 14s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 9s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 14s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 14s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 9s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 14s ago 7h 98.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 14s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 14s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 14s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 9s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 9s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 9s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 9s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:25:24.538 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 9s ago 7h 71.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:25:24.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 14s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:25:24.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:25:24.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 14s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:25:24.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:25:24.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:25:24.849 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:25:24.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:24 smithi078 ceph-mon[99959]: pgmap v12761: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:24.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:24 smithi078 ceph-mon[91136]: pgmap v12761: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:25.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:24 smithi146 ceph-mon[103278]: pgmap v12761: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:25:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:25:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:25:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:25:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:25:25.119 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:25:25.424 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:25:25.424 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:25.424 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[99959]: from='client.92903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[99959]: from='client.72993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[99959]: from='client.72999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2259158877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2622181677' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[91136]: from='client.92903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[91136]: from='client.72993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[91136]: from='client.72999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2259158877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2622181677' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:25 smithi146 ceph-mon[103278]: from='client.92903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:25 smithi146 ceph-mon[103278]: from='client.72993 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:25 smithi146 ceph-mon[103278]: from='client.72999 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2259158877' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2622181677' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:26 smithi146 ceph-mon[103278]: from='client.92927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:26.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:26 smithi146 ceph-mon[103278]: pgmap v12762: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:26 smithi078 ceph-mon[99959]: from='client.92927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:27.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:26 smithi078 ceph-mon[99959]: pgmap v12762: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:26 smithi078 ceph-mon[91136]: from='client.92927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:26 smithi078 ceph-mon[91136]: pgmap v12762: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:28.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:28 smithi078 ceph-mon[91136]: pgmap v12763: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:28.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:28.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:28 smithi078 ceph-mon[99959]: pgmap v12763: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:28.982 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:28 smithi146 ceph-mon[103278]: pgmap v12763: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:29.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:30.639 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:25:30] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T03:25:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:30 smithi146 ceph-mon[103278]: pgmap v12764: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[91136]: pgmap v12764: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[99959]: pgmap v12764: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:31 smithi146 ceph-mon[103278]: pgmap v12765: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:32.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-mon[99959]: pgmap v12765: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-mon[91136]: pgmap v12765: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:31.963Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:31.963Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:31.964Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:34.630 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:34 smithi146 ceph-mon[103278]: pgmap v12766: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:34 smithi078 ceph-mon[91136]: pgmap v12766: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:34 smithi078 ceph-mon[99959]: pgmap v12766: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:35 smithi146 ceph-mon[103278]: pgmap v12767: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[91136]: pgmap v12767: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:35 smithi078 ceph-mon[99959]: pgmap v12767: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:38 smithi146 ceph-mon[103278]: pgmap v12768: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:38 smithi078 ceph-mon[91136]: pgmap v12768: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:38 smithi078 ceph-mon[99959]: pgmap v12768: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:40 smithi146 ceph-mon[103278]: pgmap v12769: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:40 smithi078 ceph-mon[91136]: pgmap v12769: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:40 smithi078 ceph-mon[99959]: pgmap v12769: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:25:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:25:40] "GET /metrics HTTP/1.1" 200 38999 "" "Prometheus/2.43.0" 2024-09-21T03:25:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:41.964Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:41.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:41.965Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:42 smithi146 ceph-mon[103278]: pgmap v12770: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:42 smithi078 ceph-mon[99959]: pgmap v12770: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:42 smithi078 ceph-mon[91136]: pgmap v12770: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:44.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:44 smithi146 ceph-mon[103278]: pgmap v12771: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:44 smithi078 ceph-mon[91136]: pgmap v12771: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:44 smithi078 ceph-mon[99959]: pgmap v12771: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:46.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:46 smithi146 ceph-mon[103278]: pgmap v12772: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:46 smithi078 ceph-mon[91136]: pgmap v12772: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:46 smithi078 ceph-mon[99959]: pgmap v12772: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:48.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:48 smithi146 ceph-mon[103278]: pgmap v12773: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:48 smithi078 ceph-mon[91136]: pgmap v12773: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:48 smithi078 ceph-mon[99959]: pgmap v12773: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:50.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[91136]: pgmap v12774: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:50.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:50.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:50.949 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:50.950 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:25:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:25:50.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[99959]: pgmap v12774: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:50.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:50.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:50.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:50 smithi146 ceph-mon[103278]: pgmap v12774: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:51.965Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:51.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:25:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:25:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:25:51.966Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:25:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:52 smithi146 ceph-mon[103278]: pgmap v12775: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:52 smithi078 ceph-mon[91136]: pgmap v12775: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:25:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:52 smithi078 ceph-mon[99959]: pgmap v12775: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:54 smithi146 ceph-mon[103278]: pgmap v12776: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:54 smithi078 ceph-mon[91136]: pgmap v12776: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:55.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:54 smithi078 ceph-mon[99959]: pgmap v12776: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:25:55.717 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:25:56.247 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:25:56.247 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 7h - - 2024-09-21T03:25:56.247 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 7h - - 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 6s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 1s ago 7h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 6s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 1s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 6s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 6s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 1s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 6s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 6s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 1s ago 7h 15.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 6s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (6h) 6s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 6s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 6s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 1s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 1s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 1s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 1s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 1s ago 7h 71.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 6s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 6s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:25:56.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:25:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:56 smithi146 ceph-mon[103278]: pgmap v12777: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[91136]: pgmap v12777: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:25:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:56 smithi078 ceph-mon[99959]: pgmap v12777: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:56.560 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:25:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:25:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:25:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:25:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:25:56.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:25:56.561 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:25:56.829 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:25:56.830 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:25:56.830 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:25:56.830 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:25:57.142 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:25:57.142 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:57.142 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:25:57.142 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:57.142 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:25:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:57 smithi146 ceph-mon[103278]: from='client.92939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:57 smithi146 ceph-mon[103278]: from='client.92945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:57 smithi146 ceph-mon[103278]: from='client.92951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:57 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1000418172' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[91136]: from='client.92939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[91136]: from='client.92945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[91136]: from='client.92951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1000418172' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[99959]: from='client.92939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[99959]: from='client.92945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[99959]: from='client.92951 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:57 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1000418172' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:25:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:58 smithi146 ceph-mon[103278]: from='client.92963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2353012939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:58 smithi146 ceph-mon[103278]: pgmap v12778: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:58 smithi078 ceph-mon[91136]: from='client.92963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2353012939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:58 smithi078 ceph-mon[91136]: pgmap v12778: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:58 smithi078 ceph-mon[99959]: from='client.92963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:25:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2353012939' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:25:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:58 smithi078 ceph-mon[99959]: pgmap v12778: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:25:59.515 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:25:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:25:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:25:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:25:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:00 smithi146 ceph-mon[103278]: pgmap v12779: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:00.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:00 smithi078 ceph-mon[91136]: pgmap v12779: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:00 smithi078 ceph-mon[99959]: pgmap v12779: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:26:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:26:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:26:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:01.966Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:01.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:01.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:03.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:02 smithi146 ceph-mon[103278]: pgmap v12780: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:02 smithi078 ceph-mon[99959]: pgmap v12780: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:02 smithi078 ceph-mon[91136]: pgmap v12780: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:04 smithi146 ceph-mon[103278]: pgmap v12781: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:04 smithi078 ceph-mon[99959]: pgmap v12781: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:04 smithi078 ceph-mon[91136]: pgmap v12781: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:06 smithi146 ceph-mon[103278]: pgmap v12782: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:06 smithi078 ceph-mon[91136]: pgmap v12782: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:06 smithi078 ceph-mon[99959]: pgmap v12782: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:08.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:08 smithi146 ceph-mon[103278]: pgmap v12783: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:08 smithi078 ceph-mon[91136]: pgmap v12783: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:08 smithi078 ceph-mon[99959]: pgmap v12783: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[91136]: pgmap v12784: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:10.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:26:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[99959]: pgmap v12784: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:10.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:10 smithi146 ceph-mon[103278]: pgmap v12784: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:12.133 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:11.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:12.133 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:11.967Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:12.133 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:11.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:12 smithi146 ceph-mon[103278]: pgmap v12785: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:13.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:12 smithi078 ceph-mon[99959]: pgmap v12785: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:13.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:12 smithi078 ceph-mon[91136]: pgmap v12785: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:14.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:14 smithi146 ceph-mon[103278]: pgmap v12786: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:14 smithi078 ceph-mon[91136]: pgmap v12786: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:14 smithi078 ceph-mon[99959]: pgmap v12786: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:16 smithi146 ceph-mon[103278]: pgmap v12787: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[99959]: pgmap v12787: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:16.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:16 smithi078 ceph-mon[91136]: pgmap v12787: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:18.980 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:18 smithi146 ceph-mon[103278]: pgmap v12788: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:18 smithi078 ceph-mon[91136]: pgmap v12788: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:18 smithi078 ceph-mon[99959]: pgmap v12788: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:20 smithi146 ceph-mon[103278]: pgmap v12789: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:20 smithi078 ceph-mon[91136]: pgmap v12789: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:20 smithi078 ceph-mon[99959]: pgmap v12789: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:26:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:26:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:26:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:21.967Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:21.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:21.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:22 smithi146 ceph-mon[103278]: pgmap v12790: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:22 smithi078 ceph-mon[91136]: pgmap v12790: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:22 smithi078 ceph-mon[99959]: pgmap v12790: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:24.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:24 smithi078 ceph-mon[91136]: pgmap v12791: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:24 smithi078 ceph-mon[99959]: pgmap v12791: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:24 smithi146 ceph-mon[103278]: pgmap v12791: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:26 smithi146 ceph-mon[103278]: pgmap v12792: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:26 smithi078 ceph-mon[91136]: pgmap v12792: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:26 smithi078 ceph-mon[99959]: pgmap v12792: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:27.436 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:26:27.960 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:26:27.960 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 7h - - 2024-09-21T03:26:27.960 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 7h - - 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 17s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 12s ago 7h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 17s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 12s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 17s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 17s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 12s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 17s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 17s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 12s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 17s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 17s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (6h) 17s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 17s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:26:27.961 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 12s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 12s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 12s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 12s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 12s ago 7h 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 17s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 17s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:26:27.962 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:26:28.270 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:26:28.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:26:28.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:26:28.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:28.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:26:28.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:26:28.271 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:26:28.539 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:26:28.539 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:26:28.540 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:26:28.850 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:26:28.851 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:26:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:26:28.851 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:26:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:26:28.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[99959]: from='client.73059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:28.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[99959]: pgmap v12793: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:28.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2729117410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:28.851 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:28.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[91136]: from='client.73059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:28.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[91136]: pgmap v12793: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:28.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2729117410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:28.852 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:28.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:28 smithi146 ceph-mon[103278]: from='client.73059 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:28.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:28 smithi146 ceph-mon[103278]: pgmap v12793: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:28.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2729117410' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:26:28.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:29.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:29 smithi146 ceph-mon[103278]: from='client.92981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:29.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:29 smithi146 ceph-mon[103278]: from='client.73071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:29.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:29 smithi146 ceph-mon[103278]: from='client.73083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:29.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:29 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1457414098' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[91136]: from='client.92981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[91136]: from='client.73071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[91136]: from='client.73083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1457414098' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[99959]: from='client.92981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[99959]: from='client.73071 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[99959]: from='client.73083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:26:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:29 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1457414098' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:26:30.660 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:26:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:26:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:30 smithi146 ceph-mon[103278]: pgmap v12794: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:30.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:30 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[99959]: pgmap v12794: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[91136]: pgmap v12794: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:31.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:30 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:32.249 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:31.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:32.249 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:31.968Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:32.249 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:31.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:32 smithi146 ceph-mon[103278]: pgmap v12795: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:33.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:33.033 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:32 smithi078 ceph-mon[99959]: pgmap v12795: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:33.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:33.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:32 smithi078 ceph-mon[91136]: pgmap v12795: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:34 smithi146 ceph-mon[103278]: pgmap v12796: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:35.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:34 smithi078 ceph-mon[91136]: pgmap v12796: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:35.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:34 smithi078 ceph-mon[99959]: pgmap v12796: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.801 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:36 smithi146 ceph-mon[103278]: pgmap v12797: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[91136]: pgmap v12797: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.802 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:36.803 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:36 smithi078 ceph-mon[99959]: pgmap v12797: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:38 smithi146 ceph-mon[103278]: pgmap v12798: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:38 smithi078 ceph-mon[91136]: pgmap v12798: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:38 smithi078 ceph-mon[99959]: pgmap v12798: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:40 smithi146 ceph-mon[103278]: pgmap v12799: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:40 smithi078 ceph-mon[91136]: pgmap v12799: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:41.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:40 smithi078 ceph-mon[99959]: pgmap v12799: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:26:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:26:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:26:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:41.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:41.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:41.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:42 smithi146 ceph-mon[103278]: pgmap v12800: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:42 smithi078 ceph-mon[91136]: pgmap v12800: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:42 smithi078 ceph-mon[99959]: pgmap v12800: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:44.004 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:44 smithi146 ceph-mon[103278]: pgmap v12801: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:44 smithi078 ceph-mon[91136]: pgmap v12801: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:44 smithi078 ceph-mon[99959]: pgmap v12801: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:46 smithi146 ceph-mon[103278]: pgmap v12802: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:46 smithi078 ceph-mon[91136]: pgmap v12802: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:46 smithi078 ceph-mon[99959]: pgmap v12802: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:48.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:48 smithi146 ceph-mon[103278]: pgmap v12803: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:48 smithi078 ceph-mon[91136]: pgmap v12803: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:48 smithi078 ceph-mon[99959]: pgmap v12803: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:50.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:50 smithi146 ceph-mon[103278]: pgmap v12804: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:50.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.696 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:50 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:26:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:26:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[91136]: pgmap v12804: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[99959]: pgmap v12804: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:50.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:50.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:50 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:52.163 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:51.968Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:52.163 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:51.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:26:52.163 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:26:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:26:51.969Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:26:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:52 smithi146 ceph-mon[103278]: pgmap v12805: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:53.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:52 smithi078 ceph-mon[91136]: pgmap v12805: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:26:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:52 smithi078 ceph-mon[99959]: pgmap v12805: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:54.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:54 smithi078 ceph-mon[91136]: pgmap v12806: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:54.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:54 smithi078 ceph-mon[99959]: pgmap v12806: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:54 smithi146 ceph-mon[103278]: pgmap v12806: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:26:56.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.899 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:56 smithi146 ceph-mon[103278]: pgmap v12807: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:56.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[91136]: pgmap v12807: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:56.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:26:56.901 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:56 smithi078 ceph-mon[99959]: pgmap v12807: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:58 smithi146 ceph-mon[103278]: pgmap v12808: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:26:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:58 smithi078 ceph-mon[91136]: pgmap v12808: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:26:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:58 smithi078 ceph-mon[99959]: pgmap v12808: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:26:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:26:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:26:59.142 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:26:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:26:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 7h - - 2024-09-21T03:26:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 7h - - 2024-09-21T03:26:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 9s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 4s ago 7h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 9s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 4s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 9s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 9s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 4s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 9s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 9s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 4s ago 7h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:26:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 9s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 9s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 9s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6h) 9s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 4s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 4s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 4s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 4s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 4s ago 7h 72.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:26:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:26:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:26:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:26:59.980 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:26:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:26:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:26:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:26:59.980 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:26:59.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:27:00.249 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:27:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:27:00.250 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:27:00.250 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:27:00.558 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:27:00.558 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:00.558 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:00.558 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:00.559 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:00 smithi146 ceph-mon[103278]: from='client.73095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:00 smithi146 ceph-mon[103278]: from='client.93017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:00 smithi146 ceph-mon[103278]: pgmap v12809: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:00 smithi146 ceph-mon[103278]: from='client.93023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/382452758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/927086218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:27:00] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[99959]: from='client.73095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[99959]: from='client.93017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[99959]: pgmap v12809: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[99959]: from='client.93023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/382452758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/927086218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[91136]: from='client.73095 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[91136]: from='client.93017 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[91136]: pgmap v12809: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[91136]: from='client.93023 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/382452758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:01.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/927086218' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:01.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:01 smithi078 ceph-mon[91136]: from='client.73119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:01.974 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:01 smithi078 ceph-mon[99959]: from='client.73119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:01 smithi146 ceph-mon[103278]: from='client.73119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:01.969Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:01.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:01.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:02 smithi146 ceph-mon[103278]: pgmap v12810: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:02 smithi078 ceph-mon[91136]: pgmap v12810: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:02 smithi078 ceph-mon[99959]: pgmap v12810: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:04 smithi146 ceph-mon[103278]: pgmap v12811: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:05.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:04 smithi078 ceph-mon[99959]: pgmap v12811: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:04 smithi078 ceph-mon[91136]: pgmap v12811: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:06 smithi146 ceph-mon[103278]: pgmap v12812: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:07.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:06 smithi078 ceph-mon[91136]: pgmap v12812: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:07.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:06 smithi078 ceph-mon[99959]: pgmap v12812: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:08.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:08 smithi146 ceph-mon[103278]: pgmap v12813: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:08 smithi078 ceph-mon[91136]: pgmap v12813: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:08 smithi078 ceph-mon[99959]: pgmap v12813: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:10 smithi146 ceph-mon[103278]: pgmap v12814: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:10 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:27:10] "GET /metrics HTTP/1.1" 200 38998 "" "Prometheus/2.43.0" 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[91136]: pgmap v12814: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[99959]: pgmap v12814: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:10 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:11.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:11.970Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:11.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:12.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:12.931 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:12 smithi078 ceph-mon[91136]: pgmap v12815: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:12.931 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:12.932 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:12 smithi078 ceph-mon[99959]: pgmap v12815: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:12 smithi146 ceph-mon[103278]: pgmap v12815: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:13.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:13.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:14 smithi146 ceph-mon[103278]: pgmap v12816: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:14 smithi078 ceph-mon[99959]: pgmap v12816: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:14 smithi078 ceph-mon[91136]: pgmap v12816: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:16 smithi146 ceph-mon[103278]: pgmap v12817: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[91136]: pgmap v12817: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[99959]: pgmap v12817: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:18 smithi146 ceph-mon[103278]: pgmap v12818: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:19.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:18 smithi078 ceph-mon[91136]: pgmap v12818: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:18 smithi078 ceph-mon[99959]: pgmap v12818: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:20 smithi146 ceph-mon[103278]: pgmap v12819: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:21.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:20 smithi078 ceph-mon[99959]: pgmap v12819: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:20 smithi078 ceph-mon[91136]: pgmap v12819: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:27:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:27:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:27:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:21.970Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:21.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:21.971Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:22 smithi146 ceph-mon[103278]: pgmap v12820: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:22 smithi078 ceph-mon[91136]: pgmap v12820: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:22 smithi078 ceph-mon[99959]: pgmap v12820: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:24.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:24 smithi078 ceph-mon[99959]: pgmap v12821: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:24.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:24 smithi078 ceph-mon[91136]: pgmap v12821: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:24 smithi146 ceph-mon[103278]: pgmap v12821: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:26 smithi146 ceph-mon[103278]: pgmap v12822: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:26 smithi078 ceph-mon[91136]: pgmap v12822: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:26 smithi078 ceph-mon[99959]: pgmap v12822: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:28 smithi146 ceph-mon[103278]: pgmap v12823: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:28 smithi078 ceph-mon[91136]: pgmap v12823: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:28 smithi078 ceph-mon[99959]: pgmap v12823: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:30.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:30 smithi146 ceph-mon[103278]: pgmap v12824: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:30.862 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:27:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:27:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:27:30.863 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:30 smithi078 ceph-mon[99959]: pgmap v12824: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:30.863 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:30 smithi078 ceph-mon[91136]: pgmap v12824: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:30.863 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:27:31.391 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:27:31.391 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 7h - - 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 7h - - 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 0s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 15s ago 7h 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 0s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 15s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 0s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 0s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 15s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 0s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 0s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:27:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 15s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 0s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 0s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 0s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 0s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (6h) 15s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 15s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 15s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 15s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 15s ago 7h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 0s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:27:31.393 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 0s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:27:31.394 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:27:31.703 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:27:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:27:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:27:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:27:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:27:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:27:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:27:31.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:27:31.973 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:27:31.973 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:27:31.973 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:27:31.973 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:27:31.973 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:27:31.974 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:27:31.974 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:27:31.974 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:27:31.974 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:27:31.974 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:27:31.974 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:27:31.975 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:27:31.975 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:27:31.975 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:27:31.975 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:31.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:31.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:31.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='client.73131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:31.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='client.73137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:31.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:31.976 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='client.73143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:31.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: pgmap v12825: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:31.977 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1028228455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:31.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:31.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:31.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:31.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='client.73131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:31.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='client.73137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:31.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:31.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='client.73143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:31.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: pgmap v12825: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:31.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1028228455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='client.73131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='client.73137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='client.73143 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: pgmap v12825: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:27:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1028228455' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:27:32.281 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:27:32.281 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:32.282 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:27:32.282 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:32.282 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:27:32.282 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:31.972Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:32.282 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:31.972Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:32.282 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:31.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:32 smithi078 ceph-mon[91136]: from='client.93071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1137157767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:32 smithi078 ceph-mon[99959]: from='client.93071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1137157767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:32 smithi146 ceph-mon[103278]: from='client.93071 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:27:33.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1137157767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:27:34.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:33 smithi078 ceph-mon[91136]: pgmap v12826: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:33 smithi078 ceph-mon[99959]: pgmap v12826: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:33 smithi146 ceph-mon[103278]: pgmap v12826: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:36 smithi146 ceph-mon[103278]: pgmap v12827: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[91136]: pgmap v12827: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[99959]: pgmap v12827: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:38 smithi146 ceph-mon[103278]: pgmap v12828: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:39.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:38 smithi078 ceph-mon[91136]: pgmap v12828: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:38 smithi078 ceph-mon[99959]: pgmap v12828: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:40 smithi146 ceph-mon[103278]: pgmap v12829: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:40 smithi078 ceph-mon[91136]: pgmap v12829: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:40 smithi078 ceph-mon[99959]: pgmap v12829: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:27:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:27:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:27:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:41.973Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:41.973Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:41.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:42 smithi146 ceph-mon[103278]: pgmap v12830: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:42 smithi078 ceph-mon[91136]: pgmap v12830: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:42 smithi078 ceph-mon[99959]: pgmap v12830: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:43.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:44 smithi146 ceph-mon[103278]: pgmap v12831: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:44 smithi078 ceph-mon[91136]: pgmap v12831: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:44 smithi078 ceph-mon[99959]: pgmap v12831: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:46 smithi146 ceph-mon[103278]: pgmap v12832: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:47.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:46 smithi078 ceph-mon[91136]: pgmap v12832: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:46 smithi078 ceph-mon[99959]: pgmap v12832: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:48 smithi146 ceph-mon[103278]: pgmap v12833: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:48 smithi078 ceph-mon[91136]: pgmap v12833: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:48 smithi078 ceph-mon[99959]: pgmap v12833: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:50.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:50 smithi078 ceph-mon[99959]: pgmap v12834: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:50.923 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:27:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:27:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:27:50.924 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:50 smithi078 ceph-mon[91136]: pgmap v12834: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:50 smithi146 ceph-mon[103278]: pgmap v12834: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:51 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:51 smithi146 ceph-mon[103278]: pgmap v12835: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[91136]: pgmap v12835: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:27:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-mon[99959]: pgmap v12835: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:52.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:51.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:52.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:51.974Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:27:52.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:27:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:27:51.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:27:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:54 smithi146 ceph-mon[103278]: pgmap v12836: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:55.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:54 smithi078 ceph-mon[91136]: pgmap v12836: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:54 smithi078 ceph-mon[99959]: pgmap v12836: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:27:56.960 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[99959]: pgmap v12837: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[91136]: pgmap v12837: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:56.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:56 smithi146 ceph-mon[103278]: pgmap v12837: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:27:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:58 smithi146 ceph-mon[103278]: pgmap v12838: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:27:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:58 smithi078 ceph-mon[91136]: pgmap v12838: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:27:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:27:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:58 smithi078 ceph-mon[99959]: pgmap v12838: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:27:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:27:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:00 smithi146 ceph-mon[103278]: pgmap v12839: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:00 smithi078 ceph-mon[91136]: pgmap v12839: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:00 smithi078 ceph-mon[99959]: pgmap v12839: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:28:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:28:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:28:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:01.974Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:01.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:01.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:02.572 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:02 smithi146 ceph-mon[103278]: pgmap v12840: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:03.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:02 smithi078 ceph-mon[99959]: pgmap v12840: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:02 smithi078 ceph-mon[91136]: pgmap v12840: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 7h - - 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 7h - - 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 12s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 7s ago 7h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 12s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 7s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 12s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 12s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:28:03.099 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 7s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 12s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 12s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 7s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 12s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 12s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 12s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 12s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 7s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 7s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 7s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 7s ago 7h 97.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 7s ago 7h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 12s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 12s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:28:03.100 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:28:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:28:03.413 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:28:03.679 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:28:03.680 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:28:03.990 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:28:03.991 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:28:03.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:28:03.991 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:28:03.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:28:03.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:03 smithi078 ceph-mon[99959]: from='client.93083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:03.991 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4226131350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:03.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:03 smithi078 ceph-mon[91136]: from='client.93083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:03.991 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4226131350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:03 smithi146 ceph-mon[103278]: from='client.93083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4226131350' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:04 smithi146 ceph-mon[103278]: from='client.73173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:04 smithi146 ceph-mon[103278]: from='client.73179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:04 smithi146 ceph-mon[103278]: pgmap v12841: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:04 smithi146 ceph-mon[103278]: from='client.73191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:04 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2832581796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[91136]: from='client.73173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[91136]: from='client.73179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[91136]: pgmap v12841: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[91136]: from='client.73191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2832581796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[99959]: from='client.73173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[99959]: from='client.73179 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[99959]: pgmap v12841: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:05.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[99959]: from='client.73191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:05.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:04 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2832581796' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:07.003 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:06 smithi078 ceph-mon[99959]: pgmap v12842: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:07.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:06 smithi078 ceph-mon[91136]: pgmap v12842: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:06 smithi146 ceph-mon[103278]: pgmap v12842: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:08 smithi146 ceph-mon[103278]: pgmap v12843: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:08 smithi078 ceph-mon[91136]: pgmap v12843: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:08 smithi078 ceph-mon[99959]: pgmap v12843: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:10 smithi146 ceph-mon[103278]: pgmap v12844: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:11.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:10 smithi078 ceph-mon[99959]: pgmap v12844: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:10 smithi078 ceph-mon[91136]: pgmap v12844: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:28:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:28:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:28:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:12.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:12 smithi146 ceph-mon[103278]: pgmap v12845: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[99959]: pgmap v12845: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:12 smithi078 ceph-mon[91136]: pgmap v12845: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:12.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:11.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:12.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:11.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:12.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:11.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:14.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:15.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:14 smithi078 ceph-mon[99959]: pgmap v12846: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:15.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:14 smithi078 ceph-mon[91136]: pgmap v12846: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:14 smithi146 ceph-mon[103278]: pgmap v12846: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[91136]: pgmap v12847: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[99959]: pgmap v12847: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:16 smithi146 ceph-mon[103278]: pgmap v12847: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:18 smithi146 ceph-mon[103278]: pgmap v12848: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:19.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:18 smithi078 ceph-mon[99959]: pgmap v12848: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:18 smithi078 ceph-mon[91136]: pgmap v12848: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:20 smithi146 ceph-mon[103278]: pgmap v12849: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:21.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:20 smithi078 ceph-mon[91136]: pgmap v12849: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:20 smithi078 ceph-mon[99959]: pgmap v12849: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:28:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:28:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:28:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:21.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:21.975Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:21.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:22 smithi146 ceph-mon[103278]: pgmap v12850: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:22 smithi078 ceph-mon[99959]: pgmap v12850: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:22 smithi078 ceph-mon[91136]: pgmap v12850: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:24.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:24 smithi078 ceph-mon[91136]: pgmap v12851: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:24.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:24 smithi078 ceph-mon[99959]: pgmap v12851: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:24 smithi146 ceph-mon[103278]: pgmap v12851: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:26 smithi146 ceph-mon[103278]: pgmap v12852: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:27.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:26 smithi078 ceph-mon[91136]: pgmap v12852: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:26 smithi078 ceph-mon[99959]: pgmap v12852: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:28 smithi146 ceph-mon[103278]: pgmap v12853: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:28 smithi078 ceph-mon[99959]: pgmap v12853: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:28 smithi078 ceph-mon[91136]: pgmap v12853: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:29 smithi146 ceph-mon[103278]: pgmap v12854: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:29 smithi078 ceph-mon[91136]: pgmap v12854: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:29 smithi078 ceph-mon[99959]: pgmap v12854: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:28:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:28:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[99959]: pgmap v12855: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:32.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:32 smithi078 ceph-mon[91136]: pgmap v12855: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:31.975Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:31.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:31.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:32.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.473 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.474 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:32.474 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:32.474 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:32 smithi146 ceph-mon[103278]: pgmap v12855: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:34.279 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:28:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:34 smithi146 ceph-mon[103278]: pgmap v12856: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:34 smithi078 ceph-mon[91136]: pgmap v12856: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:34 smithi078 ceph-mon[99959]: pgmap v12856: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:34.807 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:28:34.807 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 7h - - 2024-09-21T03:28:34.807 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 7h - - 2024-09-21T03:28:34.807 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 3s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 18s ago 7h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 3s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 18s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 3s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 3s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 18s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 3s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 3s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 18s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 3s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 3s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 3s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:28:34.808 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 3s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 18s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (6h) 18s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 18s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 18s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 18s ago 7h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 3s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 3s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:28:34.809 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:28:35.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:28:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:28:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:28:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:35.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:28:35.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:28:35.391 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:28:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:28:35.391 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:28:35.392 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:28:35.725 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:28:35.725 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:28:35.725 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:28:35.726 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:28:35.726 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:28:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:35 smithi146 ceph-mon[103278]: from='client.73203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:35 smithi146 ceph-mon[103278]: from='client.93125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3447497789' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:35 smithi078 ceph-mon[99959]: from='client.73203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:35 smithi078 ceph-mon[99959]: from='client.93125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3447497789' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:35 smithi078 ceph-mon[91136]: from='client.73203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:35 smithi078 ceph-mon[91136]: from='client.93125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3447497789' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:28:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: from='client.93131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: from='client.93143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: pgmap v12857: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/547890141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: from='client.93131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: from='client.93143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: pgmap v12857: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/547890141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: from='client.93131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: from='client.93143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: pgmap v12857: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/547890141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:28:37.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:37.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:38 smithi146 ceph-mon[103278]: pgmap v12858: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:38 smithi078 ceph-mon[91136]: pgmap v12858: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:38 smithi078 ceph-mon[99959]: pgmap v12858: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:41.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:40 smithi146 ceph-mon[103278]: pgmap v12859: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:41.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:40 smithi078 ceph-mon[91136]: pgmap v12859: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:28:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:28:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:28:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:40 smithi078 ceph-mon[99959]: pgmap v12859: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:41.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:41.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:41.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:42 smithi146 ceph-mon[103278]: pgmap v12860: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:42 smithi078 ceph-mon[91136]: pgmap v12860: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:42 smithi078 ceph-mon[99959]: pgmap v12860: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:44 smithi146 ceph-mon[103278]: pgmap v12861: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:44 smithi078 ceph-mon[99959]: pgmap v12861: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:44 smithi078 ceph-mon[91136]: pgmap v12861: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:45 smithi146 ceph-mon[103278]: pgmap v12862: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:45 smithi078 ceph-mon[99959]: pgmap v12862: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:45 smithi078 ceph-mon[91136]: pgmap v12862: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:48 smithi146 ceph-mon[103278]: pgmap v12863: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:48 smithi078 ceph-mon[91136]: pgmap v12863: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:48 smithi078 ceph-mon[99959]: pgmap v12863: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:50 smithi146 ceph-mon[103278]: pgmap v12864: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:50 smithi078 ceph-mon[91136]: pgmap v12864: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:50 smithi078 ceph-mon[99959]: pgmap v12864: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:28:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:28:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:28:52.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:51.976Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:51.976Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:28:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:28:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:28:51.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:28:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:52 smithi146 ceph-mon[103278]: pgmap v12865: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[99959]: pgmap v12865: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:52.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:52.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:28:52.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:52 smithi078 ceph-mon[91136]: pgmap v12865: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:28:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:54 smithi146 ceph-mon[103278]: pgmap v12866: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:55.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:54 smithi078 ceph-mon[91136]: pgmap v12866: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:55.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:54 smithi078 ceph-mon[99959]: pgmap v12866: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[99959]: pgmap v12867: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[91136]: pgmap v12867: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:56.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:56 smithi146 ceph-mon[103278]: pgmap v12867: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:28:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:58 smithi146 ceph-mon[103278]: pgmap v12868: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:28:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:58 smithi078 ceph-mon[91136]: pgmap v12868: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:28:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:28:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:58 smithi078 ceph-mon[99959]: pgmap v12868: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:28:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:28:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:00 smithi146 ceph-mon[103278]: pgmap v12869: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:00 smithi078 ceph-mon[91136]: pgmap v12869: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:00 smithi078 ceph-mon[99959]: pgmap v12869: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:29:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:29:00] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:29:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:01.977Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:01.977Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:01.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:02.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:02.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:02 smithi078 ceph-mon[99959]: pgmap v12870: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:02.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:02.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:02 smithi078 ceph-mon[91136]: pgmap v12870: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:02 smithi146 ceph-mon[103278]: pgmap v12870: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:05.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:04 smithi146 ceph-mon[103278]: pgmap v12871: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:05.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:04 smithi078 ceph-mon[91136]: pgmap v12871: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:05.171 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:04 smithi078 ceph-mon[99959]: pgmap v12871: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:06.021 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:29:06.546 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:29:06.547 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 7h - - 2024-09-21T03:29:06.547 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 7h - - 2024-09-21T03:29:06.547 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 15s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:29:06.547 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 10s ago 7h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:29:06.547 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 15s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:29:06.547 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 10s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 15s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 15s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 10s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 15s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 15s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 10s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 15s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 15s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 15s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 15s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:29:06.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 10s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 10s ago 7h 93.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 10s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 10s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 10s ago 7h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 15s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 15s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:29:06.549 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:29:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:29:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:29:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:29:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:29:06.862 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:29:06.862 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:06 smithi078 ceph-mon[99959]: pgmap v12872: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:06.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:06 smithi078 ceph-mon[91136]: pgmap v12872: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:07.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:06 smithi146 ceph-mon[103278]: pgmap v12872: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:29:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:29:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:29:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:29:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:29:07.132 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:29:07.440 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:29:07.440 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:07.440 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:07.440 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:07.440 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:07 smithi146 ceph-mon[103278]: from='client.73239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:07 smithi146 ceph-mon[103278]: from='client.73245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:07 smithi146 ceph-mon[103278]: from='client.93167 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2245122758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:07 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3640080444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[99959]: from='client.73239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[99959]: from='client.73245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[99959]: from='client.93167 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2245122758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3640080444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[91136]: from='client.73239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[91136]: from='client.73245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[91136]: from='client.93167 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2245122758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:07 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3640080444' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:08 smithi146 ceph-mon[103278]: from='client.93179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:09.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:08 smithi146 ceph-mon[103278]: pgmap v12873: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:09.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:08 smithi078 ceph-mon[91136]: from='client.93179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:09.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:08 smithi078 ceph-mon[91136]: pgmap v12873: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:08 smithi078 ceph-mon[99959]: from='client.93179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:09.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:08 smithi078 ceph-mon[99959]: pgmap v12873: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:09 smithi146 ceph-mon[103278]: pgmap v12874: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:10.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:09 smithi078 ceph-mon[91136]: pgmap v12874: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:09 smithi078 ceph-mon[99959]: pgmap v12874: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:29:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:29:10] "GET /metrics HTTP/1.1" 200 39001 "" "Prometheus/2.43.0" 2024-09-21T03:29:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:11.978Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:11.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:11.978Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:12 smithi146 ceph-mon[103278]: pgmap v12875: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[91136]: pgmap v12875: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:12.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:12 smithi078 ceph-mon[99959]: pgmap v12875: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:14.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:14 smithi146 ceph-mon[103278]: pgmap v12876: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:15.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:14 smithi078 ceph-mon[91136]: pgmap v12876: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:14 smithi078 ceph-mon[99959]: pgmap v12876: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:16 smithi146 ceph-mon[103278]: pgmap v12877: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[91136]: pgmap v12877: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:17.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[99959]: pgmap v12877: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:18.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:18 smithi078 ceph-mon[91136]: pgmap v12878: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:18.944 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:18 smithi078 ceph-mon[99959]: pgmap v12878: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:19.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:18 smithi146 ceph-mon[103278]: pgmap v12878: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:20.871 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:20 smithi078 ceph-mon[91136]: pgmap v12879: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:20.871 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:20 smithi078 ceph-mon[99959]: pgmap v12879: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:20.872 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:29:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:29:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:29:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:20 smithi146 ceph-mon[103278]: pgmap v12879: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:21 smithi146 ceph-mon[103278]: pgmap v12880: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:22.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-mon[91136]: pgmap v12880: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-mon[99959]: pgmap v12880: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:29:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:21.979Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:21.979Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:21.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:25.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:24 smithi146 ceph-mon[103278]: pgmap v12881: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:25.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:24 smithi078 ceph-mon[91136]: pgmap v12881: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:25.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:24 smithi078 ceph-mon[99959]: pgmap v12881: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:27.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:26 smithi146 ceph-mon[103278]: pgmap v12882: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:27.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:26 smithi078 ceph-mon[91136]: pgmap v12882: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:27.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:26 smithi078 ceph-mon[99959]: pgmap v12882: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:28 smithi146 ceph-mon[103278]: pgmap v12883: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:28 smithi078 ceph-mon[91136]: pgmap v12883: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:29.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:28 smithi078 ceph-mon[99959]: pgmap v12883: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:29 smithi078 ceph-mon[91136]: pgmap v12884: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:29 smithi078 ceph-mon[99959]: pgmap v12884: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:29 smithi146 ceph-mon[103278]: pgmap v12884: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:29:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:29:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:29:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:31.980Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:31.980Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:31.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:32.854 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:32.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:32.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:32.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:32.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:32 smithi146 ceph-mon[103278]: pgmap v12885: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[91136]: pgmap v12885: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:32 smithi078 ceph-mon[99959]: pgmap v12885: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:34.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:34 smithi146 ceph-mon[103278]: pgmap v12886: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:35.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:34 smithi078 ceph-mon[91136]: pgmap v12886: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:35.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:34 smithi078 ceph-mon[99959]: pgmap v12886: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:37.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:36 smithi146 ceph-mon[103278]: pgmap v12887: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[91136]: pgmap v12887: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:37.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[99959]: pgmap v12887: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:37.738 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:29:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:29:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 7h - - 2024-09-21T03:29:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 7h - - 2024-09-21T03:29:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 6s ago 7h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:29:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 1s ago 7h 88.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:29:38.312 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 6s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 1s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 6s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 6s ago 7h 560M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 1s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 6s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 6s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 1s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 6s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 6s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 6s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 6s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 1s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 1s ago 7h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:29:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (6h) 1s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:29:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 1s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:29:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 1s ago 7h 73.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:29:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 6s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:29:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:29:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 6s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:29:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:29:38.626 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:29:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:29:38.627 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:29:38.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:38 smithi078 ceph-mon[91136]: pgmap v12888: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:38.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/30893680' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:38.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:38 smithi078 ceph-mon[99959]: pgmap v12888: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:38.908 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/30893680' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:38.908 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:29:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:29:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:29:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:29:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:29:38.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:29:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:38 smithi146 ceph-mon[103278]: pgmap v12888: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/30893680' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:29:39.232 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:29:39.232 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:29:39.232 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:29:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:39 smithi146 ceph-mon[103278]: from='client.73275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:39 smithi146 ceph-mon[103278]: from='client.73281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:39 smithi146 ceph-mon[103278]: from='client.73287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:39 smithi146 ceph-mon[103278]: from='client.73299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2993036415' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:39 smithi146 ceph-mon[103278]: pgmap v12889: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:40.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[99959]: from='client.73275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[99959]: from='client.73281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[99959]: from='client.73287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[99959]: from='client.73299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2993036415' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[99959]: pgmap v12889: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[91136]: from='client.73275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[91136]: from='client.73281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[91136]: from='client.73287 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[91136]: from='client.73299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:29:40.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2993036415' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:29:40.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:39 smithi078 ceph-mon[91136]: pgmap v12889: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:29:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:29:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:29:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:41.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:41.981Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:41.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:42 smithi146 ceph-mon[103278]: pgmap v12890: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:42 smithi078 ceph-mon[91136]: pgmap v12890: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:42 smithi078 ceph-mon[99959]: pgmap v12890: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:44 smithi146 ceph-mon[103278]: pgmap v12891: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:44 smithi078 ceph-mon[91136]: pgmap v12891: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:44 smithi078 ceph-mon[99959]: pgmap v12891: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:45 smithi146 ceph-mon[103278]: pgmap v12892: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:46.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:45 smithi078 ceph-mon[91136]: pgmap v12892: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:46.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:45 smithi078 ceph-mon[99959]: pgmap v12892: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:48 smithi146 ceph-mon[103278]: pgmap v12893: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:48 smithi078 ceph-mon[91136]: pgmap v12893: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:49.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:48 smithi078 ceph-mon[99959]: pgmap v12893: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:51.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:50 smithi146 ceph-mon[103278]: pgmap v12894: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:50 smithi078 ceph-mon[91136]: pgmap v12894: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:51.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:50 smithi078 ceph-mon[99959]: pgmap v12894: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:29:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:29:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:29:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:51.981Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:51.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:29:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:29:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:29:51.982Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:29:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:52 smithi146 ceph-mon[103278]: pgmap v12895: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:52 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[99959]: pgmap v12895: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:29:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[91136]: pgmap v12895: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:52 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:55.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:54 smithi146 ceph-mon[103278]: pgmap v12896: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:55.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:54 smithi078 ceph-mon[91136]: pgmap v12896: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:55.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:54 smithi078 ceph-mon[99959]: pgmap v12896: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:29:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:56 smithi146 ceph-mon[103278]: pgmap v12897: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[91136]: pgmap v12897: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[99959]: pgmap v12897: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:29:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:57 smithi078 ceph-mon[91136]: pgmap v12898: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:57 smithi078 ceph-mon[99959]: pgmap v12898: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:57 smithi146 ceph-mon[103278]: pgmap v12898: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:29:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:29:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:00.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:29:59 smithi078 ceph-mon[91136]: pgmap v12899: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:00.001 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:29:59 smithi078 ceph-mon[99959]: pgmap v12899: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:29:59 smithi146 ceph-mon[103278]: pgmap v12899: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:00.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T03:29:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:00.779 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:30:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:30:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:30:01.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:01 smithi146 ceph-mon[103278]: pgmap v12900: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:02.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-mon[91136]: pgmap v12900: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-mon[99959]: pgmap v12900: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:01.982Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:01.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:02.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:01.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:03 smithi078 ceph-mon[91136]: pgmap v12901: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:03 smithi078 ceph-mon[99959]: pgmap v12901: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:03 smithi146 ceph-mon[103278]: pgmap v12901: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:05 smithi146 ceph-mon[103278]: pgmap v12902: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:05 smithi078 ceph-mon[91136]: pgmap v12902: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:05 smithi078 ceph-mon[99959]: pgmap v12902: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:07 smithi078 ceph-mon[99959]: pgmap v12903: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:07 smithi078 ceph-mon[91136]: pgmap v12903: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:07 smithi146 ceph-mon[103278]: pgmap v12903: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:09.525 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:30:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:09 smithi078 ceph-mon[99959]: from='client.73311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:09 smithi078 ceph-mon[99959]: pgmap v12904: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:09 smithi078 ceph-mon[91136]: from='client.73311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:09 smithi078 ceph-mon[91136]: pgmap v12904: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:10.051 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:30:10.051 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 7h - - 2024-09-21T03:30:10.051 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 7h - - 2024-09-21T03:30:10.051 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 18s ago 7h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:30:10.051 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 13s ago 7h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:30:10.051 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 18s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 13s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 18s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 18s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 13s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 18s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 13s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 18s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 18s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 18s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 18s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 13s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 13s ago 7h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:30:10.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 13s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:30:10.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (6h) 13s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:30:10.053 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 13s ago 7h 68.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:30:10.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 18s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:30:10.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:30:10.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 18s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:30:10.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:30:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:09 smithi146 ceph-mon[103278]: from='client.73311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:09 smithi146 ceph-mon[103278]: pgmap v12904: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:10.367 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:30:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:30:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:30:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:10.367 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:30:10.368 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:30:10.635 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:30:10.636 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:30:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:30:10.946 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:10.946 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:10.946 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:10.946 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:10.947 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[99959]: from='client.93233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[99959]: from='client.93239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2127371731' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[99959]: from='client.73335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[91136]: from='client.93233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[91136]: from='client.93239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:10.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2127371731' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:10.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:10 smithi078 ceph-mon[91136]: from='client.73335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:11.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:10 smithi146 ceph-mon[103278]: from='client.93233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:10 smithi146 ceph-mon[103278]: from='client.93239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2127371731' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:11.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:10 smithi146 ceph-mon[103278]: from='client.73335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2955378992' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-mon[91136]: pgmap v12905: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2955378992' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-mon[99959]: pgmap v12905: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:11.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:11.983Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:11.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2955378992' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:12.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:11 smithi146 ceph-mon[103278]: pgmap v12905: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:13.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:12 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:12 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:13.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:12 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:14.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:14.152 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:13 smithi146 ceph-mon[103278]: pgmap v12906: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:13 smithi078 ceph-mon[91136]: pgmap v12906: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:13 smithi078 ceph-mon[99959]: pgmap v12906: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:15.902 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:15 smithi146 ceph-mon[103278]: pgmap v12907: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:16.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:15 smithi078 ceph-mon[91136]: pgmap v12907: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:15 smithi078 ceph-mon[99959]: pgmap v12907: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[91136]: pgmap v12908: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:17 smithi078 ceph-mon[99959]: pgmap v12908: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:17 smithi146 ceph-mon[103278]: pgmap v12908: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:20.034 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:19 smithi078 ceph-mon[91136]: pgmap v12909: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:20.034 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:19 smithi078 ceph-mon[99959]: pgmap v12909: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:19 smithi146 ceph-mon[103278]: pgmap v12909: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:20.644 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:30:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:30:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:30:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:22.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-mon[99959]: pgmap v12910: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-mon[91136]: pgmap v12910: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:21.983Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:21.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:22.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:21.984Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:21 smithi146 ceph-mon[103278]: pgmap v12910: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:23 smithi078 ceph-mon[91136]: pgmap v12911: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:23 smithi078 ceph-mon[99959]: pgmap v12911: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:23 smithi146 ceph-mon[103278]: pgmap v12911: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:26.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:25 smithi078 ceph-mon[99959]: pgmap v12912: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:25 smithi078 ceph-mon[91136]: pgmap v12912: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:25 smithi146 ceph-mon[103278]: pgmap v12912: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:27 smithi078 ceph-mon[91136]: pgmap v12913: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:27 smithi078 ceph-mon[99959]: pgmap v12913: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:27 smithi146 ceph-mon[103278]: pgmap v12913: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:29.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:30.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:29 smithi078 ceph-mon[91136]: pgmap v12914: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:29 smithi078 ceph-mon[99959]: pgmap v12914: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:29 smithi146 ceph-mon[103278]: pgmap v12914: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:30:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:30:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:30:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-mon[91136]: pgmap v12915: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-mon[99959]: pgmap v12915: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:31.984Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:32.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:31.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:32.037 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:31.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:31 smithi146 ceph-mon[103278]: pgmap v12915: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:33.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:32 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:32 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:32 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:34.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:33 smithi146 ceph-mon[103278]: pgmap v12916: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:34.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:33 smithi078 ceph-mon[91136]: pgmap v12916: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:34.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:33 smithi078 ceph-mon[99959]: pgmap v12916: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:35 smithi146 ceph-mon[103278]: pgmap v12917: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:35 smithi078 ceph-mon[91136]: pgmap v12917: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:35 smithi078 ceph-mon[99959]: pgmap v12917: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:37 smithi146 ceph-mon[103278]: pgmap v12918: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:38.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[99959]: pgmap v12918: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:38.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:37 smithi078 ceph-mon[91136]: pgmap v12918: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:39.896 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:39 smithi078 ceph-mon[99959]: pgmap v12919: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:39.896 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:39 smithi078 ceph-mon[91136]: pgmap v12919: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:39 smithi146 ceph-mon[103278]: pgmap v12919: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:40.909 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:30:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:30:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:30:41.240 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:30:41.791 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 7h - - 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 7h - - 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 9s ago 7h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 4s ago 7h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 9s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 4s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 9s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 9s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 4s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 9s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 9s ago 7h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 4s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 9s ago 7h 98.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:30:41.792 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 9s ago 7h 105M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 9s ago 7h 85.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 9s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 4s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 4s ago 7h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 4s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 4s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 4s ago 7h 69.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (6h) 9s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (6h) 9s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:30:41.793 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (6h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:30:41.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[99959]: from='client.93263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:41.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:41.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[99959]: from='client.93269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:41.843 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[99959]: pgmap v12920: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:41.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[91136]: from='client.93263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:41.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:41.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[91136]: from='client.93269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:41.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-mon[91136]: pgmap v12920: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:42.117 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:41.985Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:42.117 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:41.985Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:42.117 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:41.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:30:42.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:30:42.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:30:42.119 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:30:42.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:30:42.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:30:42.119 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:30:42.148 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:41 smithi146 ceph-mon[103278]: from='client.93263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:42.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:42.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:41 smithi146 ceph-mon[103278]: from='client.93269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:42.149 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:41 smithi146 ceph-mon[103278]: pgmap v12920: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:42.396 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:30:42.397 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:30:42.715 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:30:42.716 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:30:42.716 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:30:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[91136]: from='client.93275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/651438643' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[91136]: from='client.93287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2887495884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[99959]: from='client.93275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/651438643' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[99959]: from='client.93287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2887495884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:42 smithi146 ceph-mon[103278]: from='client.93275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/651438643' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:30:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:42 smithi146 ceph-mon[103278]: from='client.93287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:30:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2887495884' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:30:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:43 smithi078 ceph-mon[91136]: pgmap v12921: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:43 smithi078 ceph-mon[99959]: pgmap v12921: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:43 smithi146 ceph-mon[103278]: pgmap v12921: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:45 smithi078 ceph-mon[91136]: pgmap v12922: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:45 smithi078 ceph-mon[99959]: pgmap v12922: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:45 smithi146 ceph-mon[103278]: pgmap v12922: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:48.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:47 smithi078 ceph-mon[99959]: pgmap v12923: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:47 smithi078 ceph-mon[91136]: pgmap v12923: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:47 smithi146 ceph-mon[103278]: pgmap v12923: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:49 smithi078 ceph-mon[91136]: pgmap v12924: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:49 smithi078 ceph-mon[99959]: pgmap v12924: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:49 smithi146 ceph-mon[103278]: pgmap v12924: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:30:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:30:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:30:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:30:52.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-mon[91136]: pgmap v12925: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-mon[99959]: pgmap v12925: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:51.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:51.986Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:30:52.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:30:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:30:51.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:30:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:30:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:51 smithi146 ceph-mon[103278]: pgmap v12925: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.357 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.357 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:53.357 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:54 smithi146 ceph-mon[103278]: pgmap v12926: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:30:54.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:54 smithi078 ceph-mon[91136]: pgmap v12926: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:30:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:54 smithi078 ceph-mon[99959]: pgmap v12926: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:30:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:55 smithi146 ceph-mon[103278]: pgmap v12927: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:30:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:55 smithi078 ceph-mon[99959]: pgmap v12927: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:30:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:55 smithi078 ceph-mon[91136]: pgmap v12927: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:30:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:58 smithi146 ceph-mon[103278]: pgmap v12928: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:30:58.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[91136]: pgmap v12928: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:30:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:58 smithi078 ceph-mon[99959]: pgmap v12928: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:30:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:30:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:30:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:30:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:30:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:00.367 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:00 smithi078 ceph-mon[91136]: pgmap v12929: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T03:31:00.367 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:00 smithi078 ceph-mon[99959]: pgmap v12929: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T03:31:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:00 smithi146 ceph-mon[103278]: pgmap v12929: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-21T03:31:00.679 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:31:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:31:00] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:31:01.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:01.988 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-mon[99959]: pgmap v12930: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T03:31:01.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:01.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-mon[91136]: pgmap v12930: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T03:31:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:01 smithi146 ceph-mon[103278]: pgmap v12930: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 38 op/s 2024-09-21T03:31:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:01.986Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:01.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:01.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:04.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:03 smithi078 ceph-mon[91136]: pgmap v12931: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-21T03:31:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:03 smithi078 ceph-mon[99959]: pgmap v12931: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-21T03:31:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:03 smithi146 ceph-mon[103278]: pgmap v12931: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-21T03:31:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:05 smithi078 ceph-mon[91136]: pgmap v12932: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:31:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:05 smithi078 ceph-mon[99959]: pgmap v12932: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:31:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:05 smithi146 ceph-mon[103278]: pgmap v12932: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:31:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:07 smithi078 ceph-mon[91136]: pgmap v12933: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:31:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:07 smithi078 ceph-mon[99959]: pgmap v12933: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:31:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:07 smithi146 ceph-mon[103278]: pgmap v12933: 161 pgs: 161 active+clean; 585 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-09-21T03:31:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:09 smithi078 ceph-mon[91136]: pgmap v12934: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T03:31:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:09 smithi078 ceph-mon[99959]: pgmap v12934: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T03:31:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:09 smithi146 ceph-mon[103278]: pgmap v12934: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T03:31:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:31:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:31:10] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:31:12.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-mon[91136]: pgmap v12935: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T03:31:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-mon[99959]: pgmap v12935: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T03:31:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:11.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:11.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:12.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:11.987Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:11 smithi146 ceph-mon[103278]: pgmap v12935: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 100 op/s 2024-09-21T03:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:31:13.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 7h - - 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 7h - - 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 1s ago 7h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 16s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 1s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 16s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 1s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 1s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 16s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 1s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 1s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 16s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:31:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 1s ago 7h 98.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 1s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 1s ago 7h 86.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 1s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 16s ago 7h 121M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 16s ago 7h 94.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 16s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 16s ago 7h 97.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 16s ago 7h 69.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 1s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 1s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:31:13.530 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:31:13.837 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:31:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:31:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:31:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:31:13.851 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:31:14.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:31:14.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:31:14.420 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:31:14.420 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:14.420 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:14.420 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:14.420 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[99959]: from='client.73383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[99959]: from='client.73389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[99959]: from='client.73395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[99959]: pgmap v12936: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3113932932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[91136]: from='client.73383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[91136]: from='client.73389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[91136]: from='client.73395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:14.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[91136]: pgmap v12936: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-21T03:31:14.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3113932932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:14 smithi146 ceph-mon[103278]: from='client.73383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:14 smithi146 ceph-mon[103278]: from='client.73389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:14 smithi146 ceph-mon[103278]: from='client.73395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:14 smithi146 ceph-mon[103278]: pgmap v12936: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-21T03:31:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3113932932' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:15.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:15 smithi146 ceph-mon[103278]: from='client.93323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:15.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/954387596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:15.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:15 smithi078 ceph-mon[99959]: from='client.93323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/954387596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:15 smithi078 ceph-mon[91136]: from='client.93323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/954387596' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:16.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:16 smithi146 ceph-mon[103278]: pgmap v12937: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-21T03:31:16.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:16 smithi078 ceph-mon[91136]: pgmap v12937: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-21T03:31:16.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:16 smithi078 ceph-mon[99959]: pgmap v12937: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-21T03:31:18.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.443 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:18 smithi146 ceph-mon[103278]: pgmap v12938: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:31:18.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[91136]: pgmap v12938: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:18 smithi078 ceph-mon[99959]: pgmap v12938: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:31:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:19 smithi078 ceph-mon[91136]: pgmap v12939: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-21T03:31:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:19 smithi078 ceph-mon[99959]: pgmap v12939: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-21T03:31:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:19 smithi146 ceph-mon[103278]: pgmap v12939: 161 pgs: 161 active+clean; 585 KiB data, 394 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-09-21T03:31:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:31:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:31:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:31:21.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:21.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-mon[99959]: pgmap v12940: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-21T03:31:21.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:21.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-mon[91136]: pgmap v12940: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-21T03:31:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:21 smithi146 ceph-mon[103278]: pgmap v12940: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-21T03:31:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:21.987Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:22.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:21.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:21.988Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:23.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:23 smithi078 ceph-mon[99959]: pgmap v12941: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T03:31:23.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:23 smithi078 ceph-mon[91136]: pgmap v12941: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T03:31:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:23 smithi146 ceph-mon[103278]: pgmap v12941: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-21T03:31:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:25 smithi078 ceph-mon[91136]: pgmap v12942: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T03:31:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:25 smithi078 ceph-mon[99959]: pgmap v12942: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T03:31:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:25 smithi146 ceph-mon[103278]: pgmap v12942: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-21T03:31:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:27 smithi078 ceph-mon[91136]: pgmap v12943: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:31:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:27 smithi078 ceph-mon[99959]: pgmap v12943: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:31:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:27 smithi146 ceph-mon[103278]: pgmap v12943: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:31:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:29.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:29.306 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:29 smithi146 ceph-mon[103278]: pgmap v12944: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:31:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:29 smithi078 ceph-mon[91136]: pgmap v12944: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:31:30.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:29 smithi078 ceph-mon[99959]: pgmap v12944: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:31:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:31:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:31:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:31:32.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-mon[91136]: pgmap v12945: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:31:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-mon[99959]: pgmap v12945: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:31:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:31.988Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:31.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:32.036 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:31.989Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:31 smithi146 ceph-mon[103278]: pgmap v12945: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:31:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:34.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:34 smithi146 ceph-mon[103278]: pgmap v12946: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:31:34.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:34 smithi078 ceph-mon[91136]: pgmap v12946: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:31:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:34 smithi078 ceph-mon[99959]: pgmap v12946: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:31:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:35 smithi146 ceph-mon[103278]: pgmap v12947: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:31:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:35 smithi078 ceph-mon[91136]: pgmap v12947: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:31:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:35 smithi078 ceph-mon[99959]: pgmap v12947: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:31:38.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:38 smithi146 ceph-mon[103278]: pgmap v12948: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:31:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[99959]: pgmap v12948: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:38 smithi078 ceph-mon[91136]: pgmap v12948: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:31:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:39 smithi146 ceph-mon[103278]: pgmap v12949: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:31:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:39 smithi078 ceph-mon[99959]: pgmap v12949: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:31:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:39 smithi078 ceph-mon[91136]: pgmap v12949: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:31:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:31:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:31:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:31:42.020 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:42.021 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-mon[99959]: pgmap v12950: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:42.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:42.021 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-mon[91136]: pgmap v12950: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:42.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:41.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:42.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:41.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:42.021 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:41.990Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:41 smithi146 ceph-mon[103278]: pgmap v12950: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:43.990 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:43.990 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:44.708 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:31:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:44 smithi146 ceph-mon[103278]: pgmap v12951: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:45.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:44 smithi078 ceph-mon[91136]: pgmap v12951: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:45.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:44 smithi078 ceph-mon[99959]: pgmap v12951: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:31:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 7h - - 2024-09-21T03:31:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 7h - - 2024-09-21T03:31:45.236 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 12s ago 7h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 7s ago 7h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 12s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 7s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 12s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 12s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 7s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 12s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 12s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 7s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 12s ago 7h 99.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:31:45.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 12s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 12s ago 7h 86.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 12s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 7s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 7s ago 7h 94.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 7s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 7s ago 7h 98.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 7s ago 7h 69.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 12s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 12s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:31:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:31:45.548 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:31:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:31:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:31:45.550 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:31:45.819 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:31:45.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:31:45.821 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:45 smithi078 ceph-mon[99959]: from='client.73419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:45.821 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2785219862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:45.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:45 smithi078 ceph-mon[91136]: from='client.73419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:45.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2785219862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:45 smithi146 ceph-mon[103278]: from='client.73419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2785219862' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:31:46.131 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:31:46.132 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:46.132 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:31:46.132 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:46.132 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:31:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:46 smithi146 ceph-mon[103278]: from='client.93341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:46 smithi146 ceph-mon[103278]: from='client.73431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:47.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:46 smithi146 ceph-mon[103278]: pgmap v12952: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:47.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:46 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3126072474' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[99959]: from='client.93341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[99959]: from='client.73431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:47.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[99959]: pgmap v12952: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:47.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3126072474' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[91136]: from='client.93341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[91136]: from='client.73431 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[91136]: pgmap v12952: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:47.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:46 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3126072474' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:31:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:47 smithi146 ceph-mon[103278]: from='client.73443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:48.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:47 smithi078 ceph-mon[91136]: from='client.73443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:47 smithi078 ceph-mon[99959]: from='client.73443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:31:49.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:48 smithi146 ceph-mon[103278]: pgmap v12953: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:49.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:48 smithi078 ceph-mon[99959]: pgmap v12953: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:49.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:48 smithi078 ceph-mon[91136]: pgmap v12953: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:49 smithi146 ceph-mon[103278]: pgmap v12954: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:49 smithi078 ceph-mon[91136]: pgmap v12954: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:50.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:49 smithi078 ceph-mon[99959]: pgmap v12954: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:31:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:31:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:31:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:51 smithi146 ceph-mon[103278]: pgmap v12955: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-mon[91136]: pgmap v12955: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:31:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-mon[99959]: pgmap v12955: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:51.990Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:51.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:31:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:31:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:31:51.991Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:31:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:54.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:54 smithi146 ceph-mon[103278]: pgmap v12956: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:54.888 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:54 smithi078 ceph-mon[99959]: pgmap v12956: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:54.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:54 smithi078 ceph-mon[91136]: pgmap v12956: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:31:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:55 smithi146 ceph-mon[103278]: pgmap v12957: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:55 smithi078 ceph-mon[91136]: pgmap v12957: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:56.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:55 smithi078 ceph-mon[99959]: pgmap v12957: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:58.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:58 smithi146 ceph-mon[103278]: pgmap v12958: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:58.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[91136]: pgmap v12958: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:31:58.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:58 smithi078 ceph-mon[99959]: pgmap v12958: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:31:59.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:31:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:31:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:31:59.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:31:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:00.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:00 smithi146 ceph-mon[103278]: pgmap v12959: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:00.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:00 smithi078 ceph-mon[91136]: pgmap v12959: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:00 smithi078 ceph-mon[99959]: pgmap v12959: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:32:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:32:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:32:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:01 smithi146 ceph-mon[103278]: pgmap v12960: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-mon[99959]: pgmap v12960: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-mon[91136]: pgmap v12960: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:01.991Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:01.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:01.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:03.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:03 smithi078 ceph-mon[91136]: pgmap v12961: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:03.979 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:03 smithi078 ceph-mon[99959]: pgmap v12961: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:03 smithi146 ceph-mon[103278]: pgmap v12961: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:05 smithi146 ceph-mon[103278]: pgmap v12962: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:05 smithi078 ceph-mon[91136]: pgmap v12962: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:05 smithi078 ceph-mon[99959]: pgmap v12962: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:07 smithi146 ceph-mon[103278]: pgmap v12963: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:08.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:07 smithi078 ceph-mon[91136]: pgmap v12963: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:07 smithi078 ceph-mon[99959]: pgmap v12963: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:09 smithi146 ceph-mon[103278]: pgmap v12964: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:09 smithi078 ceph-mon[91136]: pgmap v12964: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:09 smithi078 ceph-mon[99959]: pgmap v12964: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:32:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:32:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:32:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:11 smithi146 ceph-mon[103278]: pgmap v12965: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-mon[91136]: pgmap v12965: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-mon[99959]: pgmap v12965: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:11.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:11.992Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:11.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:14.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:14.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:14 smithi078 ceph-mon[99959]: pgmap v12966: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:14.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:14.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:14 smithi078 ceph-mon[91136]: pgmap v12966: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:14 smithi146 ceph-mon[103278]: pgmap v12966: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:15 smithi146 ceph-mon[103278]: pgmap v12967: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:16.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:15 smithi078 ceph-mon[91136]: pgmap v12967: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:16.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:15 smithi078 ceph-mon[99959]: pgmap v12967: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:16.429 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:32:16.953 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:32:16.953 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 7h - - 2024-09-21T03:32:16.953 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 7h - - 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 4s ago 7h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 19s ago 7h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 4s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 19s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 4s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 4s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 19s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 4s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 4s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 19s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 4s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:32:16.954 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 4s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 4s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 4s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 19s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 19s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 19s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 19s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 19s ago 7h 71.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 4s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 4s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:32:16.955 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:32:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:32:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:32:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:32:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:32:17.262 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:32:17.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:16 smithi078 ceph-mon[99959]: from='client.73455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:16 smithi078 ceph-mon[99959]: from='client.93377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:16 smithi078 ceph-mon[91136]: from='client.73455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:16 smithi078 ceph-mon[91136]: from='client.93377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:16 smithi146 ceph-mon[103278]: from='client.73455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:16 smithi146 ceph-mon[103278]: from='client.93377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:32:17.844 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:32:17.844 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:17.844 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:17.844 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:17.844 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:17.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: from='client.73467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1597700376' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:17.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: from='client.93395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:17.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:17.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:17.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:17.859 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:17 smithi146 ceph-mon[103278]: pgmap v12968: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:18.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: from='client.73467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1597700376' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: from='client.93395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[91136]: pgmap v12968: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: from='client.73467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1597700376' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: from='client.93395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:17 smithi078 ceph-mon[99959]: pgmap v12968: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:18.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/225994300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/225994300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/225994300' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:19 smithi146 ceph-mon[103278]: pgmap v12969: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:20.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:19 smithi078 ceph-mon[99959]: pgmap v12969: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:19 smithi078 ceph-mon[91136]: pgmap v12969: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:32:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:32:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:32:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:21 smithi146 ceph-mon[103278]: pgmap v12970: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:22.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-mon[91136]: pgmap v12970: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-mon[99959]: pgmap v12970: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:21.992Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:21.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:21.993Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:23.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:23 smithi078 ceph-mon[91136]: pgmap v12971: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:23.884 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:23 smithi078 ceph-mon[99959]: pgmap v12971: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:23 smithi146 ceph-mon[103278]: pgmap v12971: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:26.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:25 smithi146 ceph-mon[103278]: pgmap v12972: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:26.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:25 smithi078 ceph-mon[91136]: pgmap v12972: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:25 smithi078 ceph-mon[99959]: pgmap v12972: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:27 smithi146 ceph-mon[103278]: pgmap v12973: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:28.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:27 smithi078 ceph-mon[91136]: pgmap v12973: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:27 smithi078 ceph-mon[99959]: pgmap v12973: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:29.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:29 smithi146 ceph-mon[103278]: pgmap v12974: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:30.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:29 smithi078 ceph-mon[91136]: pgmap v12974: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:29 smithi078 ceph-mon[99959]: pgmap v12974: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:32:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:32:30] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:32:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:32 smithi078 ceph-mon[99959]: pgmap v12975: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:32 smithi078 ceph-mon[91136]: pgmap v12975: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:31.993Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:31.994Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:31.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:32 smithi146 ceph-mon[103278]: pgmap v12975: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:33 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:33 smithi146 ceph-mon[103278]: pgmap v12976: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[91136]: pgmap v12976: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:34.019 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:33 smithi078 ceph-mon[99959]: pgmap v12976: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:35 smithi146 ceph-mon[103278]: pgmap v12977: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:36.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:35 smithi078 ceph-mon[91136]: pgmap v12977: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:36.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:35 smithi078 ceph-mon[99959]: pgmap v12977: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:38.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:38.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:38 smithi146 ceph-mon[103278]: pgmap v12978: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[91136]: pgmap v12978: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:39.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:38 smithi078 ceph-mon[99959]: pgmap v12978: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:39 smithi146 ceph-mon[103278]: pgmap v12979: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:40.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:39 smithi078 ceph-mon[91136]: pgmap v12979: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:40.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:39 smithi078 ceph-mon[99959]: pgmap v12979: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:32:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:32:40] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:32:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:41 smithi146 ceph-mon[103278]: pgmap v12980: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-mon[91136]: pgmap v12980: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-mon[99959]: pgmap v12980: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:41.994Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:41.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:41.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:43.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:43.953 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:44 smithi146 ceph-mon[103278]: pgmap v12981: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:45.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:44 smithi078 ceph-mon[91136]: pgmap v12981: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:45.031 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:44 smithi078 ceph-mon[99959]: pgmap v12981: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:46.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:45 smithi078 ceph-mon[91136]: pgmap v12982: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:46.226 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:45 smithi078 ceph-mon[99959]: pgmap v12982: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:46.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:45 smithi146 ceph-mon[103278]: pgmap v12982: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:48.050 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:47 smithi078 ceph-mon[99959]: pgmap v12983: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:48.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:47 smithi078 ceph-mon[91136]: pgmap v12983: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:48.135 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:32:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:47 smithi146 ceph-mon[103278]: pgmap v12983: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 7h - - 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 7h - - 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 15s ago 7h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 10s ago 7h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 15s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 10s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 15s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 15s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 10s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 15s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 15s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 10s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 15s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 15s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 15s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 15s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 10s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 10s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 10s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 10s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 10s ago 7h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 15s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 15s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:32:48.969 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:32:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:32:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:32:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:32:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:32:48.970 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:32:48.970 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:48 smithi078 ceph-mon[99959]: from='client.93407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:48.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:48 smithi078 ceph-mon[99959]: from='client.93413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:48.971 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:48 smithi078 ceph-mon[99959]: from='client.93419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:48.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:48 smithi078 ceph-mon[91136]: from='client.93407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:48.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:48 smithi078 ceph-mon[91136]: from='client.93413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:48.971 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:48 smithi078 ceph-mon[91136]: from='client.93419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:49.236 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:32:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:32:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:32:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:32:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:32:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:32:49.237 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:32:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:48 smithi146 ceph-mon[103278]: from='client.93407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:48 smithi146 ceph-mon[103278]: from='client.93413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:48 smithi146 ceph-mon[103278]: from='client.93419 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:49.542 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:32:49.542 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:49.542 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:32:49.543 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:49.543 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:32:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1502842956' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:49 smithi146 ceph-mon[103278]: from='client.93431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4033655572' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:49 smithi146 ceph-mon[103278]: pgmap v12984: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1502842956' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[91136]: from='client.93431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4033655572' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[91136]: pgmap v12984: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1502842956' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[99959]: from='client.93431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4033655572' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:32:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:49 smithi078 ceph-mon[99959]: pgmap v12984: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:32:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:32:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:32:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:51 smithi146 ceph-mon[103278]: pgmap v12985: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-mon[91136]: pgmap v12985: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:32:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-mon[99959]: pgmap v12985: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:51.995Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:51.995Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:32:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:32:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:32:51.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:32:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:53 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:53 smithi146 ceph-mon[103278]: pgmap v12986: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:54.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[91136]: pgmap v12986: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:53 smithi078 ceph-mon[99959]: pgmap v12986: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:32:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:55 smithi146 ceph-mon[103278]: pgmap v12987: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:56.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:55 smithi078 ceph-mon[91136]: pgmap v12987: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:55 smithi078 ceph-mon[99959]: pgmap v12987: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:57 smithi078 ceph-mon[91136]: pgmap v12988: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:57 smithi078 ceph-mon[99959]: pgmap v12988: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:58.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:57 smithi146 ceph-mon[103278]: pgmap v12988: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:32:59.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:59.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:32:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:00.165 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:32:59 smithi146 ceph-mon[103278]: pgmap v12989: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:00.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:32:59 smithi078 ceph-mon[99959]: pgmap v12989: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:32:59 smithi078 ceph-mon[91136]: pgmap v12989: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:33:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:33:00] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T03:33:01.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:01.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-mon[91136]: pgmap v12990: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:01 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:01 smithi146 ceph-mon[103278]: pgmap v12990: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-mon[99959]: pgmap v12990: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:01.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:01.996Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:01.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:03 smithi146 ceph-mon[103278]: pgmap v12991: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:04.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:03 smithi078 ceph-mon[91136]: pgmap v12991: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:03 smithi078 ceph-mon[99959]: pgmap v12991: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:06.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:05 smithi078 ceph-mon[91136]: pgmap v12992: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:06.095 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:05 smithi078 ceph-mon[99959]: pgmap v12992: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:05 smithi146 ceph-mon[103278]: pgmap v12992: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:07 smithi146 ceph-mon[103278]: pgmap v12993: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:07 smithi078 ceph-mon[91136]: pgmap v12993: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:07 smithi078 ceph-mon[99959]: pgmap v12993: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:09 smithi146 ceph-mon[103278]: pgmap v12994: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:09 smithi078 ceph-mon[91136]: pgmap v12994: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:09 smithi078 ceph-mon[99959]: pgmap v12994: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:33:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:33:10] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T03:33:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:11 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:11 smithi146 ceph-mon[103278]: pgmap v12995: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-mon[91136]: pgmap v12995: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-mon[99959]: pgmap v12995: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:11.996Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:11.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:11.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:15 smithi146 ceph-mon[103278]: pgmap v12996: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:15 smithi078 ceph-mon[91136]: pgmap v12996: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:15 smithi078 ceph-mon[99959]: pgmap v12996: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:17 smithi146 ceph-mon[103278]: pgmap v12997: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:17 smithi078 ceph-mon[91136]: pgmap v12997: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:17 smithi078 ceph-mon[99959]: pgmap v12997: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:18 smithi146 ceph-mon[103278]: pgmap v12998: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[91136]: pgmap v12998: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[99959]: pgmap v12998: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:19.834 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:33:20.365 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:33:20.365 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 7h - - 2024-09-21T03:33:20.365 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 7h - - 2024-09-21T03:33:20.365 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 7s ago 7h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:33:20.365 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 2s ago 7h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:33:20.365 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 7s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 2s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 7s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 7s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 2s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 7s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 7s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 2s ago 7h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 7s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 7s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 7s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 7s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 2s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:33:20.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 2s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 2s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 2s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 2s ago 7h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 7s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 7s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:33:20.367 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:33:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:33:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:33:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:33:20.677 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:33:20.677 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:33:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:33:20.945 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:33:20.946 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:33:20.947 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:33:20.947 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:33:20.947 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:33:21.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:20 smithi146 ceph-mon[103278]: pgmap v12999: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:21.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:20 smithi146 ceph-mon[103278]: from='client.93443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:20 smithi146 ceph-mon[103278]: from='client.93449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:20 smithi146 ceph-mon[103278]: from='client.93455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.222 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1509217265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:21.256 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:33:21.256 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:21.256 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:21.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[99959]: pgmap v12999: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:21.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[99959]: from='client.93443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[99959]: from='client.93449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[99959]: from='client.93455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1509217265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[91136]: pgmap v12999: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[91136]: from='client.93443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[91136]: from='client.93449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[91136]: from='client.93455 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:21.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1509217265' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:22.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:21 smithi146 ceph-mon[103278]: from='client.73551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:22.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1125773360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:22.245 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:21 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-mon[91136]: from='client.73551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1125773360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-mon[99959]: from='client.73551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1125773360' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:21.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:21.997Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:21 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:21.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:22 smithi078 ceph-mon[91136]: pgmap v13000: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:22 smithi078 ceph-mon[99959]: pgmap v13000: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:22 smithi146 ceph-mon[103278]: pgmap v13000: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:24 smithi146 ceph-mon[103278]: pgmap v13001: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:24 smithi078 ceph-mon[91136]: pgmap v13001: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:24 smithi078 ceph-mon[99959]: pgmap v13001: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:27.023 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:26 smithi078 ceph-mon[99959]: pgmap v13002: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:27.023 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:26 smithi078 ceph-mon[91136]: pgmap v13002: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:26 smithi146 ceph-mon[103278]: pgmap v13002: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:28 smithi146 ceph-mon[103278]: pgmap v13003: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:28 smithi078 ceph-mon[91136]: pgmap v13003: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:28 smithi078 ceph-mon[99959]: pgmap v13003: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:30.982 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:33:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:33:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:33:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:30 smithi146 ceph-mon[103278]: pgmap v13004: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:31.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:30 smithi078 ceph-mon[91136]: pgmap v13004: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:30 smithi078 ceph-mon[99959]: pgmap v13004: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:32.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:31.997Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:31.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:31 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:31.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:32 smithi146 ceph-mon[103278]: pgmap v13005: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:32 smithi078 ceph-mon[91136]: pgmap v13005: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:32 smithi078 ceph-mon[99959]: pgmap v13005: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:34.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:35 smithi146 ceph-mon[103278]: pgmap v13006: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:35 smithi078 ceph-mon[91136]: pgmap v13006: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:35.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:35 smithi078 ceph-mon[99959]: pgmap v13006: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:37 smithi146 ceph-mon[103278]: pgmap v13007: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:37 smithi078 ceph-mon[91136]: pgmap v13007: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:37 smithi078 ceph-mon[99959]: pgmap v13007: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:39 smithi146 ceph-mon[103278]: pgmap v13008: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[91136]: pgmap v13008: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[99959]: pgmap v13008: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:33:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:33:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:33:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:41 smithi146 ceph-mon[103278]: pgmap v13009: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:41 smithi078 ceph-mon[91136]: pgmap v13009: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:41 smithi078 ceph-mon[99959]: pgmap v13009: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:42.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:41.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:41.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:41 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:41.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:43 smithi146 ceph-mon[103278]: pgmap v13010: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:43 smithi078 ceph-mon[99959]: pgmap v13010: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:43 smithi078 ceph-mon[91136]: pgmap v13010: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:44.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:45.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:45 smithi146 ceph-mon[103278]: pgmap v13011: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:45.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:45 smithi078 ceph-mon[91136]: pgmap v13011: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:45.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:45 smithi078 ceph-mon[99959]: pgmap v13011: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:47.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:47 smithi078 ceph-mon[91136]: pgmap v13012: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:47.543 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:47 smithi078 ceph-mon[99959]: pgmap v13012: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:47.543 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:47 smithi146 ceph-mon[103278]: pgmap v13012: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:49 smithi146 ceph-mon[103278]: pgmap v13013: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:49 smithi078 ceph-mon[91136]: pgmap v13013: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:49 smithi078 ceph-mon[99959]: pgmap v13013: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:33:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:33:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:33:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:51 smithi146 ceph-mon[103278]: pgmap v13014: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:51.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:51 smithi078 ceph-mon[99959]: pgmap v13014: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:51 smithi078 ceph-mon[91136]: pgmap v13014: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:51.553 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:33:52.092 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 7h - - 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 7h - - 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 18s ago 7h 29.4M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 14s ago 7h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 18s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 14s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 18s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 18s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 14s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 18s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 14s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 18s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 18s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:33:52.093 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 18s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 18s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 14s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 14s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 14s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 14s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 14s ago 7h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 18s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 18s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:33:52.094 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:33:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:52 smithi078 ceph-mon[91136]: from='client.73563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:52.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:52 smithi078 ceph-mon[99959]: from='client.73563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:51.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:51.998Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:33:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:33:51 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:33:51.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:33:52.409 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:33:52.409 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:33:52.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:33:52.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:33:52.410 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:33:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:52 smithi146 ceph-mon[103278]: from='client.73563 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:52.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:33:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:33:52.690 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:33:52.690 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:33:52.690 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:33:52.690 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:33:53.005 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:33:53.005 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:53.005 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:33:53.006 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:53.006 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[91136]: pgmap v13015: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[91136]: from='client.73569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[91136]: from='client.73575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/645922624' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[91136]: from='client.73587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3038989452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[99959]: pgmap v13015: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:53.275 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[99959]: from='client.73569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[99959]: from='client.73575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/645922624' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:53.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[99959]: from='client.73587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.276 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:53 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3038989452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:53 smithi146 ceph-mon[103278]: pgmap v13015: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:53 smithi146 ceph-mon[103278]: from='client.73569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:53 smithi146 ceph-mon[103278]: from='client.73575 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/645922624' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:33:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:53 smithi146 ceph-mon[103278]: from='client.73587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:33:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:53 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3038989452' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:33:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:54.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:55 smithi146 ceph-mon[103278]: pgmap v13016: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:55.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:55 smithi078 ceph-mon[91136]: pgmap v13016: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:55.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:55 smithi078 ceph-mon[99959]: pgmap v13016: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:33:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:57 smithi078 ceph-mon[91136]: pgmap v13017: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:57 smithi078 ceph-mon[99959]: pgmap v13017: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:57 smithi146 ceph-mon[103278]: pgmap v13017: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:59 smithi146 ceph-mon[103278]: pgmap v13018: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:33:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[91136]: pgmap v13018: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:33:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[99959]: pgmap v13018: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:33:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:33:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:33:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:01.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:34:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:34:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:34:01.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:01 smithi146 ceph-mon[103278]: pgmap v13019: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:01 smithi078 ceph-mon[91136]: pgmap v13019: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:01 smithi078 ceph-mon[99959]: pgmap v13019: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:01.998Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:01.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:01 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:01.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:02.475 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:03.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:03 smithi078 ceph-mon[91136]: pgmap v13020: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:03 smithi078 ceph-mon[99959]: pgmap v13020: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:03 smithi146 ceph-mon[103278]: pgmap v13020: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:05 smithi146 ceph-mon[103278]: pgmap v13021: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:05.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:05 smithi078 ceph-mon[99959]: pgmap v13021: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:05 smithi078 ceph-mon[91136]: pgmap v13021: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:07 smithi146 ceph-mon[103278]: pgmap v13022: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:07.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:07 smithi078 ceph-mon[99959]: pgmap v13022: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:07 smithi078 ceph-mon[91136]: pgmap v13022: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:09.359 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:09 smithi078 ceph-mon[99959]: pgmap v13023: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:09.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:09 smithi078 ceph-mon[91136]: pgmap v13023: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:09 smithi146 ceph-mon[103278]: pgmap v13023: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:10.960 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:34:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:34:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:34:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:11 smithi146 ceph-mon[103278]: pgmap v13024: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:11 smithi078 ceph-mon[91136]: pgmap v13024: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:11 smithi078 ceph-mon[99959]: pgmap v13024: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:12.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:11 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:11.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:11.999Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:12.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:13 smithi146 ceph-mon[103278]: pgmap v13025: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:13.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:13 smithi078 ceph-mon[99959]: pgmap v13025: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:13 smithi078 ceph-mon[91136]: pgmap v13025: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:14.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.688 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:14.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:15 smithi146 ceph-mon[103278]: pgmap v13026: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:15.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:15 smithi078 ceph-mon[99959]: pgmap v13026: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:15 smithi078 ceph-mon[91136]: pgmap v13026: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:17 smithi146 ceph-mon[103278]: pgmap v13027: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:17.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:17 smithi078 ceph-mon[91136]: pgmap v13027: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:17.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:17 smithi078 ceph-mon[99959]: pgmap v13027: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:19 smithi146 ceph-mon[103278]: pgmap v13028: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[99959]: pgmap v13028: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:19.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[91136]: pgmap v13028: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:34:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:34:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T03:34:21.624 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:21 smithi078 ceph-mon[99959]: pgmap v13029: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:21.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:21 smithi078 ceph-mon[91136]: pgmap v13029: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:21.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:21 smithi146 ceph-mon[103278]: pgmap v13029: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:21.999Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:22.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:22.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:22.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:22.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:22.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:23.299 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:34:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:23 smithi146 ceph-mon[103278]: pgmap v13030: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:23.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:23 smithi078 ceph-mon[99959]: pgmap v13030: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:23 smithi078 ceph-mon[91136]: pgmap v13030: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:23.824 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:34:23.824 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 7h - - 2024-09-21T03:34:23.824 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 7h - - 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 10s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 5s ago 7h 88.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 10s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 5s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 10s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 10s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 5s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 10s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 10s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 5s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 10s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 10s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 10s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 10s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:34:23.825 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 5s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 5s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 5s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 5s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 5s ago 7h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 10s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 10s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:34:23.826 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:34:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:34:24.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:34:24.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:34:24.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:34:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:34:24.410 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:34:24.411 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:34:24.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:24 smithi078 ceph-mon[91136]: from='client.73599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:24.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:24 smithi078 ceph-mon[91136]: from='client.73605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:24.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3602141081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:24.412 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:24 smithi078 ceph-mon[99959]: from='client.73599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:24.412 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:24 smithi078 ceph-mon[99959]: from='client.73605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:24.412 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3602141081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:24.718 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:34:24.718 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:24.719 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:24.719 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:24.719 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:24 smithi146 ceph-mon[103278]: from='client.73599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:24 smithi146 ceph-mon[103278]: from='client.73605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3602141081' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:25 smithi146 ceph-mon[103278]: pgmap v13031: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:25 smithi146 ceph-mon[103278]: from='client.93527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:25 smithi146 ceph-mon[103278]: from='client.93539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:25 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1336747313' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[91136]: pgmap v13031: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:25.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[91136]: from='client.93527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[91136]: from='client.93539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1336747313' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[99959]: pgmap v13031: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[99959]: from='client.93527 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[99959]: from='client.93539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:25 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1336747313' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 03:34:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T03:34:25.847282609Z level=info msg="Completed cleanup jobs" duration=12.06414ms 2024-09-21T03:34:27.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:27 smithi078 ceph-mon[99959]: pgmap v13032: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:27.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:27 smithi078 ceph-mon[91136]: pgmap v13032: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:27 smithi146 ceph-mon[103278]: pgmap v13032: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:29.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:29 smithi078 ceph-mon[91136]: pgmap v13033: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:29.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:29.720 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:29 smithi078 ceph-mon[99959]: pgmap v13033: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:29.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:29 smithi146 ceph-mon[103278]: pgmap v13033: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:34:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:34:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:34:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:31 smithi146 ceph-mon[103278]: pgmap v13034: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:31.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:31 smithi078 ceph-mon[91136]: pgmap v13034: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:31 smithi078 ceph-mon[99959]: pgmap v13034: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:32.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:32.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:32.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:32.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:32.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:32.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:33.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:33 smithi146 ceph-mon[103278]: pgmap v13035: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:33 smithi078 ceph-mon[91136]: pgmap v13035: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:33 smithi078 ceph-mon[99959]: pgmap v13035: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:34.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:35 smithi146 ceph-mon[103278]: pgmap v13036: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:35 smithi078 ceph-mon[91136]: pgmap v13036: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:36.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:35 smithi078 ceph-mon[99959]: pgmap v13036: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:37 smithi146 ceph-mon[103278]: pgmap v13037: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:37 smithi078 ceph-mon[91136]: pgmap v13037: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:37 smithi078 ceph-mon[99959]: pgmap v13037: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:39 smithi146 ceph-mon[103278]: pgmap v13038: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[91136]: pgmap v13038: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[99959]: pgmap v13038: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:34:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:34:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:34:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:41 smithi146 ceph-mon[103278]: pgmap v13039: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:41.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:41 smithi078 ceph-mon[91136]: pgmap v13039: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:41 smithi078 ceph-mon[99959]: pgmap v13039: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:42.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:42.000Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:42.000Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:42.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:43 smithi146 ceph-mon[103278]: pgmap v13040: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:43.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:43 smithi078 ceph-mon[99959]: pgmap v13040: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:43 smithi078 ceph-mon[91136]: pgmap v13040: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:44.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:34:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:45 smithi146 ceph-mon[103278]: pgmap v13041: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:45 smithi078 ceph-mon[99959]: pgmap v13041: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:45 smithi078 ceph-mon[91136]: pgmap v13041: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:47 smithi146 ceph-mon[103278]: pgmap v13042: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:47.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:47 smithi078 ceph-mon[99959]: pgmap v13042: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:47 smithi078 ceph-mon[91136]: pgmap v13042: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:49.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:49 smithi078 ceph-mon[99959]: pgmap v13043: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:49.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:49 smithi078 ceph-mon[91136]: pgmap v13043: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:49 smithi146 ceph-mon[103278]: pgmap v13043: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:50.664 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:34:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:34:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:34:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:51 smithi146 ceph-mon[103278]: pgmap v13044: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:51 smithi078 ceph-mon[99959]: pgmap v13044: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:51 smithi078 ceph-mon[91136]: pgmap v13044: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:52.001Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:52.001Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:34:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:34:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:34:52.002Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:34:52.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:52.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:34:53.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:53 smithi078 ceph-mon[91136]: pgmap v13045: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:53 smithi078 ceph-mon[99959]: pgmap v13045: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:53 smithi146 ceph-mon[103278]: pgmap v13045: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:55.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.012 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:54 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:54 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.013 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:34:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:54 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 7h - - 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 7h - - 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 1s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 17s ago 7h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 1s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 17s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 1s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 1s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 17s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:34:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 1s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 1s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 17s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 1s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 1s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 1s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 1s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 17s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 17s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:34:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 17s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:34:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 17s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:34:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 17s ago 7h 72.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:34:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 1s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:34:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:34:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 1s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:34:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:34:55.849 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:55 smithi078 ceph-mon[99959]: pgmap v13046: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:55.849 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:34:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:34:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:34:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:55.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:34:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:34:55.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:55 smithi078 ceph-mon[91136]: pgmap v13046: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:55 smithi146 ceph-mon[103278]: pgmap v13046: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:34:56.120 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:34:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:34:56.120 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:34:56.121 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:34:56.428 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:34:56.428 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:56.428 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:34:56.428 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:56.428 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:34:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:56 smithi146 ceph-mon[103278]: from='client.93551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:56 smithi146 ceph-mon[103278]: from='client.73641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:56 smithi146 ceph-mon[103278]: from='client.73647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/330980973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3019696222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[91136]: from='client.93551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[91136]: from='client.73641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[91136]: from='client.73647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/330980973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3019696222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:57.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[99959]: from='client.93551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[99959]: from='client.73641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[99959]: from='client.73647 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:57.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/330980973' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:34:57.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3019696222' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:34:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:57 smithi146 ceph-mon[103278]: pgmap v13047: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:57 smithi146 ceph-mon[103278]: from='client.93575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:58.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:57 smithi078 ceph-mon[91136]: pgmap v13047: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:57 smithi078 ceph-mon[91136]: from='client.93575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:34:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:57 smithi078 ceph-mon[99959]: pgmap v13047: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:34:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:57 smithi078 ceph-mon[99959]: from='client.93575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:59 smithi146 ceph-mon[103278]: pgmap v13048: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:34:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[99959]: pgmap v13048: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[91136]: pgmap v13048: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:00.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:34:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:35:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:35:00] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:35:02.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:01 smithi078 ceph-mon[99959]: pgmap v13049: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:02.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:01 smithi078 ceph-mon[91136]: pgmap v13049: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:01 smithi146 ceph-mon[103278]: pgmap v13049: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:02.002Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:02.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:35:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:02.003Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:03.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:03.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:03.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:03 smithi146 ceph-mon[103278]: pgmap v13050: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:03 smithi078 ceph-mon[91136]: pgmap v13050: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:03 smithi078 ceph-mon[99959]: pgmap v13050: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:05.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:05 smithi146 ceph-mon[103278]: pgmap v13051: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:06.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:05 smithi078 ceph-mon[91136]: pgmap v13051: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:05 smithi078 ceph-mon[99959]: pgmap v13051: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:07 smithi146 ceph-mon[103278]: pgmap v13052: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:08.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:07 smithi078 ceph-mon[91136]: pgmap v13052: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:07 smithi078 ceph-mon[99959]: pgmap v13052: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:09 smithi078 ceph-mon[91136]: pgmap v13053: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:09 smithi078 ceph-mon[99959]: pgmap v13053: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:09 smithi146 ceph-mon[103278]: pgmap v13053: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:35:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:35:10] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:35:12.004 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:11 smithi078 ceph-mon[91136]: pgmap v13054: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:12.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:11 smithi078 ceph-mon[99959]: pgmap v13054: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:11 smithi146 ceph-mon[103278]: pgmap v13054: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:12.003Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:12.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:35:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:12.004Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:13.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:13.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:13 smithi146 ceph-mon[103278]: pgmap v13055: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:13 smithi078 ceph-mon[91136]: pgmap v13055: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:13 smithi078 ceph-mon[99959]: pgmap v13055: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:14 smithi146 ceph-mon[103278]: pgmap v13056: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[91136]: pgmap v13056: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[99959]: pgmap v13056: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:17.104 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:16 smithi146 ceph-mon[103278]: pgmap v13057: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:16 smithi078 ceph-mon[91136]: pgmap v13057: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:16 smithi078 ceph-mon[99959]: pgmap v13057: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:19 smithi146 ceph-mon[103278]: pgmap v13058: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[91136]: pgmap v13058: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[99959]: pgmap v13058: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:21.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:35:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:35:20] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:35:22.005 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:21 smithi078 ceph-mon[99959]: pgmap v13059: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:22.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:21 smithi078 ceph-mon[91136]: pgmap v13059: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:21 smithi146 ceph-mon[103278]: pgmap v13059: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:22.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:22.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:35:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:22.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:23.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:23 smithi146 ceph-mon[103278]: pgmap v13060: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:24.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:23 smithi078 ceph-mon[99959]: pgmap v13060: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:24.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:23 smithi078 ceph-mon[91136]: pgmap v13060: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:25 smithi078 ceph-mon[91136]: pgmap v13061: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:25 smithi078 ceph-mon[99959]: pgmap v13061: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:26.078 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:25 smithi146 ceph-mon[103278]: pgmap v13061: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:26.722 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:35:27.247 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 7h - - 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 7h - - 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 13s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 8s ago 7h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 13s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 8s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 13s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 13s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 8s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 13s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 13s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 8s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:35:27.248 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 13s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 13s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 13s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 13s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 8s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 8s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 8s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 8s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 8s ago 7h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 13s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 13s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:35:27.249 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:35:27.555 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:35:27.555 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:35:27.555 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:35:27.556 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:35:27.827 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:35:27.827 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:35:27.827 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:35:27.827 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:35:27.827 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:35:27.828 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:35:27.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:27 smithi078 ceph-mon[99959]: pgmap v13062: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:27.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:27 smithi078 ceph-mon[99959]: from='client.93587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:27.828 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3792858109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:35:27.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:27 smithi078 ceph-mon[91136]: pgmap v13062: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:27.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:27 smithi078 ceph-mon[91136]: from='client.93587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:27.829 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3792858109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:35:28.134 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:35:28.134 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:35:28.134 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:35:28.134 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:35:28.134 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:35:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:27 smithi146 ceph-mon[103278]: pgmap v13062: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:27 smithi146 ceph-mon[103278]: from='client.93587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3792858109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[99959]: from='client.73677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[99959]: from='client.73683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/652513147' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[91136]: from='client.73677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[91136]: from='client.73683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/652513147' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:35:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:28 smithi146 ceph-mon[103278]: from='client.73677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:28 smithi146 ceph-mon[103278]: from='client.73683 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/652513147' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:35:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:29 smithi078 ceph-mon[99959]: pgmap v13063: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:29 smithi078 ceph-mon[99959]: from='client.73695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:29 smithi078 ceph-mon[91136]: pgmap v13063: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:29 smithi078 ceph-mon[91136]: from='client.73695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:29 smithi146 ceph-mon[103278]: pgmap v13063: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:29 smithi146 ceph-mon[103278]: from='client.73695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:30.685 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:35:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:35:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:35:30.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:30 smithi078 ceph-mon[99959]: pgmap v13064: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:30.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:30 smithi078 ceph-mon[91136]: pgmap v13064: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:30 smithi146 ceph-mon[103278]: pgmap v13064: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:31.963 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:31 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:31.963 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:31 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:31 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:32.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:32.004Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:32.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:35:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:32.005Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:32 smithi146 ceph-mon[103278]: pgmap v13065: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:32 smithi078 ceph-mon[91136]: pgmap v13065: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:32 smithi078 ceph-mon[99959]: pgmap v13065: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:34 smithi146 ceph-mon[103278]: pgmap v13066: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:34 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[91136]: pgmap v13066: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[99959]: pgmap v13066: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:34 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:36 smithi146 ceph-mon[103278]: pgmap v13067: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:37.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:36 smithi078 ceph-mon[99959]: pgmap v13067: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:36 smithi078 ceph-mon[91136]: pgmap v13067: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:38 smithi146 ceph-mon[103278]: pgmap v13068: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[91136]: pgmap v13068: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[99959]: pgmap v13068: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:39.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:40.959 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:35:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:35:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:35:41.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:40 smithi146 ceph-mon[103278]: pgmap v13069: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:41.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:40 smithi078 ceph-mon[91136]: pgmap v13069: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:40 smithi078 ceph-mon[99959]: pgmap v13069: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:41 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:42.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:41 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:41 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:42.005Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:42.006Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:35:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:42.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:42 smithi146 ceph-mon[103278]: pgmap v13070: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:43.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:42 smithi078 ceph-mon[91136]: pgmap v13070: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:42 smithi078 ceph-mon[99959]: pgmap v13070: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:44.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:45.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:44 smithi146 ceph-mon[103278]: pgmap v13071: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:45.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:44 smithi078 ceph-mon[99959]: pgmap v13071: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:44 smithi078 ceph-mon[91136]: pgmap v13071: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:46 smithi146 ceph-mon[103278]: pgmap v13072: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:47.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:46 smithi078 ceph-mon[99959]: pgmap v13072: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:46 smithi078 ceph-mon[91136]: pgmap v13072: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:48 smithi146 ceph-mon[103278]: pgmap v13073: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:48 smithi078 ceph-mon[91136]: pgmap v13073: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:48 smithi078 ceph-mon[99959]: pgmap v13073: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:50.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:35:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:35:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:35:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:50 smithi146 ceph-mon[103278]: pgmap v13074: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:51.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:50 smithi078 ceph-mon[91136]: pgmap v13074: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:50 smithi078 ceph-mon[99959]: pgmap v13074: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:51 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:51 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:51 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:35:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:52.007Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:52.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:52.007Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:35:52.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:35:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:35:52.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:35:53.269 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:52 smithi146 ceph-mon[103278]: pgmap v13075: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:52 smithi078 ceph-mon[91136]: pgmap v13075: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:53.289 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:52 smithi078 ceph-mon[99959]: pgmap v13075: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:55.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[91136]: pgmap v13076: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:55.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[99959]: pgmap v13076: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:55.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.466 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:55 smithi146 ceph-mon[103278]: pgmap v13076: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:35:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:57 smithi146 ceph-mon[103278]: pgmap v13077: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:57.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:57 smithi078 ceph-mon[91136]: pgmap v13077: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:57 smithi078 ceph-mon[99959]: pgmap v13077: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:58.431 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 7h - - 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 20s ago 7h - - 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 4s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 20s ago 7h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 4s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 20s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 4s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 4s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 20s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 4s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 4s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:35:58.952 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 20s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 4s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 4s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 4s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 4s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 20s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 20s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 20s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 20s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 20s ago 7h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 4s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 20s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 4s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:35:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 20s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:35:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:35:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:35:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:35:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:35:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:35:59.269 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:35:59.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: pgmap v13078: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:59.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: from='client.93623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:59.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:59.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: from='client.73713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:59.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: pgmap v13078: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: from='client.93623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: from='client.73713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.270 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:35:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: pgmap v13078: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: from='client.93623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: from='client.73713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:35:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:35:59.534 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:35:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:35:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:35:59.535 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:35:59.845 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:35:59.845 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:35:59.845 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:35:59.845 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:35:59.845 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:36:00.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:00 smithi146 ceph-mon[103278]: from='client.73719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/308782475' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:36:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:00 smithi146 ceph-mon[103278]: from='client.73731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:00.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:00 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2662464045' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:36:00.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[99959]: from='client.73719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/308782475' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[99959]: from='client.73731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2662464045' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[91136]: from='client.73719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/308782475' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[91136]: from='client.73731 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:00.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2662464045' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:36:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:36:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:36:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:36:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:01 smithi146 ceph-mon[103278]: pgmap v13079: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:01 smithi078 ceph-mon[91136]: pgmap v13079: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:01.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:01 smithi078 ceph-mon[99959]: pgmap v13079: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:02.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:02.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:02.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:02.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:03 smithi146 ceph-mon[103278]: pgmap v13080: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:03 smithi078 ceph-mon[91136]: pgmap v13080: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:03 smithi078 ceph-mon[99959]: pgmap v13080: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:05.251 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:05 smithi146 ceph-mon[103278]: pgmap v13081: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:05 smithi078 ceph-mon[91136]: pgmap v13081: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:05 smithi078 ceph-mon[99959]: pgmap v13081: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:07.448 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:07 smithi146 ceph-mon[103278]: pgmap v13082: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:07 smithi078 ceph-mon[91136]: pgmap v13082: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:07 smithi078 ceph-mon[99959]: pgmap v13082: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:09.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:09 smithi146 ceph-mon[103278]: pgmap v13083: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:09.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:09 smithi078 ceph-mon[91136]: pgmap v13083: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:09 smithi078 ceph-mon[99959]: pgmap v13083: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:10.980 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:36:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:36:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:36:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:11 smithi078 ceph-mon[91136]: pgmap v13084: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:11 smithi078 ceph-mon[99959]: pgmap v13084: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:11 smithi146 ceph-mon[103278]: pgmap v13084: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:12.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:12.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:12.279 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:12.008Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:12.279 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:12.008Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:12.279 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:12.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:13.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:13 smithi078 ceph-mon[91136]: pgmap v13085: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:13.411 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:13 smithi078 ceph-mon[99959]: pgmap v13085: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:13 smithi146 ceph-mon[103278]: pgmap v13085: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:14.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:14.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:15 smithi146 ceph-mon[103278]: pgmap v13086: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[91136]: pgmap v13086: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[99959]: pgmap v13086: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:17 smithi146 ceph-mon[103278]: pgmap v13087: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:17 smithi078 ceph-mon[91136]: pgmap v13087: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:17 smithi078 ceph-mon[99959]: pgmap v13087: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:19 smithi146 ceph-mon[103278]: pgmap v13088: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[91136]: pgmap v13088: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[99959]: pgmap v13088: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:36:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:36:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:36:21.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:21 smithi146 ceph-mon[103278]: pgmap v13089: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:21 smithi078 ceph-mon[91136]: pgmap v13089: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:21 smithi078 ceph-mon[99959]: pgmap v13089: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:22.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:22.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:22.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:22.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:23 smithi078 ceph-mon[91136]: pgmap v13090: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:23 smithi078 ceph-mon[99959]: pgmap v13090: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:23 smithi146 ceph-mon[103278]: pgmap v13090: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:25 smithi146 ceph-mon[103278]: pgmap v13091: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:25.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:25 smithi078 ceph-mon[91136]: pgmap v13091: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:25 smithi078 ceph-mon[99959]: pgmap v13091: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:27.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:27 smithi146 ceph-mon[103278]: pgmap v13092: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:27 smithi078 ceph-mon[91136]: pgmap v13092: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:27 smithi078 ceph-mon[99959]: pgmap v13092: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:29 smithi146 ceph-mon[103278]: pgmap v13093: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:29 smithi078 ceph-mon[91136]: pgmap v13093: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:29.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:29 smithi078 ceph-mon[99959]: pgmap v13093: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:30.141 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:36:30.666 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 7h - - 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 7h - - 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 16s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 11s ago 7h 90.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 16s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 11s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 16s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 16s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 11s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 16s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 16s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 11s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:36:30.667 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 16s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 16s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 16s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 16s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 11s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 11s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 11s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 11s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 11s ago 7h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 16s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 16s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:36:30.668 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:36:30.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:36:30.974 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:36:30.974 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:36:30.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:36:30.975 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:36:30.975 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:36:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:36:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:36:31.246 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:36:31.247 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:36:31.248 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:36:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:31 smithi146 ceph-mon[103278]: pgmap v13094: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:31 smithi146 ceph-mon[103278]: from='client.73743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:31 smithi146 ceph-mon[103278]: from='client.93665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:31 smithi146 ceph-mon[103278]: from='client.93671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3446122174' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:36:31.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[91136]: pgmap v13094: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[91136]: from='client.73743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[91136]: from='client.93665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[91136]: from='client.93671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3446122174' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[99959]: pgmap v13094: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[99959]: from='client.73743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[99959]: from='client.93665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[99959]: from='client.93671 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3446122174' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:36:31.561 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:36:31.561 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:36:31.561 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:36:31.561 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:36:31.561 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:36:32.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:32.009Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": context deadline exceeded" 2024-09-21T03:36:32.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:32.009Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:32.259 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:32.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:32 smithi146 ceph-mon[103278]: from='client.73767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:32.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:32 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/454824957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:36:32.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:32.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-mon[91136]: from='client.73767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/454824957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:36:32.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-mon[99959]: from='client.73767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:36:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/454824957' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:36:32.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:33 smithi146 ceph-mon[103278]: pgmap v13095: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:33.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:33 smithi078 ceph-mon[91136]: pgmap v13095: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:33.530 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:33 smithi078 ceph-mon[99959]: pgmap v13095: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:35 smithi146 ceph-mon[103278]: pgmap v13096: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[99959]: pgmap v13096: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[91136]: pgmap v13096: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:37 smithi146 ceph-mon[103278]: pgmap v13097: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:37.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:37 smithi078 ceph-mon[91136]: pgmap v13097: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:37 smithi078 ceph-mon[99959]: pgmap v13097: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:39 smithi146 ceph-mon[103278]: pgmap v13098: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[91136]: pgmap v13098: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[99959]: pgmap v13098: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:36:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:36:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:36:41.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:41 smithi078 ceph-mon[91136]: pgmap v13099: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:41.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:41 smithi078 ceph-mon[99959]: pgmap v13099: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:41 smithi146 ceph-mon[103278]: pgmap v13099: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:42.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:42.010Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:42.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:43 smithi146 ceph-mon[103278]: pgmap v13100: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:43.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:43 smithi078 ceph-mon[91136]: pgmap v13100: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:43 smithi078 ceph-mon[99959]: pgmap v13100: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:44.668 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:44.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:45.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:45 smithi146 ceph-mon[103278]: pgmap v13101: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:45 smithi078 ceph-mon[99959]: pgmap v13101: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:45 smithi078 ceph-mon[91136]: pgmap v13101: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:47.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:47 smithi146 ceph-mon[103278]: pgmap v13102: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:47 smithi078 ceph-mon[99959]: pgmap v13102: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:47 smithi078 ceph-mon[91136]: pgmap v13102: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:49 smithi146 ceph-mon[103278]: pgmap v13103: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:49.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:49 smithi078 ceph-mon[99959]: pgmap v13103: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:49 smithi078 ceph-mon[91136]: pgmap v13103: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:36:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:36:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:36:51.729 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:51 smithi078 ceph-mon[91136]: pgmap v13104: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:51.730 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:51 smithi078 ceph-mon[99959]: pgmap v13104: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:51 smithi146 ceph-mon[103278]: pgmap v13104: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:52.010Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:52.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:36:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:36:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:36:52.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:36:52.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:52.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:52.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:36:53.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:53 smithi078 ceph-mon[91136]: pgmap v13105: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:53.699 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:53 smithi078 ceph-mon[99959]: pgmap v13105: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:53 smithi146 ceph-mon[103278]: pgmap v13105: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:55 smithi146 ceph-mon[103278]: pgmap v13106: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[91136]: pgmap v13106: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[99959]: pgmap v13106: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:57 smithi146 ceph-mon[103278]: pgmap v13107: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:57.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:57 smithi078 ceph-mon[91136]: pgmap v13107: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:57 smithi078 ceph-mon[99959]: pgmap v13107: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:59 smithi146 ceph-mon[103278]: pgmap v13108: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:36:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[91136]: pgmap v13108: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[99959]: pgmap v13108: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:36:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:36:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:37:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:37:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:37:01.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:01 smithi078 ceph-mon[91136]: pgmap v13109: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:01.740 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:01 smithi078 ceph-mon[99959]: pgmap v13109: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:01 smithi146 ceph-mon[103278]: pgmap v13109: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:01.857 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:37:02.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:02.011Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:02.011Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:02.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 7h - - 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 7h - - 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 7s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 2s ago 7h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 7s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 2s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 7s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:37:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 7s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 2s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 7s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 7s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 2s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 7s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 7s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 7s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 7s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 2s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 2s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 2s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 2s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 2s ago 7h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 7s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:37:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:37:02.388 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 7s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:37:02.388 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:37:02.696 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:37:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:37:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:37:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:37:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:37:02.698 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:37:02.698 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:02.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:02.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:37:02.966 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:37:02.966 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:37:02.966 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:37:02.966 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:37:02.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:37:02.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:37:03.274 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:37:03.275 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:37:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:37:03.275 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:37:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:37:03.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[91136]: pgmap v13110: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[91136]: from='client.73779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[91136]: from='client.73785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[91136]: from='client.93707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3583425571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3062393681' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[99959]: pgmap v13110: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[99959]: from='client.73779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[99959]: from='client.73785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[99959]: from='client.93707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3583425571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:37:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3062393681' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:37:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:03 smithi146 ceph-mon[103278]: pgmap v13110: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:03 smithi146 ceph-mon[103278]: from='client.73779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:03 smithi146 ceph-mon[103278]: from='client.73785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:03 smithi146 ceph-mon[103278]: from='client.93707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3583425571' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:37:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3062393681' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:37:04.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:04 smithi078 ceph-mon[91136]: from='client.93719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:04.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:04 smithi078 ceph-mon[99959]: from='client.93719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:04.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:04 smithi146 ceph-mon[103278]: from='client.93719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:05 smithi146 ceph-mon[103278]: pgmap v13111: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:05.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:05 smithi078 ceph-mon[91136]: pgmap v13111: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:05 smithi078 ceph-mon[99959]: pgmap v13111: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:07.655 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:07 smithi146 ceph-mon[103278]: pgmap v13112: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:07.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:07 smithi078 ceph-mon[91136]: pgmap v13112: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:07 smithi078 ceph-mon[99959]: pgmap v13112: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:09 smithi146 ceph-mon[103278]: pgmap v13113: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:09.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:09 smithi078 ceph-mon[99959]: pgmap v13113: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:09 smithi078 ceph-mon[91136]: pgmap v13113: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:37:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:37:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:37:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:11 smithi146 ceph-mon[103278]: pgmap v13114: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:11.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:11 smithi078 ceph-mon[91136]: pgmap v13114: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:11 smithi078 ceph-mon[99959]: pgmap v13114: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:12.012Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:12.012Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:12.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:12.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:12.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:13 smithi146 ceph-mon[103278]: pgmap v13115: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:13 smithi078 ceph-mon[91136]: pgmap v13115: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:13 smithi078 ceph-mon[99959]: pgmap v13115: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:14.610 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:14.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[91136]: pgmap v13116: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[99959]: pgmap v13116: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:15 smithi146 ceph-mon[103278]: pgmap v13116: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:15.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:15.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:17 smithi146 ceph-mon[103278]: pgmap v13117: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:18.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:17 smithi078 ceph-mon[91136]: pgmap v13117: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:17 smithi078 ceph-mon[99959]: pgmap v13117: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:19 smithi146 ceph-mon[103278]: pgmap v13118: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[99959]: pgmap v13118: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[91136]: pgmap v13118: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:37:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:37:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:37:22.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:21 smithi078 ceph-mon[99959]: pgmap v13119: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:22.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:21 smithi078 ceph-mon[91136]: pgmap v13119: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:21 smithi146 ceph-mon[103278]: pgmap v13119: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:22.013Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:22.013Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:22.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:23.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:23 smithi078 ceph-mon[91136]: pgmap v13120: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:23 smithi078 ceph-mon[99959]: pgmap v13120: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:23 smithi146 ceph-mon[103278]: pgmap v13120: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:25 smithi146 ceph-mon[103278]: pgmap v13121: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:26.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:25 smithi078 ceph-mon[91136]: pgmap v13121: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:25 smithi078 ceph-mon[99959]: pgmap v13121: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:27 smithi146 ceph-mon[103278]: pgmap v13122: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:27 smithi078 ceph-mon[91136]: pgmap v13122: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:27 smithi078 ceph-mon[99959]: pgmap v13122: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:30.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:29 smithi146 ceph-mon[103278]: pgmap v13123: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:30.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:29 smithi078 ceph-mon[91136]: pgmap v13123: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:29 smithi078 ceph-mon[99959]: pgmap v13123: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:37:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:37:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:37:32.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:31 smithi078 ceph-mon[91136]: pgmap v13124: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:32.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:31 smithi078 ceph-mon[99959]: pgmap v13124: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:31 smithi146 ceph-mon[103278]: pgmap v13124: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:32.014Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:32.014Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:32.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:32.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:32.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:33.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:33.570 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:37:33.831 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:33 smithi078 ceph-mon[99959]: pgmap v13125: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:33.831 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:33 smithi078 ceph-mon[91136]: pgmap v13125: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:33 smithi146 ceph-mon[103278]: pgmap v13125: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:34.119 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 7h - - 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 7h - - 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 19s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 14s ago 7h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 19s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 14s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 19s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 19s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 14s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 19s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 19s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 14s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 19s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:37:34.120 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 19s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 19s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 19s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 14s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 14s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 14s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 14s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 14s ago 7h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 19s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 19s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:37:34.121 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:37:34.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:37:34.436 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:37:34.720 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:37:34.721 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:37:34.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:34 smithi078 ceph-mon[99959]: from='client.73815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:34.721 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2220416327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:37:34.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:34 smithi078 ceph-mon[91136]: from='client.73815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:34.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2220416327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:37:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:34 smithi146 ceph-mon[103278]: from='client.73815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:35.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2220416327' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:37:35.077 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:37:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='client.93737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: pgmap v13126: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='client.93743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='client.93755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1552421320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='client.93737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: pgmap v13126: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='client.93743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='client.93755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1552421320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='client.93737 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: pgmap v13126: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='client.93743 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='client.93755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:37:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:36.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1552421320' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:37:38.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:37 smithi078 ceph-mon[91136]: pgmap v13127: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:38.010 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:37 smithi078 ceph-mon[99959]: pgmap v13127: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:37 smithi146 ceph-mon[103278]: pgmap v13127: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:39 smithi146 ceph-mon[103278]: pgmap v13128: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[91136]: pgmap v13128: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[99959]: pgmap v13128: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:40.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:37:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:37:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:37:42.016 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:41 smithi078 ceph-mon[99959]: pgmap v13129: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:42.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:41 smithi078 ceph-mon[91136]: pgmap v13129: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:41 smithi146 ceph-mon[103278]: pgmap v13129: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:42.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:42.015Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:42.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:43.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:43.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:43.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:43 smithi146 ceph-mon[103278]: pgmap v13130: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:43 smithi078 ceph-mon[91136]: pgmap v13130: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:44.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:43 smithi078 ceph-mon[99959]: pgmap v13130: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:45 smithi146 ceph-mon[103278]: pgmap v13131: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:45 smithi078 ceph-mon[99959]: pgmap v13131: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:45 smithi078 ceph-mon[91136]: pgmap v13131: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:47 smithi078 ceph-mon[91136]: pgmap v13132: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:47 smithi078 ceph-mon[99959]: pgmap v13132: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:47 smithi146 ceph-mon[103278]: pgmap v13132: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:49.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:49 smithi146 ceph-mon[103278]: pgmap v13133: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:50.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:49 smithi078 ceph-mon[91136]: pgmap v13133: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:49 smithi078 ceph-mon[99959]: pgmap v13133: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:37:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:37:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:37:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:51 smithi146 ceph-mon[103278]: pgmap v13134: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:52.017 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:51 smithi078 ceph-mon[99959]: pgmap v13134: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:52.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:51 smithi078 ceph-mon[91136]: pgmap v13134: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:52.015Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:52.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:37:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:37:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:37:52.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:37:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:53 smithi146 ceph-mon[103278]: pgmap v13135: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:54.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:53 smithi078 ceph-mon[91136]: pgmap v13135: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:37:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:53 smithi078 ceph-mon[99959]: pgmap v13135: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:55.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[99959]: pgmap v13136: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:55.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:55.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:55.981 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:55.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[91136]: pgmap v13136: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:55.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:55.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:55.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:55 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:55 smithi146 ceph-mon[103278]: pgmap v13136: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:37:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:55 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:37:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:57 smithi146 ceph-mon[103278]: pgmap v13137: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:57 smithi078 ceph-mon[91136]: pgmap v13137: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:57 smithi078 ceph-mon[99959]: pgmap v13137: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:37:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:37:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:37:59 smithi078 ceph-mon[91136]: pgmap v13138: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:37:59 smithi078 ceph-mon[99959]: pgmap v13138: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:37:59 smithi146 ceph-mon[103278]: pgmap v13138: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:38:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:38:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:02.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:01 smithi146 ceph-mon[103278]: pgmap v13139: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:02.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:01 smithi078 ceph-mon[99959]: pgmap v13139: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:01 smithi078 ceph-mon[91136]: pgmap v13139: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:02.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:02.016Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:02.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:03 smithi146 ceph-mon[103278]: pgmap v13140: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:04.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:03 smithi078 ceph-mon[99959]: pgmap v13140: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:03 smithi078 ceph-mon[91136]: pgmap v13140: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:05.372 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 7h - - 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 7h - - 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 11s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 6s ago 7h 90.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 11s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 6s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:38:05.895 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 11s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 11s ago 7h 561M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 6s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 11s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 11s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 6s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 11s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 11s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 11s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 11s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 6s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 6s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 6s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 6s ago 7h 97.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 6s ago 7h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:38:05.896 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 11s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:38:05.897 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:38:05.897 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 11s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:38:05.897 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:38:06.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:05 smithi078 ceph-mon[91136]: pgmap v13141: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:06.208 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:38:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:38:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:38:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:06.208 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:38:06.209 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:38:06.209 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:05 smithi078 ceph-mon[99959]: pgmap v13141: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:05 smithi146 ceph-mon[103278]: pgmap v13141: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:06.476 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:38:06.477 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:38:06.478 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:38:06.787 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:38:06.787 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:38:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:38:06.787 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:38:06.787 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:38:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:06 smithi146 ceph-mon[103278]: from='client.73851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:06 smithi146 ceph-mon[103278]: from='client.73857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1898150492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:38:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/86430317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[91136]: from='client.73851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[91136]: from='client.73857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1898150492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/86430317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[99959]: from='client.73851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[99959]: from='client.73857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1898150492' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:38:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/86430317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:38:08.118 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:07 smithi146 ceph-mon[103278]: pgmap v13142: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:08.118 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:07 smithi146 ceph-mon[103278]: from='client.73863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:08.118 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:07 smithi146 ceph-mon[103278]: from='client.73875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:07 smithi078 ceph-mon[91136]: pgmap v13142: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:07 smithi078 ceph-mon[91136]: from='client.73863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:07 smithi078 ceph-mon[91136]: from='client.73875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:07 smithi078 ceph-mon[99959]: pgmap v13142: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:07 smithi078 ceph-mon[99959]: from='client.73863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:07 smithi078 ceph-mon[99959]: from='client.73875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:08.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:08 smithi146 ceph-mon[103278]: pgmap v13143: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:09.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:08 smithi078 ceph-mon[91136]: pgmap v13143: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:08 smithi078 ceph-mon[99959]: pgmap v13143: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:10.905 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:38:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:38:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:38:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:10 smithi146 ceph-mon[103278]: pgmap v13144: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:11.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:10 smithi078 ceph-mon[91136]: pgmap v13144: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:11.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:10 smithi078 ceph-mon[99959]: pgmap v13144: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:12.016Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:12.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:12.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:13.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:12 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:13.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:12 smithi146 ceph-mon[103278]: pgmap v13145: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:13.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:13.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-mon[99959]: pgmap v13145: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:13.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:12 smithi078 ceph-mon[91136]: pgmap v13145: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:14.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:15.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:14 smithi146 ceph-mon[103278]: pgmap v13146: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:15.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:14 smithi078 ceph-mon[99959]: pgmap v13146: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:15.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:14 smithi078 ceph-mon[91136]: pgmap v13146: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:16.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.153 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:15 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.154 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:15 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:16.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:15 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:17.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:16 smithi146 ceph-mon[103278]: pgmap v13147: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:16 smithi078 ceph-mon[91136]: pgmap v13147: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:16 smithi078 ceph-mon[99959]: pgmap v13147: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:19.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:19 smithi146 ceph-mon[103278]: pgmap v13148: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:19.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:19 smithi078 ceph-mon[91136]: pgmap v13148: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:19 smithi078 ceph-mon[99959]: pgmap v13148: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:20.913 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:38:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:38:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:20 smithi146 ceph-mon[103278]: pgmap v13149: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[91136]: pgmap v13149: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[99959]: pgmap v13149: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:22.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:22.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:22.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:22.017Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:22.187 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:22.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:22 smithi146 ceph-mon[103278]: pgmap v13150: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:23.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-mon[91136]: pgmap v13150: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:22 smithi078 ceph-mon[99959]: pgmap v13150: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:24 smithi146 ceph-mon[103278]: pgmap v13151: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:25.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:24 smithi078 ceph-mon[91136]: pgmap v13151: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:25.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:24 smithi078 ceph-mon[99959]: pgmap v13151: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:26 smithi146 ceph-mon[103278]: pgmap v13152: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:27.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:26 smithi078 ceph-mon[91136]: pgmap v13152: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:27.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:26 smithi078 ceph-mon[99959]: pgmap v13152: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:28 smithi146 ceph-mon[103278]: pgmap v13153: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:28 smithi078 ceph-mon[91136]: pgmap v13153: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:28 smithi078 ceph-mon[99959]: pgmap v13153: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:30.928 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:38:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:38:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:38:31.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:30 smithi146 ceph-mon[103278]: pgmap v13154: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:31.285 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:30 smithi078 ceph-mon[99959]: pgmap v13154: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:30 smithi078 ceph-mon[91136]: pgmap v13154: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:32.017Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:32.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:32.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:33.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:32 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:33.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:32 smithi146 ceph-mon[103278]: pgmap v13155: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:33.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-mon[91136]: pgmap v13155: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:32 smithi078 ceph-mon[99959]: pgmap v13155: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:35.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:34 smithi146 ceph-mon[103278]: pgmap v13156: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:35.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:34 smithi078 ceph-mon[91136]: pgmap v13156: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:34 smithi078 ceph-mon[99959]: pgmap v13156: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:36.405 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:36.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:37.080 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:38:37.380 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:37 smithi078 ceph-mon[99959]: pgmap v13157: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:37.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:37 smithi078 ceph-mon[91136]: pgmap v13157: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:37 smithi146 ceph-mon[103278]: pgmap v13157: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 7h - - 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 7h - - 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 2s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 17s ago 7h 90.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 2s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 17s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 2s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:38:37.654 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 2s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 17s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 2s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 2s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 17s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 2s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 2s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 2s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 2s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 17s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 17s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 17s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 17s ago 7h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 17s ago 7h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:38:37.655 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 2s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:38:37.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:38:37.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 2s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:38:37.656 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:38:37.977 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:38:37.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:38:37.977 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:38:37.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:37.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:38:37.977 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:38:37.978 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:38:38.260 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:38:38.261 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[99959]: from='client.73887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[99959]: from='client.93809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[99959]: from='client.93815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/438492907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[91136]: from='client.73887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[91136]: from='client.93809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[91136]: from='client.93815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/438492907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:38:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:38 smithi146 ceph-mon[103278]: from='client.73887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:38 smithi146 ceph-mon[103278]: from='client.93809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:38 smithi146 ceph-mon[103278]: from='client.93815 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/438492907' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:38:38.576 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:38:38.576 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:38:38.576 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:38:38.576 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:38:38.577 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:38:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:39 smithi146 ceph-mon[103278]: pgmap v13158: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:39 smithi146 ceph-mon[103278]: from='client.73911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:39 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3037807372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:38:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:39 smithi078 ceph-mon[91136]: pgmap v13158: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:39 smithi078 ceph-mon[91136]: from='client.73911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:39 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3037807372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:38:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:39 smithi078 ceph-mon[99959]: pgmap v13158: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:39 smithi078 ceph-mon[99959]: from='client.73911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:38:39.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:39 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3037807372' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:38:41.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:38:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:38:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:38:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:41 smithi146 ceph-mon[103278]: pgmap v13159: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[99959]: pgmap v13159: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[91136]: pgmap v13159: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:42.018Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:42.018Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:42.019Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:43 smithi146 ceph-mon[103278]: pgmap v13160: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:43.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:43 smithi078 ceph-mon[99959]: pgmap v13160: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:43 smithi078 ceph-mon[91136]: pgmap v13160: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:38:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:44.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:45.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:45 smithi146 ceph-mon[103278]: pgmap v13161: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:45 smithi078 ceph-mon[99959]: pgmap v13161: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:45 smithi078 ceph-mon[91136]: pgmap v13161: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:47 smithi146 ceph-mon[103278]: pgmap v13162: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:47.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:47 smithi078 ceph-mon[99959]: pgmap v13162: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:47 smithi078 ceph-mon[91136]: pgmap v13162: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:49.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:49 smithi146 ceph-mon[103278]: pgmap v13163: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:49 smithi078 ceph-mon[91136]: pgmap v13163: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:49 smithi078 ceph-mon[99959]: pgmap v13163: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:51.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:38:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:38:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:38:51.395 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:51 smithi146 ceph-mon[103278]: pgmap v13164: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:51.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:51 smithi078 ceph-mon[91136]: pgmap v13164: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:51 smithi078 ceph-mon[99959]: pgmap v13164: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:52.019Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:52.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:38:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:38:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:38:52.020Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:38:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:53 smithi146 ceph-mon[103278]: pgmap v13165: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:53.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:53 smithi078 ceph-mon[91136]: pgmap v13165: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:38:53.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:53 smithi078 ceph-mon[99959]: pgmap v13165: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:55 smithi146 ceph-mon[103278]: pgmap v13166: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:55 smithi078 ceph-mon[91136]: pgmap v13166: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:55 smithi078 ceph-mon[99959]: pgmap v13166: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:38:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:56.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:38:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:57 smithi146 ceph-mon[103278]: pgmap v13167: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:57 smithi078 ceph-mon[91136]: pgmap v13167: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:57 smithi078 ceph-mon[99959]: pgmap v13167: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:59 smithi146 ceph-mon[103278]: pgmap v13168: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:38:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:59 smithi078 ceph-mon[91136]: pgmap v13168: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:38:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:38:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:59 smithi078 ceph-mon[99959]: pgmap v13168: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:38:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:38:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:39:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:39:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:39:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:01 smithi146 ceph-mon[103278]: pgmap v13169: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[91136]: pgmap v13169: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[99959]: pgmap v13169: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:02.020Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:02.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:02.021Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:03 smithi146 ceph-mon[103278]: pgmap v13170: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:03 smithi078 ceph-mon[99959]: pgmap v13170: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:03 smithi078 ceph-mon[91136]: pgmap v13170: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:05 smithi146 ceph-mon[103278]: pgmap v13171: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:05 smithi078 ceph-mon[91136]: pgmap v13171: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:05 smithi078 ceph-mon[99959]: pgmap v13171: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:07 smithi146 ceph-mon[103278]: pgmap v13172: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:07 smithi078 ceph-mon[91136]: pgmap v13172: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:07 smithi078 ceph-mon[99959]: pgmap v13172: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:08.869 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:39:09.432 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:39:09.432 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 7h - - 2024-09-21T03:39:09.432 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 7h - - 2024-09-21T03:39:09.432 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 14s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:39:09.432 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 9s ago 7h 90.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 14s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 9s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 14s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 14s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 9s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 14s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 14s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 9s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 14s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 14s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 14s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 14s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 9s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 9s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:39:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 9s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:39:09.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 9s ago 7h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:39:09.434 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 9s ago 7h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:39:09.434 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 14s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:39:09.434 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:39:09.434 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 14s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:39:09.434 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:39:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:09 smithi146 ceph-mon[103278]: pgmap v13173: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:09.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:09 smithi078 ceph-mon[99959]: pgmap v13173: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:09 smithi078 ceph-mon[91136]: pgmap v13173: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:09.740 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:39:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:39:10.009 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:39:10.009 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:39:10.009 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:39:10.009 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:39:10.010 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:39:10.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[91136]: from='client.73923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[91136]: from='client.73929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[91136]: from='client.73935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2692447434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[99959]: from='client.73923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[99959]: from='client.73929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[99959]: from='client.73935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2692447434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:39:10.321 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:39:10.321 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:39:10.321 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:39:10.321 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:39:10.321 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:39:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:10 smithi146 ceph-mon[103278]: from='client.73923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:10 smithi146 ceph-mon[103278]: from='client.73929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:10 smithi146 ceph-mon[103278]: from='client.73935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:10.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2692447434' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:39:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:39:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:39:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:39:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:11 smithi146 ceph-mon[103278]: pgmap v13174: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:11 smithi146 ceph-mon[103278]: from='client.73947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:11 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/710192382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:39:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:11 smithi078 ceph-mon[91136]: pgmap v13174: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:11 smithi078 ceph-mon[91136]: from='client.73947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:11 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/710192382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:39:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:11 smithi078 ceph-mon[99959]: pgmap v13174: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:11 smithi078 ceph-mon[99959]: from='client.73947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:11 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/710192382' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:39:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:12.021Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:12.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:12.022Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:13 smithi146 ceph-mon[103278]: pgmap v13175: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:13 smithi078 ceph-mon[91136]: pgmap v13175: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:13 smithi078 ceph-mon[99959]: pgmap v13175: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:15 smithi146 ceph-mon[103278]: pgmap v13176: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:15 smithi078 ceph-mon[91136]: pgmap v13176: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:15 smithi078 ceph-mon[99959]: pgmap v13176: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:16.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:17 smithi146 ceph-mon[103278]: pgmap v13177: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:17.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:17 smithi078 ceph-mon[91136]: pgmap v13177: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:17.780 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:17 smithi078 ceph-mon[99959]: pgmap v13177: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:19 smithi146 ceph-mon[103278]: pgmap v13178: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:19 smithi078 ceph-mon[91136]: pgmap v13178: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:19 smithi078 ceph-mon[99959]: pgmap v13178: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:39:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:39:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:39:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:21 smithi146 ceph-mon[103278]: pgmap v13179: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[91136]: pgmap v13179: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[99959]: pgmap v13179: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:21.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:22.022Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:22.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:22.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:23 smithi146 ceph-mon[103278]: pgmap v13180: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:23 smithi078 ceph-mon[91136]: pgmap v13180: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:23.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:23 smithi078 ceph-mon[99959]: pgmap v13180: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:25 smithi146 ceph-mon[103278]: pgmap v13181: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:25.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:25 smithi078 ceph-mon[99959]: pgmap v13181: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:25 smithi078 ceph-mon[91136]: pgmap v13181: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:27 smithi146 ceph-mon[103278]: pgmap v13182: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:27.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:27 smithi078 ceph-mon[91136]: pgmap v13182: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:27 smithi078 ceph-mon[99959]: pgmap v13182: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:29 smithi146 ceph-mon[103278]: pgmap v13183: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:29.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:29 smithi078 ceph-mon[91136]: pgmap v13183: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:29 smithi078 ceph-mon[99959]: pgmap v13183: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:39:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:39:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:39:31.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:31 smithi146 ceph-mon[103278]: pgmap v13184: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:31 smithi078 ceph-mon[91136]: pgmap v13184: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:31 smithi078 ceph-mon[99959]: pgmap v13184: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:32.022Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:32.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:32.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:33 smithi146 ceph-mon[103278]: pgmap v13185: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:33 smithi078 ceph-mon[99959]: pgmap v13185: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:33.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:33 smithi078 ceph-mon[91136]: pgmap v13185: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:33.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:35 smithi146 ceph-mon[103278]: pgmap v13186: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:35.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:35 smithi078 ceph-mon[99959]: pgmap v13186: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:35 smithi078 ceph-mon[91136]: pgmap v13186: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:36.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:36.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:37 smithi078 ceph-mon[91136]: pgmap v13187: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:37 smithi078 ceph-mon[99959]: pgmap v13187: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:37 smithi146 ceph-mon[103278]: pgmap v13187: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:39.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:39 smithi078 ceph-mon[91136]: pgmap v13188: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:39.778 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:39 smithi078 ceph-mon[99959]: pgmap v13188: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:39 smithi146 ceph-mon[103278]: pgmap v13188: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:40.641 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:39:40.760 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:39:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:39:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:39:41.212 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:39:41.212 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 7h - - 2024-09-21T03:39:41.212 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 7h - - 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 5s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 0s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 5s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 0s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 5s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 5s ago 7h 562M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 0s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 5s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 5s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 0s ago 7h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 5s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:39:41.213 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 5s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 5s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 5s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 0s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 0s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 0s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 0s ago 7h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 0s ago 7h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 5s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 5s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:39:41.214 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:39:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:39:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:39:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:39:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:39:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:39:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:39:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:41 smithi146 ceph-mon[103278]: pgmap v13189: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:41 smithi146 ceph-mon[103278]: from='client.73959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[99959]: pgmap v13189: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:41.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[99959]: from='client.73959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[91136]: pgmap v13189: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:41 smithi078 ceph-mon[91136]: from='client.73959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:41.801 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:39:41.801 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:39:41.801 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:39:41.801 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:39:41.801 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:39:41.802 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:39:42.112 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:39:42.113 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:39:42.113 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:39:42.113 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:39:42.113 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:39:42.113 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:42.022Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:42.113 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:42.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:42.113 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:42.023Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:42 smithi146 ceph-mon[103278]: from='client.93881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:42 smithi146 ceph-mon[103278]: from='client.93887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3496656570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:39:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:42 smithi146 ceph-mon[103278]: from='client.93899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1378478244' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:39:42.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[91136]: from='client.93881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[91136]: from='client.93887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3496656570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[91136]: from='client.93899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1378478244' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[99959]: from='client.93881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[99959]: from='client.93887 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3496656570' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[99959]: from='client.93899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:39:42.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1378478244' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:39:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:43 smithi146 ceph-mon[103278]: pgmap v13190: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:43 smithi078 ceph-mon[91136]: pgmap v13190: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:43 smithi078 ceph-mon[99959]: pgmap v13190: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:45 smithi146 ceph-mon[103278]: pgmap v13191: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:45.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:45 smithi078 ceph-mon[99959]: pgmap v13191: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:45 smithi078 ceph-mon[91136]: pgmap v13191: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:47 smithi146 ceph-mon[103278]: pgmap v13192: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:47 smithi078 ceph-mon[91136]: pgmap v13192: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:47 smithi078 ceph-mon[99959]: pgmap v13192: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:49 smithi146 ceph-mon[103278]: pgmap v13193: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:49 smithi078 ceph-mon[91136]: pgmap v13193: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:49 smithi078 ceph-mon[99959]: pgmap v13193: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:39:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:39:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:39:51.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:51 smithi146 ceph-mon[103278]: pgmap v13194: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:51.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:51 smithi078 ceph-mon[91136]: pgmap v13194: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:51 smithi078 ceph-mon[99959]: pgmap v13194: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:52.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:52.024Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:39:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:39:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:39:52.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:39:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:53 smithi146 ceph-mon[103278]: pgmap v13195: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:53 smithi078 ceph-mon[99959]: pgmap v13195: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:53.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:53 smithi078 ceph-mon[91136]: pgmap v13195: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:39:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:55 smithi146 ceph-mon[103278]: pgmap v13196: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:55 smithi078 ceph-mon[91136]: pgmap v13196: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:55 smithi078 ceph-mon[99959]: pgmap v13196: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:39:57.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:56 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:56 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:57.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:56 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:39:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:57 smithi146 ceph-mon[103278]: pgmap v13197: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:57 smithi078 ceph-mon[91136]: pgmap v13197: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:57 smithi078 ceph-mon[99959]: pgmap v13197: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:59.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:39:59.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:39:59 smithi078 ceph-mon[91136]: pgmap v13198: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:39:59.766 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:39:59 smithi078 ceph-mon[99959]: pgmap v13198: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:39:59 smithi146 ceph-mon[103278]: pgmap v13198: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:00.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T03:39:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:40:01.013 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:40:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:40:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:01.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:01 smithi078 ceph-mon[91136]: pgmap v13199: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:01.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:01 smithi078 ceph-mon[99959]: pgmap v13199: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:01 smithi146 ceph-mon[103278]: pgmap v13199: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:02.024Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:02.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:02.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:03 smithi146 ceph-mon[103278]: pgmap v13200: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:04.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:03 smithi078 ceph-mon[99959]: pgmap v13200: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:03 smithi078 ceph-mon[91136]: pgmap v13200: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:06.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:05 smithi146 ceph-mon[103278]: pgmap v13201: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:06.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:05 smithi078 ceph-mon[99959]: pgmap v13201: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:05 smithi078 ceph-mon[91136]: pgmap v13201: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:07 smithi146 ceph-mon[103278]: pgmap v13202: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:08.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:07 smithi078 ceph-mon[99959]: pgmap v13202: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:07 smithi078 ceph-mon[91136]: pgmap v13202: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:09 smithi146 ceph-mon[103278]: pgmap v13203: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:09 smithi078 ceph-mon[99959]: pgmap v13203: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:09 smithi078 ceph-mon[91136]: pgmap v13203: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:11.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:40:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:40:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:40:11.934 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:11 smithi078 ceph-mon[91136]: pgmap v13204: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:11.934 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:11 smithi078 ceph-mon[99959]: pgmap v13204: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:11 smithi146 ceph-mon[103278]: pgmap v13204: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:12.285 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:12.025Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:12.025Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:12.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:12.409 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:40:12.938 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:40:12.938 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 7h - - 2024-09-21T03:40:12.938 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 7h - - 2024-09-21T03:40:12.938 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 17s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:40:12.938 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 12s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:40:12.938 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 17s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 12s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 17s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 17s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 12s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 17s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 17s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 12s ago 7h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 17s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 17s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 17s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 17s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 12s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 12s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 12s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:40:12.939 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 12s ago 7h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:40:12.940 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 12s ago 7h 73.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:40:12.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 17s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:40:12.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:40:12.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 17s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:40:12.940 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:40:13.250 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:40:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:40:13.517 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:40:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:40:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:13 smithi146 ceph-mon[103278]: pgmap v13205: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:13 smithi146 ceph-mon[103278]: from='client.93911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:13 smithi146 ceph-mon[103278]: from='client.93917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/405836800' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:40:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[91136]: pgmap v13205: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[91136]: from='client.93911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[91136]: from='client.93917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/405836800' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[99959]: pgmap v13205: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[99959]: from='client.93911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[99959]: from='client.93917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/405836800' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:40:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:13.831 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:40:13.831 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:40:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:40:13.831 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:40:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:40:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:14 smithi146 ceph-mon[103278]: from='client.93923 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:14 smithi146 ceph-mon[103278]: from='client.74019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:15.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2673970883' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:40:15.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:14 smithi078 ceph-mon[99959]: from='client.93923 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:14 smithi078 ceph-mon[99959]: from='client.74019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2673970883' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:14 smithi078 ceph-mon[91136]: from='client.93923 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:14 smithi078 ceph-mon[91136]: from='client.74019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2673970883' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:40:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:15 smithi146 ceph-mon[103278]: pgmap v13206: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:16.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:15 smithi078 ceph-mon[91136]: pgmap v13206: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:16.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:15 smithi078 ceph-mon[99959]: pgmap v13206: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:16 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:16 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:17.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:16 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:18.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:17 smithi078 ceph-mon[99959]: pgmap v13207: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:17 smithi078 ceph-mon[91136]: pgmap v13207: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:17 smithi146 ceph-mon[103278]: pgmap v13207: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:19 smithi078 ceph-mon[91136]: pgmap v13208: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:19 smithi078 ceph-mon[99959]: pgmap v13208: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:19 smithi146 ceph-mon[103278]: pgmap v13208: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:20.785 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:40:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:40:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:40:21.727 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[99959]: pgmap v13209: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[91136]: pgmap v13209: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:21.728 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:21 smithi146 ceph-mon[103278]: pgmap v13209: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:22.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:22.026Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:22.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:23 smithi078 ceph-mon[91136]: pgmap v13210: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:23 smithi078 ceph-mon[99959]: pgmap v13210: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:23 smithi146 ceph-mon[103278]: pgmap v13210: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:25 smithi146 ceph-mon[103278]: pgmap v13211: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:25 smithi078 ceph-mon[91136]: pgmap v13211: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:25 smithi078 ceph-mon[99959]: pgmap v13211: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:27 smithi146 ceph-mon[103278]: pgmap v13212: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:28.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:27 smithi078 ceph-mon[91136]: pgmap v13212: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:27 smithi078 ceph-mon[99959]: pgmap v13212: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:29.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:29.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:29.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:29 smithi146 ceph-mon[103278]: pgmap v13213: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:29 smithi078 ceph-mon[91136]: pgmap v13213: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:29 smithi078 ceph-mon[99959]: pgmap v13213: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:40:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:40:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:40:31.926 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:31 smithi146 ceph-mon[103278]: pgmap v13214: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:32.027 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:31 smithi078 ceph-mon[99959]: pgmap v13214: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:32.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:31 smithi078 ceph-mon[91136]: pgmap v13214: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:32.026Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:32.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:32.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:33 smithi078 ceph-mon[91136]: pgmap v13215: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:33 smithi078 ceph-mon[99959]: pgmap v13215: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:33.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:33 smithi146 ceph-mon[103278]: pgmap v13215: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:33.937 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:35 smithi146 ceph-mon[103278]: pgmap v13216: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:35 smithi078 ceph-mon[91136]: pgmap v13216: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:35 smithi078 ceph-mon[99959]: pgmap v13216: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:36 smithi146 ceph-mon[103278]: pgmap v13217: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:36 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[99959]: pgmap v13217: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[91136]: pgmap v13217: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:37.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:36 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:38 smithi146 ceph-mon[103278]: pgmap v13218: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:38 smithi078 ceph-mon[99959]: pgmap v13218: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:38 smithi078 ceph-mon[91136]: pgmap v13218: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:40.903 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:40:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:40:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[91136]: pgmap v13219: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[99959]: pgmap v13219: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:41.955 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:42.253 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:42.027Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:42.253 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:42.027Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:42.253 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:42.028Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:41 smithi146 ceph-mon[103278]: pgmap v13219: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:44.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:43 smithi078 ceph-mon[99959]: pgmap v13220: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:44.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:44.126 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:44.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:43 smithi078 ceph-mon[91136]: pgmap v13220: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:44.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:44.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:44.127 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:40:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:43 smithi146 ceph-mon[103278]: pgmap v13220: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 7h - - 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 7h - - 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 8s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 3s ago 7h 90.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 8s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:40:44.651 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 3s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 8s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 8s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 3s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 8s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 8s ago 7h 15.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 3s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 8s ago 7h 99.2M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 8s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 8s ago 7h 86.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 8s ago 7h 106M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 3s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 3s ago 7h 95.0M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 3s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 3s ago 7h 97.8M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 3s ago 7h 69.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:40:44.652 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 8s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:40:44.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:40:44.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 8s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:40:44.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:40:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:40:44.963 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:40:45.233 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:40:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:40:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:40:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:40:45.543 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:40:45.543 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:40:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:40:45.543 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:40:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:40:46.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:45 smithi146 ceph-mon[103278]: pgmap v13221: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:46.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:45 smithi146 ceph-mon[103278]: from='client.74031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:45 smithi146 ceph-mon[103278]: from='client.74037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:45 smithi146 ceph-mon[103278]: from='client.74043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3455170580' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:40:46.070 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1129261595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:40:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[91136]: pgmap v13221: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[91136]: from='client.74031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[91136]: from='client.74037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[91136]: from='client.74043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3455170580' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:40:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1129261595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:40:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[99959]: pgmap v13221: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[99959]: from='client.74031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[99959]: from='client.74037 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[99959]: from='client.74043 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3455170580' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:40:46.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1129261595' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:40:47.249 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:46 smithi146 ceph-mon[103278]: from='client.74055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:47.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:46 smithi078 ceph-mon[91136]: from='client.74055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:46 smithi078 ceph-mon[99959]: from='client.74055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:40:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:47 smithi146 ceph-mon[103278]: pgmap v13222: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:48.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:47 smithi078 ceph-mon[91136]: pgmap v13222: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:47 smithi078 ceph-mon[99959]: pgmap v13222: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:49 smithi146 ceph-mon[103278]: pgmap v13223: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:50.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:49 smithi078 ceph-mon[91136]: pgmap v13223: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:49 smithi078 ceph-mon[99959]: pgmap v13223: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:51.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:50 smithi078 ceph-mon[91136]: pgmap v13224: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:51.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:50 smithi078 ceph-mon[99959]: pgmap v13224: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:40:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:40:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:40:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:50 smithi146 ceph-mon[103278]: pgmap v13224: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:52.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:52.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:40:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:40:52.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:40:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-mon[91136]: pgmap v13225: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-mon[99959]: pgmap v13225: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:52 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:52 smithi146 ceph-mon[103278]: pgmap v13225: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:40:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:52 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:40:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:54 smithi146 ceph-mon[103278]: pgmap v13226: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:55.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:54 smithi078 ceph-mon[91136]: pgmap v13226: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:54 smithi078 ceph-mon[99959]: pgmap v13226: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:40:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:57 smithi146 ceph-mon[103278]: pgmap v13227: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:40:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[91136]: pgmap v13227: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[99959]: pgmap v13227: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:40:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:59 smithi146 ceph-mon[103278]: pgmap v13228: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:40:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:40:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:59.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:59 smithi078 ceph-mon[99959]: pgmap v13228: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:40:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:40:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:40:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:59 smithi078 ceph-mon[91136]: pgmap v13228: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:40:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:40:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:41:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:41:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:41:01.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:01 smithi146 ceph-mon[103278]: pgmap v13229: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:41:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[99959]: pgmap v13229: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[91136]: pgmap v13229: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:01.510 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:02.028Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:02.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:02.029Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:03 smithi078 ceph-mon[99959]: pgmap v13230: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:41:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:03 smithi078 ceph-mon[91136]: pgmap v13230: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:41:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:03 smithi146 ceph-mon[103278]: pgmap v13230: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:41:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:05 smithi146 ceph-mon[103278]: pgmap v13231: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:41:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:05 smithi078 ceph-mon[91136]: pgmap v13231: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:41:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:05 smithi078 ceph-mon[99959]: pgmap v13231: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:41:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:07 smithi146 ceph-mon[103278]: pgmap v13232: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:07 smithi078 ceph-mon[99959]: pgmap v13232: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:07 smithi078 ceph-mon[91136]: pgmap v13232: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:09 smithi146 ceph-mon[103278]: pgmap v13233: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:41:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:09 smithi078 ceph-mon[91136]: pgmap v13233: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:41:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:09 smithi078 ceph-mon[99959]: pgmap v13233: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:41:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:41:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:41:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:41:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:11 smithi146 ceph-mon[103278]: pgmap v13234: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:41:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:11 smithi078 ceph-mon[91136]: pgmap v13234: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:41:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:11 smithi078 ceph-mon[99959]: pgmap v13234: 161 pgs: 161 active+clean; 585 KiB data, 398 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:41:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:12.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:12.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:12.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:13.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:13 smithi146 ceph-mon[103278]: pgmap v13235: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:41:13.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:13 smithi078 ceph-mon[91136]: pgmap v13235: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:41:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:13 smithi078 ceph-mon[99959]: pgmap v13235: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:41:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:14.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:15.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:15 smithi146 ceph-mon[103278]: pgmap v13236: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-21T03:41:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:15 smithi078 ceph-mon[91136]: pgmap v13236: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-21T03:41:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:15 smithi078 ceph-mon[99959]: pgmap v13236: 161 pgs: 161 active+clean; 585 KiB data, 406 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-21T03:41:15.840 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 7h - - 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 7h - - 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 0s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 15s ago 7h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 0s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:41:16.380 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 15s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 0s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 0s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 15s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 0s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 0s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 15s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 0s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 0s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 0s ago 7h 86.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 0s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 15s ago 7h 122M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 15s ago 7h 95.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:41:16.381 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 15s ago 7h 102M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:41:16.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 15s ago 7h 97.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:41:16.382 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 15s ago 7h 69.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:41:16.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 0s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:41:16.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:41:16.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 0s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:41:16.382 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:41:16.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:41:16.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:41:16.966 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:41:16.967 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:41:16.967 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:41:16.967 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:41:16.967 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:41:16.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:41:17.277 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:41:17.278 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:41:17.278 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:41:17.278 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:41:17.278 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:41:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='client.74067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: pgmap v13237: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='client.74073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='client.74079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1549266269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='client.74067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: pgmap v13237: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='client.74073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.279 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='client.74079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.280 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1549266269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:41:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='client.74067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: pgmap v13237: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:41:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='client.74073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='client.74079 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:17.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1549266269' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:41:18.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:18 smithi146 ceph-mon[103278]: from='client.94007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:18.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:18 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1862525973' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:41:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:18 smithi078 ceph-mon[99959]: from='client.94007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:18.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:18 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1862525973' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:41:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:18 smithi078 ceph-mon[91136]: from='client.94007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:18.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:18 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1862525973' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:41:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:19 smithi146 ceph-mon[103278]: pgmap v13238: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:41:19.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:19 smithi078 ceph-mon[91136]: pgmap v13238: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:41:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:19 smithi078 ceph-mon[99959]: pgmap v13238: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 115 op/s 2024-09-21T03:41:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:41:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:41:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:41:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:21 smithi146 ceph-mon[103278]: pgmap v13239: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:41:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[99959]: pgmap v13239: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[91136]: pgmap v13239: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 133 op/s 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:22.030Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:22.030Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:22.031Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:23.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:23 smithi078 ceph-mon[91136]: pgmap v13240: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:41:23.463 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:23.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:23 smithi078 ceph-mon[99959]: pgmap v13240: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:41:23.464 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:23 smithi146 ceph-mon[103278]: pgmap v13240: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 148 op/s 2024-09-21T03:41:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:25 smithi078 ceph-mon[91136]: pgmap v13241: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:41:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:25 smithi078 ceph-mon[99959]: pgmap v13241: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:41:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:25 smithi146 ceph-mon[103278]: pgmap v13241: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:41:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:27 smithi146 ceph-mon[103278]: pgmap v13242: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:41:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:27 smithi078 ceph-mon[91136]: pgmap v13242: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:41:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:27 smithi078 ceph-mon[99959]: pgmap v13242: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:41:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:29 smithi146 ceph-mon[103278]: pgmap v13243: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:41:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:29.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:29 smithi078 ceph-mon[99959]: pgmap v13243: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:41:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:29 smithi078 ceph-mon[91136]: pgmap v13243: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:41:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:41:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:41:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:41:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:31 smithi146 ceph-mon[103278]: pgmap v13244: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:41:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:31 smithi078 ceph-mon[99959]: pgmap v13244: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:41:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:31 smithi078 ceph-mon[91136]: pgmap v13244: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:41:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:32.031Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:32.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:32.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:33.444 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:33 smithi146 ceph-mon[103278]: pgmap v13245: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:41:33.445 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:33 smithi078 ceph-mon[99959]: pgmap v13245: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:41:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:33 smithi078 ceph-mon[91136]: pgmap v13245: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-09-21T03:41:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:35 smithi146 ceph-mon[103278]: pgmap v13246: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:41:35.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:35 smithi078 ceph-mon[91136]: pgmap v13246: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:41:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:35 smithi078 ceph-mon[99959]: pgmap v13246: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-09-21T03:41:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:37 smithi146 ceph-mon[103278]: pgmap v13247: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:41:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[99959]: pgmap v13247: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[91136]: pgmap v13247: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:39 smithi146 ceph-mon[103278]: pgmap v13248: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:39.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:39 smithi078 ceph-mon[91136]: pgmap v13248: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:39 smithi078 ceph-mon[99959]: pgmap v13248: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:41:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:41:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:41:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:41 smithi146 ceph-mon[103278]: pgmap v13249: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[91136]: pgmap v13249: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[99959]: pgmap v13249: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:41:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:42.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:42.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:42.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:43.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:43 smithi078 ceph-mon[99959]: pgmap v13250: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:43.403 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:43.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:43 smithi078 ceph-mon[91136]: pgmap v13250: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:43.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:43 smithi146 ceph-mon[103278]: pgmap v13250: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:43.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:44.437 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:44.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:45.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:45 smithi078 ceph-mon[91136]: pgmap v13251: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:45.578 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:45 smithi078 ceph-mon[99959]: pgmap v13251: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:45 smithi146 ceph-mon[103278]: pgmap v13251: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:47.571 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:47 smithi078 ceph-mon[99959]: pgmap v13252: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:47.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:47 smithi078 ceph-mon[91136]: pgmap v13252: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:47.572 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:41:47.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:47 smithi146 ceph-mon[103278]: pgmap v13252: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:48.095 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 7h - - 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 7h - - 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 11s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 6s ago 7h 91.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 11s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 6s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 11s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 11s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 6s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 11s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 11s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 6s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 11s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 11s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 11s ago 7h 87.9M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:41:48.096 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 11s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 6s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 6s ago 7h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 6s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 6s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 6s ago 7h 71.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 11s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 11s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:41:48.097 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:41:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:41:48.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:41:48.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:41:48.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:48.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:41:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:41:48.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:48 smithi078 ceph-mon[91136]: from='client.94019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:48.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:48 smithi078 ceph-mon[91136]: from='client.74109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:48.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:48 smithi078 ceph-mon[99959]: from='client.94019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:48.410 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:48 smithi078 ceph-mon[99959]: from='client.74109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:48 smithi146 ceph-mon[103278]: from='client.94019 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:48.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:48 smithi146 ceph-mon[103278]: from='client.74109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:41:48.680 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:41:48.681 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:41:48.681 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:41:48.990 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:41:48.990 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:41:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:41:48.991 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:41:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:41:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:49 smithi146 ceph-mon[103278]: pgmap v13253: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:49 smithi146 ceph-mon[103278]: from='client.74115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3509390206' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:41:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:49 smithi146 ceph-mon[103278]: from='client.94043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:49.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:49 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2581264542' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[99959]: pgmap v13253: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[99959]: from='client.74115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3509390206' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[99959]: from='client.94043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2581264542' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[91136]: pgmap v13253: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[91136]: from='client.74115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3509390206' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:41:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[91136]: from='client.94043 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:41:49.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:49 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2581264542' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:41:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:41:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:41:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:41:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:51 smithi146 ceph-mon[103278]: pgmap v13254: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:51 smithi078 ceph-mon[99959]: pgmap v13254: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:51 smithi078 ceph-mon[91136]: pgmap v13254: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:52.032Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:52.032Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:41:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:41:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:41:52.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:41:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:53 smithi146 ceph-mon[103278]: pgmap v13255: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:53 smithi078 ceph-mon[91136]: pgmap v13255: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:53 smithi078 ceph-mon[99959]: pgmap v13255: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:41:55.417 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:55 smithi146 ceph-mon[103278]: pgmap v13256: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:55 smithi078 ceph-mon[91136]: pgmap v13256: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:55 smithi078 ceph-mon[99959]: pgmap v13256: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:41:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:57 smithi146 ceph-mon[103278]: pgmap v13257: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[91136]: pgmap v13257: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[99959]: pgmap v13257: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:41:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:59 smithi146 ceph-mon[103278]: pgmap v13258: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:41:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:59.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:59 smithi078 ceph-mon[91136]: pgmap v13258: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:41:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:41:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:59 smithi078 ceph-mon[99959]: pgmap v13258: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:41:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:41:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:42:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:42:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:42:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:01 smithi146 ceph-mon[103278]: pgmap v13259: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[91136]: pgmap v13259: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[99959]: pgmap v13259: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:02.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:02.033Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:02.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:03 smithi146 ceph-mon[103278]: pgmap v13260: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:03.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:03 smithi078 ceph-mon[91136]: pgmap v13260: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:03 smithi078 ceph-mon[99959]: pgmap v13260: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:05 smithi146 ceph-mon[103278]: pgmap v13261: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:05 smithi078 ceph-mon[91136]: pgmap v13261: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:05 smithi078 ceph-mon[99959]: pgmap v13261: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:07 smithi146 ceph-mon[103278]: pgmap v13262: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:07.785 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:07 smithi078 ceph-mon[99959]: pgmap v13262: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:07 smithi078 ceph-mon[91136]: pgmap v13262: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:09 smithi146 ceph-mon[103278]: pgmap v13263: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:09 smithi078 ceph-mon[99959]: pgmap v13263: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:09 smithi078 ceph-mon[91136]: pgmap v13263: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:42:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:42:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:42:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:11 smithi146 ceph-mon[103278]: pgmap v13264: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:11 smithi078 ceph-mon[99959]: pgmap v13264: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:11 smithi078 ceph-mon[91136]: pgmap v13264: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:12.033Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:12.034Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:12.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:13 smithi146 ceph-mon[103278]: pgmap v13265: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:13 smithi078 ceph-mon[99959]: pgmap v13265: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:13 smithi078 ceph-mon[91136]: pgmap v13265: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:15.518 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:15 smithi146 ceph-mon[103278]: pgmap v13266: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:15 smithi078 ceph-mon[91136]: pgmap v13266: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:15 smithi078 ceph-mon[99959]: pgmap v13266: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:17 smithi146 ceph-mon[103278]: pgmap v13267: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[91136]: pgmap v13267: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[99959]: pgmap v13267: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:18.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:18.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:19.290 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:42:19.817 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 7h - - 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 7h - - 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 3s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 18s ago 7h 89.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 3s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 18s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 3s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 3s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 18s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 3s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 3s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:42:19.818 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 18s ago 7h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 3s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 3s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 3s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 3s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 18s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 18s ago 7h 95.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 18s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 18s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 18s ago 7h 71.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 3s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 3s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:42:19.819 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:42:19.820 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:19 smithi078 ceph-mon[99959]: pgmap v13268: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:19.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:19 smithi078 ceph-mon[91136]: pgmap v13268: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:20.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:19 smithi146 ceph-mon[103278]: pgmap v13268: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:20.124 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:42:20.125 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:42:20.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:42:20.125 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:42:20.392 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:42:20.393 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:42:20.393 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:42:20.393 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:42:20.393 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:42:20.704 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:42:20.705 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:42:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:42:20.705 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:42:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:42:20.705 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:42:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:42:20.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[91136]: from='client.74139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:20.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[91136]: from='client.94061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:20.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[91136]: from='client.74151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:20.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3805771533' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:42:20.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[99959]: from='client.74139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:20.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[99959]: from='client.94061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:20.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[99959]: from='client.74151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:20.706 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3805771533' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:42:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:20 smithi146 ceph-mon[103278]: from='client.74139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:20 smithi146 ceph-mon[103278]: from='client.94061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:20 smithi146 ceph-mon[103278]: from='client.74151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3805771533' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:42:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:21 smithi146 ceph-mon[103278]: pgmap v13269: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:21 smithi146 ceph-mon[103278]: from='client.94079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:21 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3580859099' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:42:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[99959]: pgmap v13269: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[99959]: from='client.94079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3580859099' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[91136]: pgmap v13269: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[91136]: from='client.94079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:22.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3580859099' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:42:22.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:22.531 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:22.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:22.531 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:22.035Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:22.531 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:22.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:23.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:23 smithi078 ceph-mon[99959]: pgmap v13270: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:23.989 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:23.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:23 smithi078 ceph-mon[91136]: pgmap v13270: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:23.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:23 smithi146 ceph-mon[103278]: pgmap v13270: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:25.865 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:25 smithi078 ceph-mon[99959]: pgmap v13271: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:25.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:25 smithi078 ceph-mon[91136]: pgmap v13271: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:25 smithi146 ceph-mon[103278]: pgmap v13271: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:27 smithi078 ceph-mon[91136]: pgmap v13272: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:27 smithi078 ceph-mon[99959]: pgmap v13272: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:28.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:27 smithi146 ceph-mon[103278]: pgmap v13272: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:29 smithi146 ceph-mon[103278]: pgmap v13273: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:30.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:30.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:29 smithi078 ceph-mon[99959]: pgmap v13273: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:29 smithi078 ceph-mon[91136]: pgmap v13273: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:31.035 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:42:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:42:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:42:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:31 smithi146 ceph-mon[103278]: pgmap v13274: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:31 smithi078 ceph-mon[91136]: pgmap v13274: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:31 smithi078 ceph-mon[99959]: pgmap v13274: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:32.035Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:32.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:32.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:33 smithi146 ceph-mon[103278]: pgmap v13275: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:33 smithi078 ceph-mon[91136]: pgmap v13275: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:33 smithi078 ceph-mon[99959]: pgmap v13275: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:35 smithi146 ceph-mon[103278]: pgmap v13276: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:36.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:35 smithi078 ceph-mon[91136]: pgmap v13276: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:35 smithi078 ceph-mon[99959]: pgmap v13276: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:37 smithi146 ceph-mon[103278]: pgmap v13277: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:37 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[91136]: pgmap v13277: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[99959]: pgmap v13277: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:37 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:39 smithi146 ceph-mon[103278]: pgmap v13278: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:40.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:39 smithi078 ceph-mon[91136]: pgmap v13278: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:39 smithi078 ceph-mon[99959]: pgmap v13278: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:42:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:42:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:42:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:41 smithi146 ceph-mon[103278]: pgmap v13279: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[99959]: pgmap v13279: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[91136]: pgmap v13279: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:42.036Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:42.036Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:42.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:44.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:43 smithi146 ceph-mon[103278]: pgmap v13280: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:43 smithi078 ceph-mon[99959]: pgmap v13280: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:43 smithi078 ceph-mon[91136]: pgmap v13280: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:45.839 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:45 smithi078 ceph-mon[99959]: pgmap v13281: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:45.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:45 smithi078 ceph-mon[91136]: pgmap v13281: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:45 smithi146 ceph-mon[103278]: pgmap v13281: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:47 smithi078 ceph-mon[99959]: pgmap v13282: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:47 smithi078 ceph-mon[91136]: pgmap v13282: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:47 smithi146 ceph-mon[103278]: pgmap v13282: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:49.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:49 smithi146 ceph-mon[103278]: pgmap v13283: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:49 smithi078 ceph-mon[91136]: pgmap v13283: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:49 smithi078 ceph-mon[99959]: pgmap v13283: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:50.998 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:42:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:42:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:42:50.998 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:42:51.530 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:42:51.530 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 7h - - 2024-09-21T03:42:51.530 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 7h - - 2024-09-21T03:42:51.530 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 14s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 9s ago 7h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 14s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 9s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 14s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 14s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 9s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 14s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 14s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 9s ago 7h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 14s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 14s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 14s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 14s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 9s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:42:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 9s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 9s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 9s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 9s ago 7h 71.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 14s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 14s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:42:51.532 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:42:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:51 smithi078 ceph-mon[99959]: pgmap v13284: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:51 smithi078 ceph-mon[91136]: pgmap v13284: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:42:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:42:51.846 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:42:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:51 smithi146 ceph-mon[103278]: pgmap v13284: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:42:52.114 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:42:52.115 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:42:52.115 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:42:52.115 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:42:52.115 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:42:52.115 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:52.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:52.115 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:52.037Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:42:52.115 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:42:52.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:42:52.433 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:42:52.433 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:42:52.433 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:42:52.433 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:42:52.433 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:42:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:52 smithi146 ceph-mon[103278]: from='client.74175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:52 smithi146 ceph-mon[103278]: from='client.74181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:52 smithi146 ceph-mon[103278]: from='client.74187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3638689666' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:42:53.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4164193552' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:42:53.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[99959]: from='client.74175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[99959]: from='client.74181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[99959]: from='client.74187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3638689666' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4164193552' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[91136]: from='client.74175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[91136]: from='client.74181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[91136]: from='client.74187 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3638689666' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:42:53.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4164193552' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:53 smithi146 ceph-mon[103278]: pgmap v13285: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:53 smithi146 ceph-mon[103278]: from='client.74199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:53 smithi078 ceph-mon[91136]: pgmap v13285: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:53 smithi078 ceph-mon[91136]: from='client.74199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:53 smithi078 ceph-mon[99959]: pgmap v13285: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:42:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:53 smithi078 ceph-mon[99959]: from='client.74199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:42:55.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:55 smithi146 ceph-mon[103278]: pgmap v13286: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:55 smithi078 ceph-mon[91136]: pgmap v13286: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:55 smithi078 ceph-mon[99959]: pgmap v13286: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:42:57.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:57 smithi146 ceph-mon[103278]: pgmap v13287: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:57.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:57.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:57.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:57 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[91136]: pgmap v13287: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[99959]: pgmap v13287: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:42:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:58.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:57 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:42:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:42:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:42:59 smithi146 ceph-mon[103278]: pgmap v13288: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:42:59 smithi078 ceph-mon[99959]: pgmap v13288: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:42:59 smithi078 ceph-mon[91136]: pgmap v13288: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:43:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:43:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:43:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:01 smithi146 ceph-mon[103278]: pgmap v13289: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[99959]: pgmap v13289: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:02.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.088 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[91136]: pgmap v13289: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:02.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.089 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:02.089 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:02.037Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:02.089 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:02.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:02.089 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:02.038Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:03 smithi146 ceph-mon[103278]: pgmap v13290: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:03 smithi078 ceph-mon[91136]: pgmap v13290: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:03 smithi078 ceph-mon[99959]: pgmap v13290: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:06.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:05 smithi078 ceph-mon[91136]: pgmap v13291: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:06.214 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:05 smithi078 ceph-mon[99959]: pgmap v13291: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:06.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:05 smithi146 ceph-mon[103278]: pgmap v13291: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:07 smithi078 ceph-mon[91136]: pgmap v13292: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:07 smithi078 ceph-mon[99959]: pgmap v13292: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:07 smithi146 ceph-mon[103278]: pgmap v13292: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:09 smithi146 ceph-mon[103278]: pgmap v13293: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:09 smithi078 ceph-mon[91136]: pgmap v13293: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:09 smithi078 ceph-mon[99959]: pgmap v13293: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:43:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:43:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:43:12.099 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:11 smithi078 ceph-mon[91136]: pgmap v13294: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:12.099 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:11 smithi078 ceph-mon[99959]: pgmap v13294: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:12.099 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:12.038Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:12.100 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:12.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:12.100 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:12.039Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:11 smithi146 ceph-mon[103278]: pgmap v13294: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:13 smithi146 ceph-mon[103278]: pgmap v13295: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:13 smithi078 ceph-mon[91136]: pgmap v13295: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:13 smithi078 ceph-mon[99959]: pgmap v13295: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:16.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:15 smithi146 ceph-mon[103278]: pgmap v13296: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:15 smithi078 ceph-mon[91136]: pgmap v13296: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:15 smithi078 ceph-mon[99959]: pgmap v13296: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:18.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:17 smithi146 ceph-mon[103278]: pgmap v13297: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:18.212 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.213 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:17 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[91136]: pgmap v13297: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[99959]: pgmap v13297: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:18.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:17 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:19.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:18 smithi146 ceph-mon[103278]: pgmap v13298: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:19.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:18 smithi078 ceph-mon[91136]: pgmap v13298: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:19.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:18 smithi078 ceph-mon[99959]: pgmap v13298: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:21.012 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:43:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:43:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:43:21.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:21 smithi146 ceph-mon[103278]: pgmap v13299: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:21.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:21 smithi078 ceph-mon[91136]: pgmap v13299: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:21.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:21 smithi078 ceph-mon[99959]: pgmap v13299: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:22.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:22.039Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:22.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:22.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:22.435 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:22.040Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:22.746 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[99959]: pgmap v13300: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[99959]: from='client.74211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[91136]: pgmap v13300: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:23.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:22 smithi078 ceph-mon[91136]: from='client.74211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:22 smithi146 ceph-mon[103278]: pgmap v13300: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:22 smithi146 ceph-mon[103278]: from='client.74211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 7h - - 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 7h - - 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 6s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 1s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 6s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 1s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 6s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:43:23.309 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 6s ago 7h 568M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 1s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 6s ago 7h 386M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 6s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 1s ago 7h 15.6M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 6s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 6s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 6s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 6s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 1s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 1s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 1s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 1s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:43:23.310 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 1s ago 7h 72.8M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:43:23.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 6s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:43:23.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:43:23.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 6s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:43:23.311 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:43:23.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:43:23.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:23.631 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:43:23.631 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:43:23.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:43:23.631 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:43:23.921 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:43:23.921 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:43:23.922 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:43:23.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:23 smithi078 ceph-mon[99959]: from='client.94133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:23 smithi078 ceph-mon[99959]: from='client.94139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.923 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3173311256' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:43:23.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:23 smithi078 ceph-mon[91136]: from='client.94133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:23 smithi078 ceph-mon[91136]: from='client.94139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:23.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3173311256' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:43:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:43:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:43:24.232 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:43:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:43:24.232 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:43:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:23 smithi146 ceph-mon[103278]: from='client.94133 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:23 smithi146 ceph-mon[103278]: from='client.94139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3173311256' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:43:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:24 smithi146 ceph-mon[103278]: from='client.94151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:25.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:24 smithi146 ceph-mon[103278]: pgmap v13301: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:25.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1790922426' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:43:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:24 smithi078 ceph-mon[91136]: from='client.94151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:24 smithi078 ceph-mon[91136]: pgmap v13301: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:25.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1790922426' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:43:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:24 smithi078 ceph-mon[99959]: from='client.94151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:24 smithi078 ceph-mon[99959]: pgmap v13301: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:25.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1790922426' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:43:27.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:27 smithi078 ceph-mon[91136]: pgmap v13302: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:27.164 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:27 smithi078 ceph-mon[99959]: pgmap v13302: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:27.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:27 smithi146 ceph-mon[103278]: pgmap v13302: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:29 smithi078 ceph-mon[99959]: pgmap v13303: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:29 smithi078 ceph-mon[91136]: pgmap v13303: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:29 smithi146 ceph-mon[103278]: pgmap v13303: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:31.020 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:43:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:43:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:43:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:31 smithi078 ceph-mon[91136]: pgmap v13304: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:31 smithi078 ceph-mon[99959]: pgmap v13304: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:31 smithi146 ceph-mon[103278]: pgmap v13304: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:32.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:32.041Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:32.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:33 smithi078 ceph-mon[91136]: pgmap v13305: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:33 smithi078 ceph-mon[99959]: pgmap v13305: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:33.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:33 smithi146 ceph-mon[103278]: pgmap v13305: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:35.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:35 smithi078 ceph-mon[91136]: pgmap v13306: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:35.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:35 smithi078 ceph-mon[99959]: pgmap v13306: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:35 smithi146 ceph-mon[103278]: pgmap v13306: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:37.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:37 smithi146 ceph-mon[103278]: pgmap v13307: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:37.285 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:37 smithi078 ceph-mon[91136]: pgmap v13307: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:37.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:37 smithi078 ceph-mon[99959]: pgmap v13307: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:38.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:39 smithi146 ceph-mon[103278]: pgmap v13308: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:39 smithi078 ceph-mon[91136]: pgmap v13308: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:39 smithi078 ceph-mon[99959]: pgmap v13308: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:43:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:43:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:43:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:41 smithi146 ceph-mon[103278]: pgmap v13309: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:41 smithi078 ceph-mon[91136]: pgmap v13309: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:41 smithi078 ceph-mon[99959]: pgmap v13309: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:42.041Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:42.042Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:42.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:43.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:42 smithi146 ceph-mon[103278]: pgmap v13310: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:42 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[91136]: pgmap v13310: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[99959]: pgmap v13310: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:43.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:42 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:44 smithi146 ceph-mon[103278]: pgmap v13311: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:44 smithi078 ceph-mon[91136]: pgmap v13311: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:44 smithi078 ceph-mon[99959]: pgmap v13311: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:47.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:46 smithi146 ceph-mon[103278]: pgmap v13312: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:47.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:46 smithi078 ceph-mon[91136]: pgmap v13312: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:47.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:47 smithi078 ceph-mon[99959]: pgmap v13312: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:49.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:49 smithi146 ceph-mon[103278]: pgmap v13313: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:49.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:49 smithi078 ceph-mon[91136]: pgmap v13313: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:49.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:49 smithi078 ceph-mon[99959]: pgmap v13313: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:51.006 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:43:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:43:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:43:51.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:51 smithi146 ceph-mon[103278]: pgmap v13314: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:51.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:51 smithi078 ceph-mon[91136]: pgmap v13314: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:51.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:51 smithi078 ceph-mon[99959]: pgmap v13314: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:52.042Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:52.043Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:43:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:43:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:43:52.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:43:53.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:53 smithi146 ceph-mon[103278]: pgmap v13315: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:53.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:53 smithi078 ceph-mon[91136]: pgmap v13315: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:53.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:53 smithi078 ceph-mon[99959]: pgmap v13315: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:53.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:43:54.528 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:43:55.054 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:43:55.054 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 7h - - 2024-09-21T03:43:55.054 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 7h - - 2024-09-21T03:43:55.054 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 17s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:43:55.054 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 13s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 17s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 13s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 17s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 17s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 13s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 17s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 17s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:43:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 13s ago 7h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 17s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 17s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 17s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 17s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 13s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 13s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:43:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 13s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:43:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 13s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:43:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 13s ago 7h 73.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:43:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 17s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:43:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:43:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 17s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:43:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:43:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:55 smithi146 ceph-mon[103278]: pgmap v13316: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:55 smithi146 ceph-mon[103278]: from='client.74247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:55.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:55 smithi146 ceph-mon[103278]: from='client.94169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:55 smithi078 ceph-mon[91136]: pgmap v13316: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:55 smithi078 ceph-mon[91136]: from='client.74247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:55.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:55 smithi078 ceph-mon[91136]: from='client.94169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:55.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:55 smithi078 ceph-mon[99959]: pgmap v13316: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:55 smithi078 ceph-mon[99959]: from='client.74247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:55.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:55 smithi078 ceph-mon[99959]: from='client.94169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:43:55.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:43:55.369 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:43:55.635 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:43:55.635 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:43:55.635 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:43:55.635 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:43:55.636 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:43:55.945 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:43:55.945 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:43:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:43:55.945 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:43:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:43:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[99959]: from='client.94175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2125050557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:43:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[99959]: from='client.94187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1854232324' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:43:56.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[91136]: from='client.94175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:56.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2125050557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:43:56.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[91136]: from='client.94187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:56.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:56 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1854232324' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:43:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:56 smithi146 ceph-mon[103278]: from='client.94175 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2125050557' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:43:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:56 smithi146 ceph-mon[103278]: from='client.94187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:43:56.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:56 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1854232324' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:43:57.271 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:57 smithi146 ceph-mon[103278]: pgmap v13317: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:57.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:57 smithi078 ceph-mon[91136]: pgmap v13317: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:57.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:57 smithi078 ceph-mon[99959]: pgmap v13317: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:43:58.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.535 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:58.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:43:59.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:59 smithi146 ceph-mon[103278]: pgmap v13318: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:59.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:43:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:59 smithi078 ceph-mon[91136]: pgmap v13318: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:43:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:43:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:59 smithi078 ceph-mon[99959]: pgmap v13318: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:43:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:43:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:44:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:44:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:44:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:01 smithi146 ceph-mon[103278]: pgmap v13319: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:01 smithi078 ceph-mon[91136]: pgmap v13319: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:01 smithi078 ceph-mon[99959]: pgmap v13319: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:02.043Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:02.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:02.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:03 smithi146 ceph-mon[103278]: pgmap v13320: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[91136]: pgmap v13320: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[99959]: pgmap v13320: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:05 smithi146 ceph-mon[103278]: pgmap v13321: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:05 smithi078 ceph-mon[91136]: pgmap v13321: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:05 smithi078 ceph-mon[99959]: pgmap v13321: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:07 smithi146 ceph-mon[103278]: pgmap v13322: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:07.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:07 smithi078 ceph-mon[91136]: pgmap v13322: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:07 smithi078 ceph-mon[99959]: pgmap v13322: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:09 smithi078 ceph-mon[91136]: pgmap v13323: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:09 smithi078 ceph-mon[99959]: pgmap v13323: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:09 smithi146 ceph-mon[103278]: pgmap v13323: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:10.839 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:44:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:44:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:44:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:11 smithi146 ceph-mon[103278]: pgmap v13324: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:11.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:11 smithi078 ceph-mon[91136]: pgmap v13324: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:11 smithi078 ceph-mon[99959]: pgmap v13324: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:12.044Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:12.044Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:12.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:13 smithi146 ceph-mon[103278]: pgmap v13325: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:13 smithi078 ceph-mon[91136]: pgmap v13325: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:13 smithi078 ceph-mon[99959]: pgmap v13325: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:15 smithi146 ceph-mon[103278]: pgmap v13326: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:15.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:15 smithi078 ceph-mon[91136]: pgmap v13326: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:15 smithi078 ceph-mon[99959]: pgmap v13326: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:17 smithi146 ceph-mon[103278]: pgmap v13327: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:17 smithi078 ceph-mon[91136]: pgmap v13327: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:17 smithi078 ceph-mon[99959]: pgmap v13327: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:18.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.412 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:18.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:19 smithi146 ceph-mon[103278]: pgmap v13328: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:19 smithi078 ceph-mon[91136]: pgmap v13328: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:19 smithi078 ceph-mon[99959]: pgmap v13328: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:44:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:44:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:44:21.747 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:21 smithi146 ceph-mon[103278]: pgmap v13329: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:21 smithi078 ceph-mon[91136]: pgmap v13329: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:21 smithi078 ceph-mon[99959]: pgmap v13329: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:22.044Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:22.045Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:22.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:23 smithi146 ceph-mon[103278]: pgmap v13330: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[91136]: pgmap v13330: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[99959]: pgmap v13330: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:25 smithi146 ceph-mon[103278]: pgmap v13331: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:25 smithi078 ceph-mon[91136]: pgmap v13331: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:25 smithi078 ceph-mon[99959]: pgmap v13331: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:26.240 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:44:26.266 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 03:44:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T03:44:25.847651609Z level=info msg="Completed cleanup jobs" duration=12.220067ms 2024-09-21T03:44:26.765 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:44:26.765 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 9s ago 7h - - 2024-09-21T03:44:26.765 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 7h - - 2024-09-21T03:44:26.765 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 9s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:44:26.765 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 4s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 9s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 4s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 9s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 9s ago 7h 563M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 4s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 9s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 9s ago 7h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 4s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 9s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 9s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 9s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 9s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 4s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:44:26.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 4s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 4s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 4s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 4s ago 7h 70.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 9s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 9s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:44:26.767 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:44:27.074 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:44:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:44:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:44:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:44:27.075 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:44:27.350 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:44:27.351 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:44:27.351 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[91136]: pgmap v13332: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[91136]: from='client.94199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[91136]: from='client.74289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[91136]: from='client.74295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/176131749' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[99959]: pgmap v13332: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:27.351 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[99959]: from='client.94199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.352 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[99959]: from='client.74289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.352 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[99959]: from='client.74295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.352 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/176131749' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:27 smithi146 ceph-mon[103278]: pgmap v13332: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:27 smithi146 ceph-mon[103278]: from='client.94199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:27 smithi146 ceph-mon[103278]: from='client.74289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:27 smithi146 ceph-mon[103278]: from='client.74295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/176131749' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:44:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:44:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:44:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:44:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:44:27.661 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:44:28.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:28 smithi146 ceph-mon[103278]: from='client.74307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:28.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:28 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3044328904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:44:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:28 smithi078 ceph-mon[91136]: from='client.74307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:28.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:28 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3044328904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:44:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:28 smithi078 ceph-mon[99959]: from='client.74307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:28.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:28 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3044328904' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:44:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:29 smithi146 ceph-mon[103278]: pgmap v13333: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:29 smithi078 ceph-mon[91136]: pgmap v13333: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:29 smithi078 ceph-mon[99959]: pgmap v13333: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:30.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:44:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:44:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:44:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:31 smithi146 ceph-mon[103278]: pgmap v13334: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:31.523 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:31 smithi078 ceph-mon[99959]: pgmap v13334: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:31.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:31 smithi078 ceph-mon[91136]: pgmap v13334: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:32.170 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:32.045Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:32.170 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:32.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:32.170 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:32.046Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:33 smithi146 ceph-mon[103278]: pgmap v13335: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:33 smithi078 ceph-mon[91136]: pgmap v13335: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:33 smithi078 ceph-mon[99959]: pgmap v13335: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:35 smithi078 ceph-mon[91136]: pgmap v13336: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:35 smithi078 ceph-mon[99959]: pgmap v13336: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:35 smithi146 ceph-mon[103278]: pgmap v13336: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:37 smithi078 ceph-mon[91136]: pgmap v13337: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:37 smithi078 ceph-mon[99959]: pgmap v13337: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:37 smithi146 ceph-mon[103278]: pgmap v13337: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:38.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.753 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:39.691 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:39 smithi146 ceph-mon[103278]: pgmap v13338: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:39 smithi078 ceph-mon[91136]: pgmap v13338: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:39 smithi078 ceph-mon[99959]: pgmap v13338: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:44:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:44:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:44:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:41 smithi146 ceph-mon[103278]: pgmap v13339: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:41 smithi078 ceph-mon[99959]: pgmap v13339: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:41 smithi078 ceph-mon[91136]: pgmap v13339: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:42.046Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:42.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:42.047Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:43 smithi146 ceph-mon[103278]: pgmap v13340: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[99959]: pgmap v13340: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[91136]: pgmap v13340: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:43.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:44.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:44.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:45 smithi146 ceph-mon[103278]: pgmap v13341: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:45 smithi078 ceph-mon[91136]: pgmap v13341: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:45 smithi078 ceph-mon[99959]: pgmap v13341: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:47 smithi146 ceph-mon[103278]: pgmap v13342: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:47 smithi078 ceph-mon[91136]: pgmap v13342: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:47 smithi078 ceph-mon[99959]: pgmap v13342: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:49.652 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:49 smithi078 ceph-mon[91136]: pgmap v13343: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:49.652 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:49 smithi078 ceph-mon[99959]: pgmap v13343: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:49 smithi146 ceph-mon[103278]: pgmap v13343: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:50.882 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:44:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:44:50] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:44:51.484 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:51 smithi078 ceph-mon[99959]: pgmap v13344: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:51.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:51 smithi078 ceph-mon[91136]: pgmap v13344: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:51 smithi146 ceph-mon[103278]: pgmap v13344: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:52.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:52.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:52.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:52.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:44:52.191 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:44:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:44:52.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:44:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:53 smithi078 ceph-mon[91136]: pgmap v13345: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:53 smithi078 ceph-mon[99959]: pgmap v13345: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:53 smithi146 ceph-mon[103278]: pgmap v13345: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:44:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:55 smithi146 ceph-mon[103278]: pgmap v13346: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:55 smithi078 ceph-mon[91136]: pgmap v13346: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:55 smithi078 ceph-mon[99959]: pgmap v13346: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:57 smithi146 ceph-mon[103278]: pgmap v13347: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:57.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:57 smithi078 ceph-mon[91136]: pgmap v13347: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:57.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:57 smithi078 ceph-mon[99959]: pgmap v13347: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:44:57.944 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:44:58.469 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:44:58.473 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 7h - - 2024-09-21T03:44:58.473 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 7h - - 2024-09-21T03:44:58.473 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 0s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:44:58.473 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 16s ago 7h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 0s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 16s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 0s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 0s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 16s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 0s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 0s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 16s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 0s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 0s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:44:58.474 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 0s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 0s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 16s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 16s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 16s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 16s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 16s ago 7h 72.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 0s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 0s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:44:58.475 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:44:58.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:44:58.779 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:44:58.779 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.779 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.780 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:58.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:44:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:44:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:44:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:44:59.360 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:44:59.361 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:44:59.361 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:44:59.361 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:44:59.361 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:44:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: from='client.74319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: pgmap v13348: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: from='client.74325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: from='client.74331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1315884002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:44:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:44:59 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2410545377' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:44:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: from='client.74319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: pgmap v13348: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: from='client.74325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: from='client.74331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1315884002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2410545377' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: from='client.74319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: pgmap v13348: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: from='client.74325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: from='client.74331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1315884002' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:44:59.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:44:59 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2410545377' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:45:00.945 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:00 smithi146 ceph-mon[103278]: from='client.74343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:01.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:00 smithi078 ceph-mon[91136]: from='client.74343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:01.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:00 smithi078 ceph-mon[99959]: from='client.74343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:01.037 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:45:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:45:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:45:01.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:01 smithi146 ceph-mon[103278]: pgmap v13349: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:01 smithi078 ceph-mon[91136]: pgmap v13349: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:01 smithi078 ceph-mon[99959]: pgmap v13349: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:02.047Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:02.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:02.048Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:03.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:03 smithi146 ceph-mon[103278]: pgmap v13350: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[91136]: pgmap v13350: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[99959]: pgmap v13350: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:03.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:05 smithi146 ceph-mon[103278]: pgmap v13351: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:05 smithi078 ceph-mon[91136]: pgmap v13351: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:05 smithi078 ceph-mon[99959]: pgmap v13351: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:07 smithi146 ceph-mon[103278]: pgmap v13352: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:07 smithi078 ceph-mon[91136]: pgmap v13352: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:07 smithi078 ceph-mon[99959]: pgmap v13352: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:09 smithi146 ceph-mon[103278]: pgmap v13353: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:09 smithi078 ceph-mon[91136]: pgmap v13353: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:09 smithi078 ceph-mon[99959]: pgmap v13353: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:10.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:45:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:45:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:45:11.750 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:11 smithi078 ceph-mon[99959]: pgmap v13354: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:11.750 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:11 smithi078 ceph-mon[91136]: pgmap v13354: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:11 smithi146 ceph-mon[103278]: pgmap v13354: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:12.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:12.048Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:12.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:12.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:12.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:12.049Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:13 smithi078 ceph-mon[91136]: pgmap v13355: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:13 smithi078 ceph-mon[99959]: pgmap v13355: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:13 smithi146 ceph-mon[103278]: pgmap v13355: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:14.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:15.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:15 smithi078 ceph-mon[91136]: pgmap v13356: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:15 smithi078 ceph-mon[99959]: pgmap v13356: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:15 smithi146 ceph-mon[103278]: pgmap v13356: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:17 smithi078 ceph-mon[91136]: pgmap v13357: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:17 smithi078 ceph-mon[99959]: pgmap v13357: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:17.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:17 smithi146 ceph-mon[103278]: pgmap v13357: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:19.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.001 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:18 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:18 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:19.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:18 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:19 smithi146 ceph-mon[103278]: pgmap v13358: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:19 smithi078 ceph-mon[91136]: pgmap v13358: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:19 smithi078 ceph-mon[99959]: pgmap v13358: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:45:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:45:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:45:22.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:21 smithi146 ceph-mon[103278]: pgmap v13359: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:21 smithi078 ceph-mon[99959]: pgmap v13359: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:21 smithi078 ceph-mon[91136]: pgmap v13359: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:22.049Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:22.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:22.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:24.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:23 smithi146 ceph-mon[103278]: pgmap v13360: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[91136]: pgmap v13360: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[99959]: pgmap v13360: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:26.018 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:25 smithi146 ceph-mon[103278]: pgmap v13361: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:25 smithi078 ceph-mon[91136]: pgmap v13361: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:25 smithi078 ceph-mon[99959]: pgmap v13361: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:27 smithi146 ceph-mon[103278]: pgmap v13362: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:27 smithi078 ceph-mon[91136]: pgmap v13362: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:27 smithi078 ceph-mon[99959]: pgmap v13362: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:29.654 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:45:30.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:29 smithi146 ceph-mon[103278]: pgmap v13363: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:30.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:29 smithi078 ceph-mon[99959]: pgmap v13363: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:29 smithi078 ceph-mon[91136]: pgmap v13363: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 7h - - 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 7h - - 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 12s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 7s ago 7h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 12s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 7s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 12s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:45:30.180 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 12s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 7s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 12s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 12s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 7s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 12s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 12s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 12s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 12s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 7s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 7s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 7s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:45:30.181 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 7s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:45:30.182 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 7s ago 7h 72.9M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:45:30.182 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 12s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:45:30.182 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:45:30.182 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 12s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:45:30.182 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:45:30.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:45:30.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:45:30.492 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:45:30.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:45:30.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:45:30.492 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:45:30.761 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:45:30.761 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:45:30.761 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:45:30.761 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:45:30.761 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:45:30.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:45:30.762 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:45:30.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:30 smithi078 ceph-mon[99959]: from='client.74355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:30.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2257829109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:45:30.763 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:45:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:45:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:45:30.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:30 smithi078 ceph-mon[91136]: from='client.74355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:30.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2257829109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:45:31.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:30 smithi146 ceph-mon[103278]: from='client.74355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:31.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2257829109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:45:31.076 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:45:31.076 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:45:31.076 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:45:31.076 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:45:31.076 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:45:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:31 smithi146 ceph-mon[103278]: from='client.74361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:31 smithi146 ceph-mon[103278]: pgmap v13364: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:31 smithi146 ceph-mon[103278]: from='client.74367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:31 smithi146 ceph-mon[103278]: from='client.74379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2366794149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[99959]: from='client.74361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[99959]: pgmap v13364: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[99959]: from='client.74367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[99959]: from='client.74379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2366794149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[91136]: from='client.74361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[91136]: pgmap v13364: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[91136]: from='client.74367 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[91136]: from='client.74379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:45:32.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2366794149' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:45:32.385 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:32.050Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:32.385 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:32.050Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:32.385 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:32.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:33 smithi146 ceph-mon[103278]: pgmap v13365: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:33 smithi078 ceph-mon[91136]: pgmap v13365: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:34.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:33 smithi078 ceph-mon[99959]: pgmap v13365: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:34.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:35 smithi146 ceph-mon[103278]: pgmap v13366: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:36.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:35 smithi078 ceph-mon[91136]: pgmap v13366: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:36.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:35 smithi078 ceph-mon[99959]: pgmap v13366: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:37 smithi146 ceph-mon[103278]: pgmap v13367: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:38.035 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:37 smithi078 ceph-mon[91136]: pgmap v13367: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:37 smithi078 ceph-mon[99959]: pgmap v13367: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:38 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:38 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:38 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:40.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:39 smithi146 ceph-mon[103278]: pgmap v13368: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:39 smithi078 ceph-mon[91136]: pgmap v13368: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:39 smithi078 ceph-mon[99959]: pgmap v13368: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:40.873 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:45:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:45:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:45:41.211 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:40 smithi146 ceph-mon[103278]: pgmap v13369: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:41.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:40 smithi078 ceph-mon[91136]: pgmap v13369: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:41.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:40 smithi078 ceph-mon[99959]: pgmap v13369: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:42.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:42.051Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:42.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:43 smithi146 ceph-mon[103278]: pgmap v13370: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[91136]: pgmap v13370: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[99959]: pgmap v13370: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:45 smithi146 ceph-mon[103278]: pgmap v13371: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:45 smithi078 ceph-mon[91136]: pgmap v13371: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:45 smithi078 ceph-mon[99959]: pgmap v13371: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:47 smithi146 ceph-mon[103278]: pgmap v13372: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:47 smithi078 ceph-mon[91136]: pgmap v13372: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:47 smithi078 ceph-mon[99959]: pgmap v13372: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:49 smithi146 ceph-mon[103278]: pgmap v13373: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:49 smithi078 ceph-mon[91136]: pgmap v13373: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:49 smithi078 ceph-mon[99959]: pgmap v13373: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:45:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:45:50] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:45:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:51 smithi146 ceph-mon[103278]: pgmap v13374: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:52.035 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:51 smithi078 ceph-mon[99959]: pgmap v13374: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:51 smithi078 ceph-mon[91136]: pgmap v13374: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:52.305 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:52.051Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:52.305 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:52.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:45:52.305 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:45:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:45:52.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:45:53.983 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:53 smithi078 ceph-mon[91136]: pgmap v13375: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:53.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:53.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:53 smithi078 ceph-mon[99959]: pgmap v13375: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:53.984 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:53 smithi146 ceph-mon[103278]: pgmap v13375: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:54.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:45:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:55 smithi146 ceph-mon[103278]: pgmap v13376: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:55 smithi078 ceph-mon[91136]: pgmap v13376: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:55 smithi078 ceph-mon[99959]: pgmap v13376: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:57 smithi146 ceph-mon[103278]: pgmap v13377: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:57 smithi078 ceph-mon[99959]: pgmap v13377: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:57 smithi078 ceph-mon[91136]: pgmap v13377: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:45:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:58 smithi146 ceph-mon[103278]: pgmap v13378: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:45:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[99959]: pgmap v13378: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[91136]: pgmap v13378: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:45:59.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:45:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:46:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:46:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:46:01.376 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:01 smithi078 ceph-mon[99959]: pgmap v13379: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:01.376 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:01 smithi078 ceph-mon[91136]: pgmap v13379: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:01.376 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:46:01.420 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:01 smithi146 ceph-mon[103278]: pgmap v13379: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:01.903 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:46:01.903 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 7h - - 2024-09-21T03:46:01.903 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 7h - - 2024-09-21T03:46:01.903 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 3s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 19s ago 7h 91.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 3s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 19s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 3s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 3s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 19s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 3s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 3s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 19s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 3s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 3s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 3s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 3s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:46:01.904 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 19s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 19s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 19s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 19s ago 7h 73.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 3s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 3s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:46:01.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:46:02.215 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:46:02.216 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:46:02.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-mon[99959]: from='client.94307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:02.217 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-mon[99959]: from='client.94313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:02.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-mon[91136]: from='client.94307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:02.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-mon[91136]: from='client.94313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:02.217 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:02.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:02.217 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:02.052Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:02.217 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:02.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:02.483 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:46:02.484 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:46:02.510 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:02 smithi146 ceph-mon[103278]: from='client.94307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:02.511 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:02 smithi146 ceph-mon[103278]: from='client.94313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:02.798 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:46:02.798 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:46:02.798 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:46:02.798 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:46:02.798 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:46:03.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='client.74403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: pgmap v13380: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2870153619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='client.74415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2344516280' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='client.74403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:03.326 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: pgmap v13380: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2870153619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='client.74415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2344516280' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.327 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='client.74403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: pgmap v13380: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2870153619' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='client.74415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2344516280' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:05.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:05 smithi078 ceph-mon[91136]: pgmap v13381: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:05.358 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:05 smithi078 ceph-mon[99959]: pgmap v13381: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:05.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:05 smithi146 ceph-mon[103278]: pgmap v13381: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:07.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:07 smithi146 ceph-mon[103278]: pgmap v13382: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:07 smithi078 ceph-mon[91136]: pgmap v13382: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:07 smithi078 ceph-mon[99959]: pgmap v13382: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:09 smithi146 ceph-mon[103278]: pgmap v13383: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:09 smithi078 ceph-mon[91136]: pgmap v13383: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:09 smithi078 ceph-mon[99959]: pgmap v13383: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:46:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:46:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:46:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:11 smithi146 ceph-mon[103278]: pgmap v13384: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:11 smithi078 ceph-mon[91136]: pgmap v13384: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:11 smithi078 ceph-mon[99959]: pgmap v13384: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:12.423 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:12.052Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:12.423 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:12.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:12.423 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:12.053Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:13.499 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:13 smithi078 ceph-mon[99959]: pgmap v13385: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:13.500 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:13.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:13 smithi078 ceph-mon[91136]: pgmap v13385: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:13.500 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:13 smithi146 ceph-mon[103278]: pgmap v13385: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:13.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:15.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:15 smithi146 ceph-mon[103278]: pgmap v13386: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:15 smithi078 ceph-mon[91136]: pgmap v13386: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:15 smithi078 ceph-mon[99959]: pgmap v13386: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:17 smithi146 ceph-mon[103278]: pgmap v13387: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:17.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:17 smithi078 ceph-mon[91136]: pgmap v13387: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:17 smithi078 ceph-mon[99959]: pgmap v13387: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:19 smithi146 ceph-mon[103278]: pgmap v13388: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[91136]: pgmap v13388: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[99959]: pgmap v13388: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:19.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:46:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:46:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:46:21.246 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:21 smithi146 ceph-mon[103278]: pgmap v13389: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:21 smithi078 ceph-mon[91136]: pgmap v13389: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:21 smithi078 ceph-mon[99959]: pgmap v13389: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:22.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:22.053Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:22.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:22.054Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:22.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:22.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:23.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:23 smithi146 ceph-mon[103278]: pgmap v13390: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[99959]: pgmap v13390: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[91136]: pgmap v13390: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:23.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:25 smithi146 ceph-mon[103278]: pgmap v13391: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:25 smithi078 ceph-mon[91136]: pgmap v13391: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:25 smithi078 ceph-mon[99959]: pgmap v13391: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:27.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:27 smithi146 ceph-mon[103278]: pgmap v13392: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:27.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:27 smithi078 ceph-mon[91136]: pgmap v13392: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:27 smithi078 ceph-mon[99959]: pgmap v13392: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:29 smithi146 ceph-mon[103278]: pgmap v13393: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:29 smithi078 ceph-mon[99959]: pgmap v13393: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:29 smithi078 ceph-mon[91136]: pgmap v13393: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:46:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:46:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:46:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:31 smithi146 ceph-mon[103278]: pgmap v13394: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:31 smithi078 ceph-mon[91136]: pgmap v13394: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:31 smithi078 ceph-mon[99959]: pgmap v13394: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:32.054Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:32.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:32.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:33.107 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:46:33.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:33 smithi078 ceph-mon[99959]: pgmap v13395: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:33.409 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:33.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:33 smithi078 ceph-mon[91136]: pgmap v13395: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:33.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:33 smithi146 ceph-mon[103278]: pgmap v13395: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 7h - - 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 7h - - 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 15s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 10s ago 7h 90.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 15s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 10s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 15s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:46:33.651 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 15s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 10s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 15s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 15s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 10s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 15s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 15s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 15s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 15s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 10s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 10s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 10s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:46:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 10s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:46:33.653 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 10s ago 7h 73.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:46:33.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 15s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:46:33.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 10s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:46:33.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 15s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:46:33.653 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:46:33.969 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:46:33.970 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:46:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:46:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:46:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:46:33.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:46:33.971 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:46:34.245 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:46:34.245 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:46:34.245 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:46:34.245 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:46:34.246 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:46:34.246 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[99959]: from='client.74427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[99959]: from='client.94349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[99959]: from='client.94355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/874429092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[91136]: from='client.74427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[91136]: from='client.94349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[91136]: from='client.94355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/874429092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:46:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:34 smithi146 ceph-mon[103278]: from='client.74427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:34 smithi146 ceph-mon[103278]: from='client.94349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:34 smithi146 ceph-mon[103278]: from='client.94355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:34.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/874429092' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:46:34.564 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:46:34.564 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:46:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:46:34.564 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:46:34.564 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:46:35.320 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:35 smithi078 ceph-mon[91136]: pgmap v13396: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:35.320 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:35 smithi078 ceph-mon[91136]: from='client.94367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:35.320 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:35 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/863563913' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:46:35.320 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:35 smithi078 ceph-mon[99959]: pgmap v13396: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:35.321 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:35 smithi078 ceph-mon[99959]: from='client.94367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:35.321 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:35 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/863563913' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:46:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:35 smithi146 ceph-mon[103278]: pgmap v13396: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:35 smithi146 ceph-mon[103278]: from='client.94367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:46:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:35 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/863563913' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:46:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:37 smithi146 ceph-mon[103278]: pgmap v13397: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:37 smithi078 ceph-mon[91136]: pgmap v13397: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:37 smithi078 ceph-mon[99959]: pgmap v13397: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:39 smithi146 ceph-mon[103278]: pgmap v13398: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[91136]: pgmap v13398: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[99959]: pgmap v13398: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:39.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:46:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:46:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:46:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:41 smithi146 ceph-mon[103278]: pgmap v13399: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:41 smithi078 ceph-mon[99959]: pgmap v13399: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:41 smithi078 ceph-mon[91136]: pgmap v13399: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:42.055Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:42.055Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:42.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:43.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:43 smithi146 ceph-mon[103278]: pgmap v13400: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:43.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:43.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.470 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[99959]: pgmap v13400: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[91136]: pgmap v13400: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:43.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:43.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:44.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:45 smithi146 ceph-mon[103278]: pgmap v13401: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:45 smithi078 ceph-mon[91136]: pgmap v13401: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:45 smithi078 ceph-mon[99959]: pgmap v13401: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:47 smithi146 ceph-mon[103278]: pgmap v13402: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:47 smithi078 ceph-mon[99959]: pgmap v13402: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:47 smithi078 ceph-mon[91136]: pgmap v13402: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:49 smithi146 ceph-mon[103278]: pgmap v13403: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:49 smithi078 ceph-mon[99959]: pgmap v13403: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:49 smithi078 ceph-mon[91136]: pgmap v13403: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:46:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:46:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:46:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:51 smithi146 ceph-mon[103278]: pgmap v13404: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:51 smithi078 ceph-mon[91136]: pgmap v13404: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:51 smithi078 ceph-mon[99959]: pgmap v13404: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:52.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:52.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:52.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:52.056Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:46:52.438 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:46:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:46:52.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:46:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:53 smithi146 ceph-mon[103278]: pgmap v13405: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:53 smithi078 ceph-mon[91136]: pgmap v13405: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:53 smithi078 ceph-mon[99959]: pgmap v13405: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:46:55.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:55 smithi078 ceph-mon[91136]: pgmap v13406: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:55.398 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:55 smithi078 ceph-mon[99959]: pgmap v13406: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:55 smithi146 ceph-mon[103278]: pgmap v13406: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:57 smithi078 ceph-mon[91136]: pgmap v13407: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:57 smithi078 ceph-mon[99959]: pgmap v13407: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:57 smithi146 ceph-mon[103278]: pgmap v13407: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:46:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:59 smithi146 ceph-mon[103278]: pgmap v13408: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:46:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[99959]: pgmap v13408: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[91136]: pgmap v13408: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:46:59.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:46:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:47:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:47:00] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:47:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:01 smithi146 ceph-mon[103278]: pgmap v13409: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:01 smithi078 ceph-mon[91136]: pgmap v13409: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:01 smithi078 ceph-mon[99959]: pgmap v13409: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:02.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:02.056Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:02.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:02.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:02.312 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:02.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[99959]: pgmap v13410: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[91136]: pgmap v13410: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:03 smithi146 ceph-mon[103278]: pgmap v13410: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:04.862 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:47:05.392 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:47:05.392 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 7h - - 2024-09-21T03:47:05.392 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 7h - - 2024-09-21T03:47:05.392 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 7s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 2s ago 7h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 7s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 2s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 7s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 7s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 2s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 7s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 7s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 2s ago 7h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 7s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 7s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 7s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 7s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:47:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 2s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 2s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 2s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 2s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 2s ago 7h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 7s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 2s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 7s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:47:05.394 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:47:05.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:05 smithi078 ceph-mon[99959]: pgmap v13411: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:05.704 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:05 smithi078 ceph-mon[99959]: from='client.74463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:05.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:05 smithi078 ceph-mon[91136]: pgmap v13411: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:05.705 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:05 smithi078 ceph-mon[91136]: from='client.74463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:47:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:47:05.706 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:47:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:05 smithi146 ceph-mon[103278]: pgmap v13411: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:05 smithi146 ceph-mon[103278]: from='client.74463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:47:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:47:05.974 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:47:05.975 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:47:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:47:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:47:06.306 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:47:06.307 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:47:06.307 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:47:06.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:06 smithi146 ceph-mon[103278]: from='client.94385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:06 smithi146 ceph-mon[103278]: from='client.74475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1450232799' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:47:06.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:06 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1027521447' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:47:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[91136]: from='client.94385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[91136]: from='client.74475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1450232799' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:47:06.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1027521447' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:47:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[99959]: from='client.94385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:06.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[99959]: from='client.74475 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1450232799' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:47:06.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:06 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1027521447' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:47:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:07 smithi146 ceph-mon[103278]: from='client.94403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:07 smithi146 ceph-mon[103278]: pgmap v13412: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:07 smithi078 ceph-mon[91136]: from='client.94403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:07 smithi078 ceph-mon[91136]: pgmap v13412: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:07 smithi078 ceph-mon[99959]: from='client.94403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:07 smithi078 ceph-mon[99959]: pgmap v13412: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:09.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:09 smithi146 ceph-mon[103278]: pgmap v13413: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:09 smithi078 ceph-mon[91136]: pgmap v13413: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:09 smithi078 ceph-mon[99959]: pgmap v13413: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:47:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:47:10] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:47:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:11 smithi146 ceph-mon[103278]: pgmap v13414: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:11 smithi078 ceph-mon[91136]: pgmap v13414: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:11 smithi078 ceph-mon[99959]: pgmap v13414: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:12.314 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:12.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:12.314 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:12.057Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:12.314 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:12.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:13 smithi146 ceph-mon[103278]: pgmap v13415: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:13 smithi078 ceph-mon[91136]: pgmap v13415: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:13 smithi078 ceph-mon[99959]: pgmap v13415: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:14.650 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:14.650 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:15.654 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:15 smithi078 ceph-mon[91136]: pgmap v13416: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:15.654 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:15 smithi078 ceph-mon[99959]: pgmap v13416: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:15 smithi146 ceph-mon[103278]: pgmap v13416: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:17 smithi078 ceph-mon[91136]: pgmap v13417: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:17 smithi078 ceph-mon[99959]: pgmap v13417: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:17.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:17 smithi146 ceph-mon[103278]: pgmap v13417: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:19 smithi146 ceph-mon[103278]: pgmap v13418: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[99959]: pgmap v13418: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[91136]: pgmap v13418: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:19.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:47:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:47:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:47:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:21 smithi146 ceph-mon[103278]: pgmap v13419: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:21 smithi078 ceph-mon[91136]: pgmap v13419: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:21.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:21 smithi078 ceph-mon[99959]: pgmap v13419: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:22.325 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:22.057Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:22.325 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:22.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:22.325 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:22.058Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:23.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:23 smithi146 ceph-mon[103278]: pgmap v13420: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:23.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:23.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.770 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[99959]: pgmap v13420: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[91136]: pgmap v13420: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:23.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:25 smithi146 ceph-mon[103278]: pgmap v13421: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:25 smithi078 ceph-mon[99959]: pgmap v13421: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:25 smithi078 ceph-mon[91136]: pgmap v13421: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:27 smithi146 ceph-mon[103278]: pgmap v13422: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:27 smithi078 ceph-mon[91136]: pgmap v13422: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:27 smithi078 ceph-mon[99959]: pgmap v13422: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:29.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:29 smithi146 ceph-mon[103278]: pgmap v13423: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:29 smithi078 ceph-mon[91136]: pgmap v13423: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:29 smithi078 ceph-mon[99959]: pgmap v13423: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:47:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:47:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:47:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:31 smithi078 ceph-mon[99959]: pgmap v13424: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:31 smithi078 ceph-mon[91136]: pgmap v13424: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:32.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:31 smithi146 ceph-mon[103278]: pgmap v13424: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:32.335 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:32.058Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:32.336 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:32.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:32.336 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:32.059Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:33 smithi078 ceph-mon[99959]: pgmap v13425: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:33 smithi078 ceph-mon[91136]: pgmap v13425: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:33 smithi146 ceph-mon[103278]: pgmap v13425: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:35.694 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:35 smithi078 ceph-mon[99959]: pgmap v13426: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:35.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:35 smithi078 ceph-mon[91136]: pgmap v13426: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:36.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:35 smithi146 ceph-mon[103278]: pgmap v13426: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:36.611 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 18s ago 7h - - 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 13s ago 7h - - 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 18s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 13s ago 7h 92.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 18s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 13s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 18s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 18s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 13s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:47:37.161 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 18s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 18s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 13s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 18s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 18s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 18s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 18s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 13s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 13s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 13s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 13s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 13s ago 7h 73.3M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 18s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 13s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:47:37.162 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 18s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:47:37.163 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 13s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:47:37.491 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:47:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:47:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:47:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:47:37.492 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:47:37.768 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:47:37.769 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:47:37.769 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:47:37.769 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:47:37.769 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:47:37.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[99959]: pgmap v13427: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:47:37.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[99959]: from='client.94415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:37.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[99959]: from='client.74505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:37.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2011200363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:47:37.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[91136]: pgmap v13427: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:47:37.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[91136]: from='client.94415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:37.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[91136]: from='client.74505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:37.770 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2011200363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:47:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:37 smithi146 ceph-mon[103278]: pgmap v13427: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:47:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:37 smithi146 ceph-mon[103278]: from='client.94415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:37 smithi146 ceph-mon[103278]: from='client.74505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:38.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2011200363' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:47:38.084 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:47:38.084 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:47:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:47:38.085 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:47:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:47:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:38 smithi078 ceph-mon[91136]: from='client.94427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:38 smithi078 ceph-mon[91136]: from='client.94439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:38.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:38 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4271418576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:47:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:38 smithi078 ceph-mon[99959]: from='client.94427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:38.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:38 smithi078 ceph-mon[99959]: from='client.94439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:38.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:38 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4271418576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:47:39.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:38 smithi146 ceph-mon[103278]: from='client.94427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:38 smithi146 ceph-mon[103278]: from='client.94439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:47:39.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:38 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4271418576' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:47:40.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:39 smithi146 ceph-mon[103278]: pgmap v13428: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[91136]: pgmap v13428: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[99959]: pgmap v13428: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:47:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:47:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:47:42.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:41 smithi146 ceph-mon[103278]: pgmap v13429: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:41 smithi078 ceph-mon[91136]: pgmap v13429: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:41 smithi078 ceph-mon[99959]: pgmap v13429: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:42.346 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:42.059Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:42.347 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:42.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:42.347 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:42.060Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:43.987 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:43 smithi146 ceph-mon[103278]: pgmap v13430: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:43.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:43.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:43.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:43.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[91136]: pgmap v13430: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[99959]: pgmap v13430: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:44.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:47:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:47:45.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:45 smithi146 ceph-mon[103278]: pgmap v13431: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:45 smithi078 ceph-mon[91136]: pgmap v13431: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:45 smithi078 ceph-mon[99959]: pgmap v13431: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:47 smithi146 ceph-mon[103278]: pgmap v13432: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:47 smithi078 ceph-mon[91136]: pgmap v13432: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:47 smithi078 ceph-mon[99959]: pgmap v13432: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:50.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:49 smithi146 ceph-mon[103278]: pgmap v13433: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:49 smithi078 ceph-mon[91136]: pgmap v13433: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:49 smithi078 ceph-mon[99959]: pgmap v13433: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:47:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:47:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:47:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:51 smithi146 ceph-mon[103278]: pgmap v13434: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:51 smithi078 ceph-mon[91136]: pgmap v13434: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:51 smithi078 ceph-mon[99959]: pgmap v13434: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:52.357 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:52.060Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:52.357 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:52.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:47:52.357 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:47:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:47:52.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:47:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:53 smithi146 ceph-mon[103278]: pgmap v13435: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:53 smithi078 ceph-mon[99959]: pgmap v13435: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:53 smithi078 ceph-mon[91136]: pgmap v13435: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:47:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:55 smithi146 ceph-mon[103278]: pgmap v13436: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:55 smithi078 ceph-mon[91136]: pgmap v13436: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:55 smithi078 ceph-mon[99959]: pgmap v13436: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:47:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:57 smithi078 ceph-mon[91136]: pgmap v13437: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:57 smithi078 ceph-mon[99959]: pgmap v13437: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:47:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:57 smithi146 ceph-mon[103278]: pgmap v13437: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:59 smithi146 ceph-mon[103278]: pgmap v13438: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:47:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[91136]: pgmap v13438: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[99959]: pgmap v13438: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:47:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:48:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:48:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:48:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:01 smithi146 ceph-mon[103278]: pgmap v13439: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:01 smithi078 ceph-mon[99959]: pgmap v13439: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:01 smithi078 ceph-mon[91136]: pgmap v13439: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:02.366 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:02.061Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:02.366 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:02.061Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:02.367 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:02.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:03 smithi146 ceph-mon[103278]: pgmap v13440: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[91136]: pgmap v13440: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[99959]: pgmap v13440: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:05 smithi146 ceph-mon[103278]: pgmap v13441: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:05 smithi078 ceph-mon[91136]: pgmap v13441: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:05 smithi078 ceph-mon[99959]: pgmap v13441: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:07 smithi146 ceph-mon[103278]: pgmap v13442: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:07 smithi078 ceph-mon[91136]: pgmap v13442: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:07 smithi078 ceph-mon[99959]: pgmap v13442: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:08.380 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 7h - - 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 7h - - 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 10s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 5s ago 7h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 10s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 5s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:48:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 10s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 10s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 5s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 10s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 10s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 5s ago 7h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 10s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 10s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 10s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 10s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 5s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 5s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 5s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 5s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 5s ago 7h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:48:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 10s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:48:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 5s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:48:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 10s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:48:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:09.220 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:48:09.221 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:48:09.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:48:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:48:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:48:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:48:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:48:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:48:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:48:09.495 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:48:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[91136]: pgmap v13443: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[91136]: from='client.94451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[91136]: from='client.74541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:09.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/4202925207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:48:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[99959]: pgmap v13443: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:09.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[99959]: from='client.94451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:09.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[99959]: from='client.74541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:09.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/4202925207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:48:09.804 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:48:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:48:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:48:09.805 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:48:09.805 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:48:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:09 smithi146 ceph-mon[103278]: pgmap v13443: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:09 smithi146 ceph-mon[103278]: from='client.94451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:09 smithi146 ceph-mon[103278]: from='client.74541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/4202925207' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:48:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:10 smithi146 ceph-mon[103278]: from='client.94463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:11.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:10 smithi146 ceph-mon[103278]: from='client.94475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:11.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:10 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3776248433' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:48:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:48:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:48:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-mon[91136]: from='client.94463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-mon[91136]: from='client.94475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:11.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3776248433' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:48:11.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-mon[99959]: from='client.94463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-mon[99959]: from='client.94475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:11.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:10 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3776248433' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:48:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:11 smithi146 ceph-mon[103278]: pgmap v13444: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:11 smithi078 ceph-mon[99959]: pgmap v13444: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:11 smithi078 ceph-mon[91136]: pgmap v13444: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:12.377 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:12.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:12.377 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:12.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:12.377 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:12.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:13 smithi146 ceph-mon[103278]: pgmap v13445: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:13 smithi078 ceph-mon[91136]: pgmap v13445: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:13 smithi078 ceph-mon[99959]: pgmap v13445: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:16.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:15 smithi146 ceph-mon[103278]: pgmap v13446: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:15 smithi078 ceph-mon[91136]: pgmap v13446: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:15 smithi078 ceph-mon[99959]: pgmap v13446: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:18.007 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:17 smithi078 ceph-mon[91136]: pgmap v13447: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:18.007 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:17 smithi078 ceph-mon[99959]: pgmap v13447: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:17 smithi146 ceph-mon[103278]: pgmap v13447: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:19 smithi146 ceph-mon[103278]: pgmap v13448: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:19 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[91136]: pgmap v13448: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[99959]: pgmap v13448: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:20.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:20.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:19 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:48:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:48:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:48:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:21 smithi146 ceph-mon[103278]: pgmap v13449: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:21 smithi078 ceph-mon[91136]: pgmap v13449: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:21 smithi078 ceph-mon[99959]: pgmap v13449: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:22.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:22.062Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:22.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:23 smithi146 ceph-mon[103278]: pgmap v13450: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[91136]: pgmap v13450: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[99959]: pgmap v13450: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:26.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:25 smithi146 ceph-mon[103278]: pgmap v13451: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:25 smithi078 ceph-mon[99959]: pgmap v13451: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:25 smithi078 ceph-mon[91136]: pgmap v13451: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:27 smithi146 ceph-mon[103278]: pgmap v13452: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:27 smithi078 ceph-mon[91136]: pgmap v13452: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:27 smithi078 ceph-mon[99959]: pgmap v13452: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:29 smithi146 ceph-mon[103278]: pgmap v13453: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:29 smithi078 ceph-mon[91136]: pgmap v13453: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:29 smithi078 ceph-mon[99959]: pgmap v13453: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:30.901 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:48:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:48:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:48:31.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:30 smithi146 ceph-mon[103278]: pgmap v13454: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:31.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:30 smithi078 ceph-mon[91136]: pgmap v13454: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:31.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:30 smithi078 ceph-mon[99959]: pgmap v13454: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:32.397 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:32.062Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:32.397 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:32.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:32.397 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:32.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:33 smithi078 ceph-mon[91136]: pgmap v13455: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:33.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:33.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:33 smithi078 ceph-mon[99959]: pgmap v13455: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:33.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:33 smithi146 ceph-mon[103278]: pgmap v13455: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:33.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:35.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:35 smithi146 ceph-mon[103278]: pgmap v13456: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:35 smithi078 ceph-mon[91136]: pgmap v13456: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:35 smithi078 ceph-mon[99959]: pgmap v13456: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:37 smithi146 ceph-mon[103278]: pgmap v13457: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:37 smithi078 ceph-mon[91136]: pgmap v13457: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:37.529 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:37 smithi078 ceph-mon[99959]: pgmap v13457: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:39.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[91136]: pgmap v13458: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:39.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[99959]: pgmap v13458: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:39.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.506 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:39 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:39 smithi146 ceph-mon[103278]: pgmap v13458: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:39 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:40.100 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 1s ago 7h - - 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 16s ago 7h - - 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 1s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 16s ago 7h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 1s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 16s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 1s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 1s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 16s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:48:40.638 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 1s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 1s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 16s ago 7h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 1s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 1s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 1s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 1s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 16s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 16s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 16s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 16s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:48:40.639 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 16s ago 7h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:48:40.640 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 1s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:48:40.640 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 16s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:48:40.640 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 1s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:48:40.640 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 16s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:48:40.946 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:48:40.947 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:48:40.947 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:48:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:48:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:48:41.218 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:48:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:48:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:48:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:48:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:48:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:48:41.219 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:48:41.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:41 smithi146 ceph-mon[103278]: pgmap v13459: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:41 smithi146 ceph-mon[103278]: from='client.74571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:41 smithi146 ceph-mon[103278]: from='client.74577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:41 smithi146 ceph-mon[103278]: from='client.94499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1813848618' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:48:41.533 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:48:41.533 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:48:41.534 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:48:41.534 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:48:41.534 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:48:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[99959]: pgmap v13459: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[99959]: from='client.74571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[99959]: from='client.74577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[99959]: from='client.94499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.534 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1813848618' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:48:41.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[91136]: pgmap v13459: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[91136]: from='client.74571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[91136]: from='client.74577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[91136]: from='client.94499 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:41.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1813848618' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:48:42.407 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-mon[99959]: from='client.94511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:42.407 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3716572721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:48:42.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-mon[91136]: from='client.94511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:42.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3716572721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:48:42.408 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:42.063Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:42.408 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:42.063Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:42.408 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:42.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:42 smithi146 ceph-mon[103278]: from='client.94511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:48:42.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:42 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3716572721' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:48:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:43 smithi146 ceph-mon[103278]: pgmap v13460: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:43.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:43 smithi078 ceph-mon[91136]: pgmap v13460: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:43 smithi078 ceph-mon[99959]: pgmap v13460: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:45.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:45 smithi146 ceph-mon[103278]: pgmap v13461: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:45 smithi078 ceph-mon[91136]: pgmap v13461: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:45 smithi078 ceph-mon[99959]: pgmap v13461: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:47 smithi146 ceph-mon[103278]: pgmap v13462: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:47 smithi078 ceph-mon[99959]: pgmap v13462: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:47 smithi078 ceph-mon[91136]: pgmap v13462: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:49 smithi146 ceph-mon[103278]: pgmap v13463: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:49 smithi078 ceph-mon[99959]: pgmap v13463: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:49 smithi078 ceph-mon[91136]: pgmap v13463: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:48:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:48:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:48:51.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:51 smithi146 ceph-mon[103278]: pgmap v13464: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:51 smithi078 ceph-mon[91136]: pgmap v13464: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:51 smithi078 ceph-mon[99959]: pgmap v13464: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:52.418 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:52.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:52.419 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:52.064Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:48:52.419 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:48:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:48:52.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:48:53.500 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:53 smithi146 ceph-mon[103278]: pgmap v13465: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:53.500 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:53 smithi078 ceph-mon[91136]: pgmap v13465: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:53 smithi078 ceph-mon[99959]: pgmap v13465: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:48:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:55 smithi146 ceph-mon[103278]: pgmap v13466: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:55 smithi078 ceph-mon[91136]: pgmap v13466: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:55 smithi078 ceph-mon[99959]: pgmap v13466: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:57 smithi146 ceph-mon[103278]: pgmap v13467: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:57 smithi078 ceph-mon[91136]: pgmap v13467: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:57 smithi078 ceph-mon[99959]: pgmap v13467: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:48:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:59 smithi146 ceph-mon[103278]: pgmap v13468: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:48:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[91136]: pgmap v13468: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:59.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:59.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[99959]: pgmap v13468: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:48:59.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:48:59.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:48:59.528 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:48:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:00.842 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:49:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:49:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:49:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:01 smithi146 ceph-mon[103278]: pgmap v13469: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:01 smithi078 ceph-mon[91136]: pgmap v13469: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:01 smithi078 ceph-mon[99959]: pgmap v13469: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:02.421 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:02.064Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:02.421 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:02.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:02.421 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:02.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:03 smithi078 ceph-mon[91136]: pgmap v13470: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:03 smithi078 ceph-mon[99959]: pgmap v13470: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:03.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:03 smithi146 ceph-mon[103278]: pgmap v13470: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:03.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[91136]: pgmap v13471: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:05.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:05 smithi078 ceph-mon[99959]: pgmap v13471: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:05 smithi146 ceph-mon[103278]: pgmap v13471: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:07 smithi146 ceph-mon[103278]: pgmap v13472: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:07 smithi078 ceph-mon[91136]: pgmap v13472: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:07.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:07 smithi078 ceph-mon[99959]: pgmap v13472: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:09.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:09 smithi146 ceph-mon[103278]: pgmap v13473: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:09 smithi078 ceph-mon[91136]: pgmap v13473: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:09.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:09 smithi078 ceph-mon[99959]: pgmap v13473: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:49:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:49:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:49:11.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:11 smithi146 ceph-mon[103278]: pgmap v13474: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:11.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:11 smithi078 ceph-mon[91136]: pgmap v13474: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:11.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:11 smithi078 ceph-mon[99959]: pgmap v13474: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:11.831 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:49:12.144 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:12.065Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:12.144 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:12.065Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:12.144 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:12.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:12.358 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:49:12.358 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 7h - - 2024-09-21T03:49:12.358 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 7h - - 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 13s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 8s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 13s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 8s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 13s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 13s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 8s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 13s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 13s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 8s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 13s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 13s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:49:12.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 13s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 13s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 8s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 8s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 8s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 8s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 8s ago 7h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 13s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 8s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 13s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:49:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:49:12.430 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:12 smithi078 ceph-mon[99959]: from='client.94523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:12.431 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:12 smithi078 ceph-mon[91136]: from='client.94523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:12.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:12 smithi146 ceph-mon[103278]: from='client.94523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:49:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:49:12.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:49:12.940 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:49:12.940 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:49:12.940 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:49:12.940 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:49:12.941 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:49:13.255 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:49:13.255 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:49:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:49:13.255 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:49:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[99959]: pgmap v13475: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[99959]: from='client.94529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[99959]: from='client.94535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3176295241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[91136]: pgmap v13475: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[91136]: from='client.94529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[91136]: from='client.94535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:13.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:13 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3176295241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:49:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:13 smithi146 ceph-mon[103278]: pgmap v13475: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:13 smithi146 ceph-mon[103278]: from='client.94529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:13 smithi146 ceph-mon[103278]: from='client.94535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:13 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3176295241' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:49:14.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:14 smithi146 ceph-mon[103278]: from='client.94547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:14 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2911781060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:49:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:14 smithi078 ceph-mon[91136]: from='client.94547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:14 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2911781060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:49:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:14 smithi078 ceph-mon[99959]: from='client.94547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:14 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2911781060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:49:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:15 smithi146 ceph-mon[103278]: pgmap v13476: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:15 smithi078 ceph-mon[99959]: pgmap v13476: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:15 smithi078 ceph-mon[91136]: pgmap v13476: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:17.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:17 smithi146 ceph-mon[103278]: pgmap v13477: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:17 smithi078 ceph-mon[99959]: pgmap v13477: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:17 smithi078 ceph-mon[91136]: pgmap v13477: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:19.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:19 smithi078 ceph-mon[91136]: pgmap v13478: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:19.308 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:19 smithi078 ceph-mon[99959]: pgmap v13478: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:19.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:19 smithi146 ceph-mon[103278]: pgmap v13478: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:20.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:49:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:49:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:49:21.465 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:21 smithi078 ceph-mon[99959]: pgmap v13479: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:21.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:21 smithi078 ceph-mon[91136]: pgmap v13479: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:21 smithi146 ceph-mon[103278]: pgmap v13479: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:22.066Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:22.066Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:22.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:23 smithi078 ceph-mon[91136]: pgmap v13480: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:23 smithi078 ceph-mon[99959]: pgmap v13480: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:23.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:23 smithi146 ceph-mon[103278]: pgmap v13480: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:49:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:25 smithi146 ceph-mon[103278]: pgmap v13481: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[99959]: pgmap v13481: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[91136]: pgmap v13481: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:25.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:25.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:27 smithi146 ceph-mon[103278]: pgmap v13482: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:27 smithi078 ceph-mon[91136]: pgmap v13482: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:27 smithi078 ceph-mon[99959]: pgmap v13482: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:29 smithi146 ceph-mon[103278]: pgmap v13483: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:29 smithi078 ceph-mon[91136]: pgmap v13483: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:29 smithi078 ceph-mon[99959]: pgmap v13483: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:49:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:49:30] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:49:31.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:31 smithi146 ceph-mon[103278]: pgmap v13484: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:31.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:31 smithi078 ceph-mon[99959]: pgmap v13484: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:31.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:31 smithi078 ceph-mon[91136]: pgmap v13484: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:32.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:32.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:32.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:32.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:32.439 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:32.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:33 smithi146 ceph-mon[103278]: pgmap v13485: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:33 smithi078 ceph-mon[91136]: pgmap v13485: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:33 smithi078 ceph-mon[99959]: pgmap v13485: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:35 smithi146 ceph-mon[103278]: pgmap v13486: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:35.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:35 smithi078 ceph-mon[91136]: pgmap v13486: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:35.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:35 smithi078 ceph-mon[99959]: pgmap v13486: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:37.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:37 smithi146 ceph-mon[103278]: pgmap v13487: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:37 smithi078 ceph-mon[99959]: pgmap v13487: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:37 smithi078 ceph-mon[91136]: pgmap v13487: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:39.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:39 smithi146 ceph-mon[103278]: pgmap v13488: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:39 smithi078 ceph-mon[91136]: pgmap v13488: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:39 smithi078 ceph-mon[99959]: pgmap v13488: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:40.970 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:49:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:49:40] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:49:41.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:41 smithi078 ceph-mon[91136]: pgmap v13489: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:41.500 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:41 smithi078 ceph-mon[99959]: pgmap v13489: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:41.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:41 smithi146 ceph-mon[103278]: pgmap v13489: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:42.448 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:42.067Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:42.448 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:42.067Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:42.448 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:42.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:43.551 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:49:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:43 smithi146 ceph-mon[103278]: pgmap v13490: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:43 smithi078 ceph-mon[99959]: pgmap v13490: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:43 smithi078 ceph-mon[91136]: pgmap v13490: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 4s ago 7h - - 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 19s ago 7h - - 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 4s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 19s ago 7h 91.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 4s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:49:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 19s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 4s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 4s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 19s ago 7h 374M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 4s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 4s ago 7h 15.1M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 19s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 4s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 4s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 4s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 4s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 19s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 19s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 19s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 19s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 19s ago 7h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:49:44.087 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 4s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:49:44.088 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 19s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:49:44.088 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 4s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:49:44.088 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 19s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:49:44.400 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:49:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:49:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:49:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:49:44.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:49:44.672 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:49:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:49:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:49:44.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='client.94559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='client.74649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/940238063' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='client.94559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='client.74649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:44.674 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.675 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/940238063' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='client.94559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='client.74649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:49:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/940238063' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:49:44.982 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:49:44.982 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:49:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:49:44.982 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:49:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:49:45.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:45 smithi146 ceph-mon[103278]: from='client.74655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:45 smithi146 ceph-mon[103278]: pgmap v13491: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:45 smithi146 ceph-mon[103278]: from='client.94583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:45 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/449480851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:49:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[99959]: from='client.74655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[99959]: pgmap v13491: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[99959]: from='client.94583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/449480851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:49:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[91136]: from='client.74655 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[91136]: pgmap v13491: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:45.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[91136]: from='client.94583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:49:45.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:45 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/449480851' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:49:47.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:47 smithi146 ceph-mon[103278]: pgmap v13492: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:47 smithi078 ceph-mon[91136]: pgmap v13492: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:47 smithi078 ceph-mon[99959]: pgmap v13492: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:49.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:49 smithi146 ceph-mon[103278]: pgmap v13493: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:49 smithi078 ceph-mon[91136]: pgmap v13493: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:49 smithi078 ceph-mon[99959]: pgmap v13493: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:49:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:49:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:49:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:51 smithi146 ceph-mon[103278]: pgmap v13494: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:51 smithi078 ceph-mon[91136]: pgmap v13494: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:51 smithi078 ceph-mon[99959]: pgmap v13494: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:52.458 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:52.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:52.458 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:52.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:49:52.458 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:49:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:49:52.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:49:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:53 smithi146 ceph-mon[103278]: pgmap v13495: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:53 smithi078 ceph-mon[91136]: pgmap v13495: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:53 smithi078 ceph-mon[99959]: pgmap v13495: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:49:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:55 smithi146 ceph-mon[103278]: pgmap v13496: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:55 smithi078 ceph-mon[99959]: pgmap v13496: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:55 smithi078 ceph-mon[91136]: pgmap v13496: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:57.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:57 smithi146 ceph-mon[103278]: pgmap v13497: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:57 smithi078 ceph-mon[91136]: pgmap v13497: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:57 smithi078 ceph-mon[99959]: pgmap v13497: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:49:59.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:59 smithi146 ceph-mon[103278]: pgmap v13498: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:49:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:59 smithi078 ceph-mon[99959]: pgmap v13498: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:49:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:49:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:59 smithi078 ceph-mon[91136]: pgmap v13498: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:49:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:49:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:00.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T03:49:59.999+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:50:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:50:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:50:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:50:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:00.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:50:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:01.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:50:01.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:01 smithi146 ceph-mon[103278]: pgmap v13499: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:01 smithi078 ceph-mon[99959]: pgmap v13499: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:01 smithi078 ceph-mon[91136]: pgmap v13499: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:02.466 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:02.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:02.467 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:02.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:02.467 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:02.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:03 smithi146 ceph-mon[103278]: pgmap v13500: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:03 smithi078 ceph-mon[91136]: pgmap v13500: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:03 smithi078 ceph-mon[99959]: pgmap v13500: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:05 smithi146 ceph-mon[103278]: pgmap v13501: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[91136]: pgmap v13501: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[99959]: pgmap v13501: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:05.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:07.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:07 smithi146 ceph-mon[103278]: pgmap v13502: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:07 smithi078 ceph-mon[91136]: pgmap v13502: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:07 smithi078 ceph-mon[99959]: pgmap v13502: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:09 smithi146 ceph-mon[103278]: pgmap v13503: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:09 smithi078 ceph-mon[91136]: pgmap v13503: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:09 smithi078 ceph-mon[99959]: pgmap v13503: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:50:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:50:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:50:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:11 smithi146 ceph-mon[103278]: pgmap v13504: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:11 smithi078 ceph-mon[99959]: pgmap v13504: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:11 smithi078 ceph-mon[91136]: pgmap v13504: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:12.345 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:12.068Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:12.345 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:12.068Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:12.345 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:12.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:13 smithi146 ceph-mon[103278]: pgmap v13505: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:13.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:13 smithi078 ceph-mon[91136]: pgmap v13505: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:13 smithi078 ceph-mon[99959]: pgmap v13505: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:15.279 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:50:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:15 smithi078 ceph-mon[99959]: pgmap v13506: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:15 smithi078 ceph-mon[91136]: pgmap v13506: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:15 smithi146 ceph-mon[103278]: pgmap v13506: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:15.808 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:50:15.808 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 7h - - 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 7h - - 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 16s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 11s ago 7h 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 16s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 11s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 16s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 16s ago 7h 564M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 11s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 16s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 16s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 11s ago 7h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 16s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 16s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:50:15.809 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 16s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 16s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 11s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 11s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 11s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 11s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 11s ago 7h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 16s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 11s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 16s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:50:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:50:16.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:50:16.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:50:16.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:50:16.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:16.118 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:50:16.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:50:16.119 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:50:16.387 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:50:16.388 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:50:16.696 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:50:16.697 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:50:16.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:50:16.697 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:50:16.697 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:50:16.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[91136]: from='client.94595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[91136]: from='client.94601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[91136]: from='client.74691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.697 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2221368574' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:50:16.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[99959]: from='client.94595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.697 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[99959]: from='client.94601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[99959]: from='client.74691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.698 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:16 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2221368574' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:50:16.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:16 smithi146 ceph-mon[103278]: from='client.94595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:16 smithi146 ceph-mon[103278]: from='client.94601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:16 smithi146 ceph-mon[103278]: from='client.74691 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:16.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:16 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2221368574' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:50:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:17 smithi146 ceph-mon[103278]: pgmap v13507: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:17 smithi146 ceph-mon[103278]: from='client.94619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:17 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/50203917' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:50:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:17 smithi078 ceph-mon[91136]: pgmap v13507: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:17 smithi078 ceph-mon[91136]: from='client.94619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:17 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/50203917' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:50:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:17 smithi078 ceph-mon[99959]: pgmap v13507: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:17 smithi078 ceph-mon[99959]: from='client.94619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:17 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/50203917' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:50:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:19 smithi146 ceph-mon[103278]: pgmap v13508: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:19 smithi078 ceph-mon[91136]: pgmap v13508: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:19 smithi078 ceph-mon[99959]: pgmap v13508: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:21.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:50:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:21.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:21 smithi146 ceph-mon[103278]: pgmap v13509: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:21 smithi078 ceph-mon[91136]: pgmap v13509: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:21 smithi078 ceph-mon[99959]: pgmap v13509: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:22.256 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:22.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:22.257 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:22.069Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:22.257 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:22.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:23.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:23 smithi078 ceph-mon[99959]: pgmap v13510: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:23.941 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:23 smithi078 ceph-mon[91136]: pgmap v13510: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:23.941 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:23 smithi146 ceph-mon[103278]: pgmap v13510: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:24.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:25 smithi146 ceph-mon[103278]: pgmap v13511: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[91136]: pgmap v13511: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:26.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[99959]: pgmap v13511: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:26.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:26.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:27.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:27 smithi146 ceph-mon[103278]: pgmap v13512: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:27 smithi078 ceph-mon[91136]: pgmap v13512: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:27 smithi078 ceph-mon[99959]: pgmap v13512: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:29 smithi146 ceph-mon[103278]: pgmap v13513: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:29 smithi078 ceph-mon[91136]: pgmap v13513: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:30.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:29 smithi078 ceph-mon[99959]: pgmap v13513: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:30.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:50:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:50:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:50:32.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:31 smithi146 ceph-mon[103278]: pgmap v13514: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:32.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:31 smithi078 ceph-mon[91136]: pgmap v13514: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:32.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:31 smithi078 ceph-mon[99959]: pgmap v13514: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:32.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:32.069Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:32.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:32.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:32.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:32.070Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:34.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:33 smithi146 ceph-mon[103278]: pgmap v13515: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:34.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:33 smithi078 ceph-mon[99959]: pgmap v13515: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:33 smithi078 ceph-mon[91136]: pgmap v13515: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:35 smithi146 ceph-mon[103278]: pgmap v13516: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:36.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:35 smithi078 ceph-mon[91136]: pgmap v13516: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:36.093 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:35 smithi078 ceph-mon[99959]: pgmap v13516: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:38.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:37 smithi146 ceph-mon[103278]: pgmap v13517: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:38.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:37 smithi078 ceph-mon[91136]: pgmap v13517: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:38.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:37 smithi078 ceph-mon[99959]: pgmap v13517: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:39 smithi146 ceph-mon[103278]: pgmap v13518: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:40.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:39 smithi078 ceph-mon[99959]: pgmap v13518: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:40.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:39 smithi078 ceph-mon[91136]: pgmap v13518: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:50:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:50:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:40 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:40 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:41.965 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:41 smithi078 ceph-mon[91136]: pgmap v13519: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:41.965 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:41 smithi078 ceph-mon[99959]: pgmap v13519: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:42.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:41 smithi146 ceph-mon[103278]: pgmap v13519: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:42.070Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:42.071Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:42.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:44.221 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:43 smithi078 ceph-mon[99959]: pgmap v13520: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:44.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:44.222 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:44.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:43 smithi078 ceph-mon[91136]: pgmap v13520: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:44.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:44.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:44.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:43 smithi146 ceph-mon[103278]: pgmap v13520: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:45 smithi146 ceph-mon[103278]: pgmap v13521: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[91136]: pgmap v13521: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[99959]: pgmap v13521: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:50:46.991 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 7s ago 7h - - 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 2s ago 7h - - 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 7s ago 7h 29.6M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 2s ago 7h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 7s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 2s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 7s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:50:47.513 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 7s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 2s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 7s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 7s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 2s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 7s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 7s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 7s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 7s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 2s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 2s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 2s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 2s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 2s ago 7h 73.5M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:50:47.514 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 7s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:50:47.515 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 2s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:50:47.515 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 7s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:50:47.515 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 2s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:50:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:47 smithi078 ceph-mon[91136]: pgmap v13522: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:47 smithi078 ceph-mon[99959]: pgmap v13522: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:47.821 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:50:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:50:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:50:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:50:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:50:47.823 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:50:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:47 smithi146 ceph-mon[103278]: pgmap v13522: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:48.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:50:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:50:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:50:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:50:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:50:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:50:48.090 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:50:48.396 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:50:48.396 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:50:48.396 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:50:48.396 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:50:48.397 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:50:48.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:48 smithi146 ceph-mon[103278]: from='client.94631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:48.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:48 smithi146 ceph-mon[103278]: from='client.94637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:48.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:48 smithi146 ceph-mon[103278]: from='client.74727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:48.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1912069388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:50:48.948 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3160144625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:50:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[91136]: from='client.94631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[91136]: from='client.94637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[91136]: from='client.74727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1912069388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:50:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3160144625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:50:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[99959]: from='client.94631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[99959]: from='client.94637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[99959]: from='client.74727 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1912069388' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:50:49.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3160144625' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:50:49.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:49 smithi146 ceph-mon[103278]: from='client.74739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:49.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:49 smithi146 ceph-mon[103278]: pgmap v13523: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:50.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:49 smithi078 ceph-mon[91136]: from='client.74739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:50.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:49 smithi078 ceph-mon[91136]: pgmap v13523: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:50.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:49 smithi078 ceph-mon[99959]: from='client.74739 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:50:50.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:49 smithi078 ceph-mon[99959]: pgmap v13523: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:50:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:50:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:50:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:51 smithi146 ceph-mon[103278]: pgmap v13524: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:51 smithi078 ceph-mon[91136]: pgmap v13524: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:51 smithi078 ceph-mon[99959]: pgmap v13524: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:52.519 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:52.071Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:52.519 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:52.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:50:52.520 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:50:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:50:52.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:50:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:53 smithi078 ceph-mon[99959]: pgmap v13525: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:53.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:53 smithi078 ceph-mon[91136]: pgmap v13525: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:53.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:53 smithi146 ceph-mon[103278]: pgmap v13525: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:50:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:50:56.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:55 smithi146 ceph-mon[103278]: pgmap v13526: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:55 smithi078 ceph-mon[99959]: pgmap v13526: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:55 smithi078 ceph-mon[91136]: pgmap v13526: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:50:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:57 smithi146 ceph-mon[103278]: pgmap v13527: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:50:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:57 smithi078 ceph-mon[91136]: pgmap v13527: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:50:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:57 smithi078 ceph-mon[99959]: pgmap v13527: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-21T03:50:59.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:50:59.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:00.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:50:59 smithi146 ceph-mon[103278]: pgmap v13528: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:51:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:50:59 smithi078 ceph-mon[99959]: pgmap v13528: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:51:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:50:59 smithi078 ceph-mon[91136]: pgmap v13528: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T03:51:00.917 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:51:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:51:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:00 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:01.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:00 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:01 smithi146 ceph-mon[103278]: pgmap v13529: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T03:51:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:01 smithi078 ceph-mon[91136]: pgmap v13529: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T03:51:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:01 smithi078 ceph-mon[99959]: pgmap v13529: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T03:51:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:02.072Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:02.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:02.072Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:02.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:02.073Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:02 smithi146 ceph-mon[103278]: pgmap v13530: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:51:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:02 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-mon[91136]: pgmap v13530: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:51:03.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-mon[99959]: pgmap v13530: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T03:51:03.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:02 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:05 smithi146 ceph-mon[103278]: pgmap v13531: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:51:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[91136]: pgmap v13531: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:51:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[99959]: pgmap v13531: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T03:51:06.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:06.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:08.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:07 smithi146 ceph-mon[103278]: pgmap v13532: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:51:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:07 smithi078 ceph-mon[91136]: pgmap v13532: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:51:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:07 smithi078 ceph-mon[99959]: pgmap v13532: 161 pgs: 161 active+clean; 585 KiB data, 415 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:51:10.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:09 smithi146 ceph-mon[103278]: pgmap v13533: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:51:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:09 smithi078 ceph-mon[91136]: pgmap v13533: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:51:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:09 smithi078 ceph-mon[99959]: pgmap v13533: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-09-21T03:51:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:51:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:51:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:51:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:11 smithi146 ceph-mon[103278]: pgmap v13534: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-21T03:51:12.075 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:11 smithi078 ceph-mon[99959]: pgmap v13534: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-21T03:51:12.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:11 smithi078 ceph-mon[91136]: pgmap v13534: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-21T03:51:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:12.073Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:12.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:12.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:13 smithi146 ceph-mon[103278]: pgmap v13535: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:51:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:13 smithi078 ceph-mon[99959]: pgmap v13535: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:51:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:13 smithi078 ceph-mon[91136]: pgmap v13535: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-09-21T03:51:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:15 smithi146 ceph-mon[103278]: pgmap v13536: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-21T03:51:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:15 smithi078 ceph-mon[99959]: pgmap v13536: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-21T03:51:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:15 smithi078 ceph-mon[91136]: pgmap v13536: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s 2024-09-21T03:51:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:17 smithi146 ceph-mon[103278]: pgmap v13537: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:51:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:17 smithi078 ceph-mon[91136]: pgmap v13537: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:51:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:17 smithi078 ceph-mon[99959]: pgmap v13537: 161 pgs: 161 active+clean; 585 KiB data, 419 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-21T03:51:18.679 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 7h - - 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 14s ago 7h - - 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 19s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 14s ago 7h 91.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 19s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 14s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 19s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:51:19.207 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 19s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 14s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 19s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 19s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 14s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 19s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 19s ago 7h 108M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 19s ago 7h 88.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 19s ago 7h 107M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 14s ago 7h 123M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 14s ago 7h 95.7M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 14s ago 7h 103M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 14s ago 7h 98.6M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 14s ago 7h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 19s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:51:19.208 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 14s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:51:19.209 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 19s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:51:19.209 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 14s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:51:19.516 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:51:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:51:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:51:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:51:19.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:51:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:51:19.780 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:51:19.780 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:51:19.780 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:51:19.780 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:51:19.781 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:51:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:19 smithi078 ceph-mon[91136]: pgmap v13538: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T03:51:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:19 smithi078 ceph-mon[91136]: from='client.94667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/656542903' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:51:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:19 smithi078 ceph-mon[99959]: pgmap v13538: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T03:51:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:19 smithi078 ceph-mon[99959]: from='client.94667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/656542903' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:51:20.100 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:51:20.100 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:51:20.100 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:51:20.100 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:51:20.101 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:51:20.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:19 smithi146 ceph-mon[103278]: pgmap v13538: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 126 op/s 2024-09-21T03:51:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:19 smithi146 ceph-mon[103278]: from='client.94667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/656542903' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:51:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:51:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:51:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='client.74757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='client.74763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='client.74775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1599914186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='client.74757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='client.74763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='client.74775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1599914186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:51:21.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='client.74757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='client.74763 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='client.74775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:21.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:21.168 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1599914186' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:51:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:21 smithi146 ceph-mon[103278]: pgmap v13539: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:51:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:21 smithi078 ceph-mon[91136]: pgmap v13539: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:51:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:21 smithi078 ceph-mon[99959]: pgmap v13539: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-09-21T03:51:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:22.074Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:22.074Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:22.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:24.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:23 smithi078 ceph-mon[91136]: pgmap v13540: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T03:51:24.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:24.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:23 smithi078 ceph-mon[99959]: pgmap v13540: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T03:51:24.186 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:23 smithi146 ceph-mon[103278]: pgmap v13540: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 131 op/s 2024-09-21T03:51:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:25 smithi146 ceph-mon[103278]: pgmap v13541: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:51:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[91136]: pgmap v13541: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[99959]: pgmap v13541: 161 pgs: 161 active+clean; 585 KiB data, 423 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:28.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:27 smithi146 ceph-mon[103278]: pgmap v13542: 161 pgs: 161 active+clean; 585 KiB data, 444 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:51:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:27 smithi078 ceph-mon[91136]: pgmap v13542: 161 pgs: 161 active+clean; 585 KiB data, 444 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:51:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:27 smithi078 ceph-mon[99959]: pgmap v13542: 161 pgs: 161 active+clean; 585 KiB data, 444 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s 2024-09-21T03:51:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:30.081 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:29 smithi146 ceph-mon[103278]: pgmap v13543: 161 pgs: 161 active+clean; 585 KiB data, 444 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:51:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:29 smithi078 ceph-mon[91136]: pgmap v13543: 161 pgs: 161 active+clean; 585 KiB data, 444 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:51:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:29 smithi078 ceph-mon[99959]: pgmap v13543: 161 pgs: 161 active+clean; 585 KiB data, 444 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 107 op/s 2024-09-21T03:51:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:51:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:51:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:51:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:31 smithi146 ceph-mon[103278]: pgmap v13544: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T03:51:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:31 smithi078 ceph-mon[91136]: pgmap v13544: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T03:51:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:31 smithi078 ceph-mon[99959]: pgmap v13544: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-21T03:51:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:32.074Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:32.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:32.075Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:33 smithi146 ceph-mon[103278]: pgmap v13545: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:51:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:33 smithi078 ceph-mon[91136]: pgmap v13545: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:51:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:33 smithi078 ceph-mon[99959]: pgmap v13545: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-21T03:51:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:35 smithi146 ceph-mon[103278]: pgmap v13546: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T03:51:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:35 smithi078 ceph-mon[91136]: pgmap v13546: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T03:51:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:35 smithi078 ceph-mon[99959]: pgmap v13546: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T03:51:38.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:37 smithi146 ceph-mon[103278]: pgmap v13547: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:51:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:37 smithi078 ceph-mon[91136]: pgmap v13547: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:51:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:37 smithi078 ceph-mon[99959]: pgmap v13547: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-09-21T03:51:40.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:39 smithi078 ceph-mon[91136]: pgmap v13548: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:51:40.188 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:39 smithi078 ceph-mon[99959]: pgmap v13548: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:51:40.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:39 smithi146 ceph-mon[103278]: pgmap v13548: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 18 op/s 2024-09-21T03:51:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:51:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:51:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:51:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:41 smithi146 ceph-mon[103278]: pgmap v13549: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[99959]: pgmap v13549: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[91136]: pgmap v13549: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:42.075Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:42.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:42.076Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:43 smithi146 ceph-mon[103278]: pgmap v13550: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:43 smithi078 ceph-mon[91136]: pgmap v13550: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:43 smithi078 ceph-mon[99959]: pgmap v13550: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[91136]: pgmap v13551: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[99959]: pgmap v13551: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:45 smithi146 ceph-mon[103278]: pgmap v13551: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:51:47.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:47 smithi146 ceph-mon[103278]: pgmap v13552: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:47 smithi078 ceph-mon[91136]: pgmap v13552: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:47 smithi078 ceph-mon[99959]: pgmap v13552: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:49 smithi146 ceph-mon[103278]: pgmap v13553: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:49 smithi078 ceph-mon[91136]: pgmap v13553: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:49 smithi078 ceph-mon[99959]: pgmap v13553: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:50.390 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:51:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:51:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:51:50] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:51:50.911 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:51:50.911 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 10s ago 7h - - 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 5s ago 7h - - 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 10s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 5s ago 7h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 10s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 5s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 10s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 10s ago 7h 567M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 5s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 10s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 10s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 5s ago 7h 14.7M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 10s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 10s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:51:50.912 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 10s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 10s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 5s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 5s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 5s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 5s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 5s ago 7h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 10s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 5s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 10s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:51:50.913 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 5s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:51:51.221 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:51.222 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:51:51.223 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:51:51.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:51:51.223 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:51:51.223 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:51:51.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:51:51.223 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:51:51.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:51:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:51:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:51:51.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[99959]: pgmap v13554: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[99959]: from='client.94703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[99959]: from='client.94709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[99959]: from='client.74799 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[91136]: pgmap v13554: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[91136]: from='client.94703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[91136]: from='client.94709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:51 smithi078 ceph-mon[91136]: from='client.74799 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:51 smithi146 ceph-mon[103278]: pgmap v13554: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:51.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:51 smithi146 ceph-mon[103278]: from='client.94703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:51 smithi146 ceph-mon[103278]: from='client.94709 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.491 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:51 smithi146 ceph-mon[103278]: from='client.74799 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:51.800 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:51:51.800 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:51:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:51:51.800 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:51:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:51:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1010498301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:51:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:52 smithi146 ceph-mon[103278]: from='client.94727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:52.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:52 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3856944767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:51:52.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1010498301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:51:52.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-mon[99959]: from='client.94727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:52.532 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3856944767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:51:52.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1010498301' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:51:52.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-mon[91136]: from='client.94727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:51:52.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3856944767' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:51:52.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:52.076Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:52.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:52.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:51:52.533 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:51:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:51:52.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:51:53.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:53 smithi146 ceph-mon[103278]: pgmap v13555: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:53.489 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:53 smithi078 ceph-mon[91136]: pgmap v13555: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:53 smithi078 ceph-mon[99959]: pgmap v13555: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:53.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:51:55.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:55 smithi146 ceph-mon[103278]: pgmap v13556: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:55 smithi078 ceph-mon[91136]: pgmap v13556: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:55 smithi078 ceph-mon[99959]: pgmap v13556: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:57 smithi146 ceph-mon[103278]: pgmap v13557: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:57 smithi078 ceph-mon[91136]: pgmap v13557: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:57 smithi078 ceph-mon[99959]: pgmap v13557: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:59 smithi146 ceph-mon[103278]: pgmap v13558: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:51:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:59 smithi078 ceph-mon[91136]: pgmap v13558: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:51:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:51:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:59 smithi078 ceph-mon[99959]: pgmap v13558: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:51:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:51:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:52:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:52:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:52:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:01 smithi146 ceph-mon[103278]: pgmap v13559: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[91136]: pgmap v13559: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[99959]: pgmap v13559: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:01.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:02.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:02.077Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:02.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:03 smithi146 ceph-mon[103278]: pgmap v13560: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:03 smithi078 ceph-mon[91136]: pgmap v13560: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:03 smithi078 ceph-mon[99959]: pgmap v13560: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:05 smithi146 ceph-mon[103278]: pgmap v13561: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[91136]: pgmap v13561: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[99959]: pgmap v13561: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:07.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:07 smithi078 ceph-mon[91136]: pgmap v13562: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:07.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:07 smithi078 ceph-mon[99959]: pgmap v13562: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:07 smithi146 ceph-mon[103278]: pgmap v13562: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:09 smithi146 ceph-mon[103278]: pgmap v13563: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:09 smithi078 ceph-mon[91136]: pgmap v13563: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:09 smithi078 ceph-mon[99959]: pgmap v13563: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:52:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:52:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:52:11.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:11 smithi146 ceph-mon[103278]: pgmap v13564: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:11 smithi078 ceph-mon[91136]: pgmap v13564: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:11 smithi078 ceph-mon[99959]: pgmap v13564: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:12.077Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:12.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:12.078Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:13.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:13 smithi146 ceph-mon[103278]: pgmap v13565: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:13.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:13 smithi078 ceph-mon[91136]: pgmap v13565: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:13 smithi078 ceph-mon[99959]: pgmap v13565: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:13.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:15.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:15 smithi146 ceph-mon[103278]: pgmap v13566: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:15 smithi078 ceph-mon[91136]: pgmap v13566: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:15 smithi078 ceph-mon[99959]: pgmap v13566: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:17 smithi146 ceph-mon[103278]: pgmap v13567: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:17 smithi078 ceph-mon[91136]: pgmap v13567: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:17 smithi078 ceph-mon[99959]: pgmap v13567: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:19 smithi146 ceph-mon[103278]: pgmap v13568: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:19 smithi078 ceph-mon[91136]: pgmap v13568: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:19 smithi078 ceph-mon[99959]: pgmap v13568: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:52:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:52:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:52:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:21 smithi146 ceph-mon[103278]: pgmap v13569: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[99959]: pgmap v13569: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[91136]: pgmap v13569: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:21.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:22.099 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:52:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:22.078Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:22.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:22.079Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:22.623 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 7h - - 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 17s ago 7h - - 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 2s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 17s ago 7h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 2s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 17s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 2s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 2s ago 7h 568M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 17s ago 7h 378M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 2s ago 7h 383M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 2s ago 7h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 17s ago 7h 15.4M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:52:22.624 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 2s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 2s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 2s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 2s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 17s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 17s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 17s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 17s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 17s ago 7h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 2s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 17s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 2s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:52:22.625 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 17s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:52:22.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:52:22.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:52:23.204 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:52:23.205 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:52:23.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[91136]: from='client.74823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[91136]: pgmap v13570: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[91136]: from='client.94745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[91136]: from='client.94751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1287739943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[99959]: from='client.74823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[99959]: pgmap v13570: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[99959]: from='client.94745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[99959]: from='client.94751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.516 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:23 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1287739943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:52:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:23 smithi146 ceph-mon[103278]: from='client.74823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:23 smithi146 ceph-mon[103278]: pgmap v13570: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:23 smithi146 ceph-mon[103278]: from='client.94745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:23 smithi146 ceph-mon[103278]: from='client.94751 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:23 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1287739943' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:52:24.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:24 smithi078 ceph-mon[91136]: from='client.94763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:24.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:24 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3666938085' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:52:24.710 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:24 smithi078 ceph-mon[99959]: from='client.94763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:24.711 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:24 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3666938085' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:52:24.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:24 smithi146 ceph-mon[103278]: from='client.94763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:24.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:24 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3666938085' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:52:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:25 smithi146 ceph-mon[103278]: pgmap v13571: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[91136]: pgmap v13571: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[99959]: pgmap v13571: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:25.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:25.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:27.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:27 smithi146 ceph-mon[103278]: pgmap v13572: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:27 smithi078 ceph-mon[91136]: pgmap v13572: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:27 smithi078 ceph-mon[99959]: pgmap v13572: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:29 smithi146 ceph-mon[103278]: pgmap v13573: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:29 smithi078 ceph-mon[91136]: pgmap v13573: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:29 smithi078 ceph-mon[99959]: pgmap v13573: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:52:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:52:30] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:52:31.740 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:31 smithi146 ceph-mon[103278]: pgmap v13574: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:31 smithi078 ceph-mon[91136]: pgmap v13574: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:31 smithi078 ceph-mon[99959]: pgmap v13574: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:32.079Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:32.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:32.080Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:33.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:33 smithi146 ceph-mon[103278]: pgmap v13575: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:33.727 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:33 smithi078 ceph-mon[91136]: pgmap v13575: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:33 smithi078 ceph-mon[99959]: pgmap v13575: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:35 smithi146 ceph-mon[103278]: pgmap v13576: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:35 smithi078 ceph-mon[99959]: pgmap v13576: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:35 smithi078 ceph-mon[91136]: pgmap v13576: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:37 smithi146 ceph-mon[103278]: pgmap v13577: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:37 smithi078 ceph-mon[91136]: pgmap v13577: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:37 smithi078 ceph-mon[99959]: pgmap v13577: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:39.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:39 smithi146 ceph-mon[103278]: pgmap v13578: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:39 smithi078 ceph-mon[91136]: pgmap v13578: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:39 smithi078 ceph-mon[99959]: pgmap v13578: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:52:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:52:40] "GET /metrics HTTP/1.1" 200 39003 "" "Prometheus/2.43.0" 2024-09-21T03:52:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:41 smithi146 ceph-mon[103278]: pgmap v13579: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[99959]: pgmap v13579: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[91136]: pgmap v13579: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:42.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:42.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:42.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:43.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:43 smithi146 ceph-mon[103278]: pgmap v13580: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:43.976 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:43 smithi078 ceph-mon[91136]: pgmap v13580: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:43 smithi078 ceph-mon[99959]: pgmap v13580: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:45.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:45.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:45.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:52:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:45 smithi146 ceph-mon[103278]: pgmap v13581: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[99959]: pgmap v13581: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[91136]: pgmap v13581: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:45.902 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:52:47.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:47 smithi078 ceph-mon[91136]: pgmap v13582: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:47.877 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:47 smithi078 ceph-mon[99959]: pgmap v13582: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:48.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:47 smithi146 ceph-mon[103278]: pgmap v13582: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:49 smithi146 ceph-mon[103278]: pgmap v13583: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:49 smithi078 ceph-mon[91136]: pgmap v13583: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:49 smithi078 ceph-mon[99959]: pgmap v13583: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:52:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:52:50] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:52:52.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:51 smithi146 ceph-mon[103278]: pgmap v13584: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:51 smithi078 ceph-mon[91136]: pgmap v13584: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:52.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:51 smithi078 ceph-mon[99959]: pgmap v13584: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:52.080Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:52.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:52:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:52:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:52:52.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:52:53.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:53 smithi146 ceph-mon[103278]: pgmap v13585: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:53.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:53.815 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:52:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:53 smithi078 ceph-mon[91136]: pgmap v13585: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:54.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:53 smithi078 ceph-mon[99959]: pgmap v13585: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:54.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:52:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:52:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 13s ago 7h - - 2024-09-21T03:52:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 8s ago 7h - - 2024-09-21T03:52:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 13s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 8s ago 7h 91.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 13s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 8s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 13s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 13s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 8s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 13s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 13s ago 7h 15.0M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 8s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 13s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 13s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 13s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 13s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:52:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 8s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 8s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 8s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 8s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 8s ago 7h 74.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 13s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 8s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 13s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:52:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 8s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:52:54.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:52:54.655 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:52:54.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:54 smithi146 ceph-mon[103278]: from='client.94775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:52:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:52:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:52:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:52:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:52:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:52:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:52:54.922 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:52:54.922 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:54 smithi078 ceph-mon[99959]: from='client.94775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:54.922 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:54 smithi078 ceph-mon[91136]: from='client.94775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:55.226 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:52:55.227 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:52:55.227 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:52:55.227 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:52:55.227 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:52:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:55 smithi146 ceph-mon[103278]: from='client.74865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:55 smithi146 ceph-mon[103278]: pgmap v13586: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:55 smithi146 ceph-mon[103278]: from='client.94787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1865199960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:52:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:55 smithi146 ceph-mon[103278]: from='client.94799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:55 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2416625029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:52:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[91136]: from='client.74865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[91136]: pgmap v13586: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[91136]: from='client.94787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1865199960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:52:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[91136]: from='client.94799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2416625029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:52:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[99959]: from='client.74865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[99959]: pgmap v13586: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:52:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[99959]: from='client.94787 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1865199960' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:52:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[99959]: from='client.94799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:52:56.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:55 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2416625029' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:52:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:57 smithi146 ceph-mon[103278]: pgmap v13587: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:57 smithi078 ceph-mon[91136]: pgmap v13587: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:52:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:57 smithi078 ceph-mon[99959]: pgmap v13587: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:59 smithi146 ceph-mon[103278]: pgmap v13588: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:52:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:59 smithi078 ceph-mon[99959]: pgmap v13588: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:52:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:59 smithi078 ceph-mon[91136]: pgmap v13588: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:52:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:53:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:53:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:53:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:01 smithi146 ceph-mon[103278]: pgmap v13589: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[99959]: pgmap v13589: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[91136]: pgmap v13589: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:02.081Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:02.081Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:02.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:03 smithi146 ceph-mon[103278]: pgmap v13590: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:04.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:03 smithi078 ceph-mon[99959]: pgmap v13590: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:03 smithi078 ceph-mon[91136]: pgmap v13590: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:04.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:05 smithi146 ceph-mon[103278]: pgmap v13591: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:05 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[91136]: pgmap v13591: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[99959]: pgmap v13591: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:06.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:05 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:08.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:07 smithi146 ceph-mon[103278]: pgmap v13592: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:07 smithi078 ceph-mon[91136]: pgmap v13592: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:07 smithi078 ceph-mon[99959]: pgmap v13592: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:10.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:09 smithi146 ceph-mon[103278]: pgmap v13593: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:09 smithi078 ceph-mon[91136]: pgmap v13593: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:09 smithi078 ceph-mon[99959]: pgmap v13593: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:53:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:53:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:53:12.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:11 smithi146 ceph-mon[103278]: pgmap v13594: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:12.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:11 smithi078 ceph-mon[91136]: pgmap v13594: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:12.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:11 smithi078 ceph-mon[99959]: pgmap v13594: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:12.082Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:12.082Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:12.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:13 smithi146 ceph-mon[103278]: pgmap v13595: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:13 smithi078 ceph-mon[91136]: pgmap v13595: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:13 smithi078 ceph-mon[99959]: pgmap v13595: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:14.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:15.977 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:15 smithi146 ceph-mon[103278]: pgmap v13596: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:15 smithi078 ceph-mon[91136]: pgmap v13596: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:15 smithi078 ceph-mon[99959]: pgmap v13596: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:18.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:17 smithi146 ceph-mon[103278]: pgmap v13597: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:18.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:17 smithi078 ceph-mon[91136]: pgmap v13597: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:18.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:17 smithi078 ceph-mon[99959]: pgmap v13597: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:19 smithi146 ceph-mon[103278]: pgmap v13598: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:20.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:19 smithi078 ceph-mon[91136]: pgmap v13598: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:20.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:19 smithi078 ceph-mon[99959]: pgmap v13598: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:53:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:53:20] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[91136]: pgmap v13599: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[99959]: pgmap v13599: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.085 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.086 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:21 smithi146 ceph-mon[103278]: pgmap v13599: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:22.083Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:22.083Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:22.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:23 smithi078 ceph-mon[91136]: pgmap v13600: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:24.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:23 smithi078 ceph-mon[99959]: pgmap v13600: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:24.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:23 smithi146 ceph-mon[103278]: pgmap v13600: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:24.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:25.521 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:53:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:25 smithi146 ceph-mon[103278]: pgmap v13601: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:25 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 7h - - 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 0s ago 7h - - 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 5s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 0s ago 7h 88.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 5s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 0s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 5s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 5s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 0s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:53:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 5s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 5s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 0s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 5s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 5s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 0s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 0s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 0s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 0s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 0s ago 7h 74.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 5s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:53:26.053 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 0s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:53:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 5s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:53:26.054 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 0s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:53:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[91136]: pgmap v13601: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[99959]: pgmap v13601: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:25 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:53:26.364 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:53:26.365 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:53:26.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:53:26.635 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:53:26.957 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:53:26.957 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:53:26.957 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:53:26.957 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:53:26.957 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:53:26.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:26 smithi078 ceph-mon[99959]: from='client.74895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:26.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:26 smithi078 ceph-mon[99959]: from='client.74901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:26.957 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:26 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/768545010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:53:26.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:26 smithi078 ceph-mon[91136]: from='client.74895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:26.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:26 smithi078 ceph-mon[91136]: from='client.74901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:26.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:26 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/768545010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:53:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:26 smithi146 ceph-mon[103278]: from='client.74895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:26 smithi146 ceph-mon[103278]: from='client.74901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:27.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:26 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/768545010' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:53:28.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[91136]: from='client.74907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[91136]: pgmap v13602: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[91136]: from='client.94835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3526081060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[99959]: from='client.74907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[99959]: pgmap v13602: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[99959]: from='client.94835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:28.195 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:27 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3526081060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:53:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:27 smithi146 ceph-mon[103278]: from='client.74907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:27 smithi146 ceph-mon[103278]: pgmap v13602: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:53:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:27 smithi146 ceph-mon[103278]: from='client.94835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:27 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3526081060' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:53:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:29 smithi146 ceph-mon[103278]: pgmap v13603: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:29 smithi078 ceph-mon[91136]: pgmap v13603: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:29 smithi078 ceph-mon[99959]: pgmap v13603: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:53:30.955 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:53:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:53:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:53:32.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:31 smithi146 ceph-mon[103278]: pgmap v13604: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:31 smithi078 ceph-mon[91136]: pgmap v13604: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:31 smithi078 ceph-mon[99959]: pgmap v13604: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:32.084Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:32.084Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:32.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:34.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:33 smithi146 ceph-mon[103278]: pgmap v13605: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:34.006 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:33 smithi078 ceph-mon[91136]: pgmap v13605: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:33 smithi078 ceph-mon[99959]: pgmap v13605: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:34.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:36.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:35 smithi146 ceph-mon[103278]: pgmap v13606: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:53:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:35 smithi078 ceph-mon[91136]: pgmap v13606: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:53:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:35 smithi078 ceph-mon[99959]: pgmap v13606: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:53:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:37 smithi146 ceph-mon[103278]: pgmap v13607: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:37 smithi078 ceph-mon[91136]: pgmap v13607: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:38.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:37 smithi078 ceph-mon[99959]: pgmap v13607: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:39 smithi146 ceph-mon[103278]: pgmap v13608: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:39 smithi078 ceph-mon[91136]: pgmap v13608: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:39 smithi078 ceph-mon[99959]: pgmap v13608: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:40.966 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:53:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:53:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:53:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:41 smithi146 ceph-mon[103278]: pgmap v13609: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T03:53:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:41 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[99959]: pgmap v13609: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T03:53:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[91136]: pgmap v13609: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T03:53:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:41 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:42.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:42.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:42.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:42.085Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:42.288 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:42.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:43 smithi146 ceph-mon[103278]: pgmap v13610: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:44.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:43 smithi078 ceph-mon[91136]: pgmap v13610: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:44.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:43 smithi078 ceph-mon[99959]: pgmap v13610: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:44.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:44.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:45 smithi146 ceph-mon[103278]: pgmap v13611: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:45 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[91136]: pgmap v13611: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[99959]: pgmap v13611: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:45 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:53:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:47 smithi078 ceph-mon[91136]: pgmap v13612: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T03:53:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:47 smithi078 ceph-mon[99959]: pgmap v13612: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T03:53:48.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:47 smithi146 ceph-mon[103278]: pgmap v13612: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T03:53:49.958 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:49 smithi078 ceph-mon[99959]: pgmap v13613: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:49.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:49 smithi078 ceph-mon[91136]: pgmap v13613: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:50.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:49 smithi146 ceph-mon[103278]: pgmap v13613: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T03:53:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:53:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:53:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:53:52.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:51 smithi146 ceph-mon[103278]: pgmap v13614: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:51 smithi078 ceph-mon[91136]: pgmap v13614: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:51 smithi078 ceph-mon[99959]: pgmap v13614: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T03:53:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:52.085Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:52.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:53:52.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:53:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:53:52.086Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:53:54.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:53 smithi146 ceph-mon[103278]: pgmap v13615: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:53 smithi078 ceph-mon[91136]: pgmap v13615: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:53 smithi078 ceph-mon[99959]: pgmap v13615: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:54.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:53:56.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:55 smithi146 ceph-mon[103278]: pgmap v13616: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:53:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:55 smithi078 ceph-mon[91136]: pgmap v13616: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:53:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:55 smithi078 ceph-mon[99959]: pgmap v13616: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T03:53:57.249 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 16s ago 7h - - 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 11s ago 7h - - 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 16s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 11s ago 7h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 16s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 11s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 16s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 16s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 11s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:53:57.774 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 16s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 16s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 11s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 16s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 16s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 16s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 16s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 11s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 11s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 11s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 11s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 11s ago 7h 74.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 16s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 11s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 16s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:53:57.775 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 11s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:53:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:57 smithi078 ceph-mon[99959]: pgmap v13617: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:57 smithi078 ceph-mon[91136]: pgmap v13617: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:53:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:53:58.090 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:53:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:57 smithi146 ceph-mon[103278]: pgmap v13617: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:53:58.357 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:53:58.358 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:53:58.358 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:53:58.358 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:53:58.665 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:53:58.665 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:53:58.665 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:53:58.666 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:53:58.666 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:53:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[91136]: from='client.94847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[91136]: from='client.94853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[91136]: from='client.74943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2718438043' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:53:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:59.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1943003409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:53:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[99959]: from='client.94847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[99959]: from='client.94853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[99959]: from='client.74943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2718438043' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:53:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:59.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:58 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1943003409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:58 smithi146 ceph-mon[103278]: from='client.94847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:58 smithi146 ceph-mon[103278]: from='client.94853 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:58 smithi146 ceph-mon[103278]: from='client.74943 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2718438043' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:53:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:58 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1943003409' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:54:00.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:59 smithi146 ceph-mon[103278]: pgmap v13618: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:53:59 smithi146 ceph-mon[103278]: from='client.94871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:59 smithi078 ceph-mon[91136]: pgmap v13618: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:53:59 smithi078 ceph-mon[91136]: from='client.94871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:59 smithi078 ceph-mon[99959]: pgmap v13618: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:00.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:53:59 smithi078 ceph-mon[99959]: from='client.94871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:54:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:54:00] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:54:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:01 smithi146 ceph-mon[103278]: pgmap v13619: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:01 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[91136]: pgmap v13619: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[99959]: pgmap v13619: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:01 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:02.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:02.086Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:02.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:02.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:02.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:02.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:03 smithi146 ceph-mon[103278]: pgmap v13620: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:03 smithi078 ceph-mon[99959]: pgmap v13620: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:04.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:03 smithi078 ceph-mon[91136]: pgmap v13620: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:06.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:05 smithi146 ceph-mon[103278]: pgmap v13621: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:05 smithi078 ceph-mon[99959]: pgmap v13621: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:05 smithi078 ceph-mon[91136]: pgmap v13621: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:07.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:06 smithi146 ceph-mon[103278]: pgmap v13622: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[99959]: pgmap v13622: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:06 smithi078 ceph-mon[91136]: pgmap v13622: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:09.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:09 smithi078 ceph-mon[91136]: pgmap v13623: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:09.483 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:09 smithi078 ceph-mon[99959]: pgmap v13623: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:09.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:09 smithi146 ceph-mon[103278]: pgmap v13623: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:54:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:54:10] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:54:11.460 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:11 smithi078 ceph-mon[99959]: pgmap v13624: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:11.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:11 smithi078 ceph-mon[91136]: pgmap v13624: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:11.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:11 smithi146 ceph-mon[103278]: pgmap v13624: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:12.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:12.087Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:12.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:13 smithi146 ceph-mon[103278]: pgmap v13625: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:13.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:13 smithi078 ceph-mon[91136]: pgmap v13625: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:13.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:13 smithi078 ceph-mon[99959]: pgmap v13625: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:14.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:14.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:14.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:15.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:15 smithi146 ceph-mon[103278]: pgmap v13626: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:15.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:15 smithi078 ceph-mon[91136]: pgmap v13626: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:15.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:15 smithi078 ceph-mon[99959]: pgmap v13626: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:17.451 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:17 smithi146 ceph-mon[103278]: pgmap v13627: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:17 smithi078 ceph-mon[91136]: pgmap v13627: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:17 smithi078 ceph-mon[99959]: pgmap v13627: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:19.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:19 smithi146 ceph-mon[103278]: pgmap v13628: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:19.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:19 smithi078 ceph-mon[91136]: pgmap v13628: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:19.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:19 smithi078 ceph-mon[99959]: pgmap v13628: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:20.971 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:54:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:54:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:54:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:21 smithi146 ceph-mon[103278]: pgmap v13629: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:21 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[91136]: pgmap v13629: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[99959]: pgmap v13629: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:21.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:21 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:22.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:22.087Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:22.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:22.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:22.498 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:22.088Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:23.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:23 smithi146 ceph-mon[103278]: pgmap v13630: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:23 smithi078 ceph-mon[91136]: pgmap v13630: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:23.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:23 smithi078 ceph-mon[99959]: pgmap v13630: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:23.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:25.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:25 smithi146 ceph-mon[103278]: pgmap v13631: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:25 smithi078 ceph-mon[91136]: pgmap v13631: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:25 smithi078 ceph-mon[99959]: pgmap v13631: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:26.111 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Sep 21 03:54:25 smithi146 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-grafana-a[46870]: logger=cleanup t=2024-09-21T03:54:25.84962806Z level=info msg="Completed cleanup jobs" duration=13.381743ms 2024-09-21T03:54:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:27 smithi146 ceph-mon[103278]: pgmap v13632: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[91136]: pgmap v13632: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:27.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:27 smithi078 ceph-mon[99959]: pgmap v13632: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:28.956 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:54:29.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:29 smithi078 ceph-mon[99959]: pgmap v13633: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:29.504 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:29.505 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:29 smithi078 ceph-mon[99959]: from='client.94883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:29.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:29 smithi078 ceph-mon[91136]: pgmap v13633: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:29.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:29.505 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:29 smithi078 ceph-mon[91136]: from='client.94883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:29.505 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:54:29.505 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 7h - - 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 3s ago 7h - - 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 8s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 3s ago 7h 91.2M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 8s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 3s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 8s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 8s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 3s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 8s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 8s ago 7h 15.4M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 3s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 8s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 8s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:54:29.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 3s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 3s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 3s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 3s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 3s ago 7h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 8s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 3s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 8s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:54:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 3s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:54:29.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:29 smithi146 ceph-mon[103278]: pgmap v13633: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:29 smithi146 ceph-mon[103278]: from='client.94883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:29.819 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:54:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:54:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:54:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:54:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:54:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:54:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:54:29.821 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:54:30.090 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:54:30.090 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:54:30.090 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:54:30.091 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:54:30.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-mon[91136]: from='client.94889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:30.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-mon[91136]: from='client.74979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:30.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/168974109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:54:30.413 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-mon[99959]: from='client.94889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:30.413 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-mon[99959]: from='client.74979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:30.413 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/168974109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:54:30.414 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:54:30.414 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:54:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:54:30.414 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:54:30.414 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:54:30.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:30 smithi146 ceph-mon[103278]: from='client.94889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:30 smithi146 ceph-mon[103278]: from='client.74979 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:30.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:30 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/168974109' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:54:30.735 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:54:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:54:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:54:31.405 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:31 smithi078 ceph-mon[91136]: from='client.94907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:31.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:31 smithi078 ceph-mon[91136]: pgmap v13634: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:31.406 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:31 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2485448173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:54:31.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:31 smithi078 ceph-mon[99959]: from='client.94907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:31.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:31 smithi078 ceph-mon[99959]: pgmap v13634: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:31.406 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:31 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2485448173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:54:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:31 smithi146 ceph-mon[103278]: from='client.94907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:54:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:31 smithi146 ceph-mon[103278]: pgmap v13634: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:31.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:31 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2485448173' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:54:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:32.088Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:32.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:32.089Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:33 smithi146 ceph-mon[103278]: pgmap v13635: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:33.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:33 smithi078 ceph-mon[91136]: pgmap v13635: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:33.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:33 smithi078 ceph-mon[99959]: pgmap v13635: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:33.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:35.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:35 smithi146 ceph-mon[103278]: pgmap v13636: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:35.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:35 smithi078 ceph-mon[91136]: pgmap v13636: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:35.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:35 smithi078 ceph-mon[99959]: pgmap v13636: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:37.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:37 smithi146 ceph-mon[103278]: pgmap v13637: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:37.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:37 smithi078 ceph-mon[99959]: pgmap v13637: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:37.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:37 smithi078 ceph-mon[91136]: pgmap v13637: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:39.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:39 smithi146 ceph-mon[103278]: pgmap v13638: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:39.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:39 smithi078 ceph-mon[91136]: pgmap v13638: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:39.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:39 smithi078 ceph-mon[99959]: pgmap v13638: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:54:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:54:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:54:41.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:41 smithi146 ceph-mon[103278]: pgmap v13639: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:41.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:41 smithi078 ceph-mon[99959]: pgmap v13639: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:41.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:41 smithi078 ceph-mon[91136]: pgmap v13639: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:42.343 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:42.090Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:42.343 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:42.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:42.343 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:42.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:42.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.719 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.719 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:42.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:43 smithi146 ceph-mon[103278]: pgmap v13640: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:43 smithi078 ceph-mon[91136]: pgmap v13640: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:43 smithi078 ceph-mon[99959]: pgmap v13640: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:45 smithi146 ceph-mon[103278]: pgmap v13641: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:45 smithi078 ceph-mon[91136]: pgmap v13641: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:45 smithi078 ceph-mon[99959]: pgmap v13641: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:47 smithi146 ceph-mon[103278]: pgmap v13642: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[91136]: pgmap v13642: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[99959]: pgmap v13642: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:47.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:54:49.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:49 smithi146 ceph-mon[103278]: pgmap v13643: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:49.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:49 smithi078 ceph-mon[91136]: pgmap v13643: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:49.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:49 smithi078 ceph-mon[99959]: pgmap v13643: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:50.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:54:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:54:50] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:54:51.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:51 smithi146 ceph-mon[103278]: pgmap v13644: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:51.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:51 smithi078 ceph-mon[99959]: pgmap v13644: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:51.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:51 smithi078 ceph-mon[91136]: pgmap v13644: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:52.463 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:52.090Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:52.463 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:52.090Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:54:52.464 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:54:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:54:52.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:54:53.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:53 smithi146 ceph-mon[103278]: pgmap v13645: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:53.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:53 smithi078 ceph-mon[91136]: pgmap v13645: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:53.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:53 smithi078 ceph-mon[99959]: pgmap v13645: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:53.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:54:55.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:55 smithi146 ceph-mon[103278]: pgmap v13646: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:55.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:55 smithi078 ceph-mon[91136]: pgmap v13646: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:55.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:55 smithi078 ceph-mon[99959]: pgmap v13646: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:57.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:57 smithi146 ceph-mon[103278]: pgmap v13647: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:57.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:57 smithi078 ceph-mon[91136]: pgmap v13647: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:57.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:57 smithi078 ceph-mon[99959]: pgmap v13647: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:54:59.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:59 smithi146 ceph-mon[103278]: pgmap v13648: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:59.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:54:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:59 smithi078 ceph-mon[91136]: pgmap v13648: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:54:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:54:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:59 smithi078 ceph-mon[99959]: pgmap v13648: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:54:59.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:54:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:00.709 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:55:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:55:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:55:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:55:01.234 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:55:01.234 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 19s ago 7h - - 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 7h - - 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 19s ago 7h 29.7M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 15s ago 7h 89.0M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 19s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 15s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 19s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 19s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 15s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 19s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 19s ago 7h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 15s ago 7h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 19s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 19s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 19s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:55:01.235 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 19s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 15s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 15s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 15s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 15s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 15s ago 7h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 19s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 15s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 19s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:55:01.236 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:55:01.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:01 smithi146 ceph-mon[103278]: pgmap v13649: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:01.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:01 smithi146 ceph-mon[103278]: from='client.75003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:01 smithi078 ceph-mon[91136]: pgmap v13649: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:01 smithi078 ceph-mon[91136]: from='client.75003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:01 smithi078 ceph-mon[99959]: pgmap v13649: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:01.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:01 smithi078 ceph-mon[99959]: from='client.75003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:55:01.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:55:01.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:01.558 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:55:01.558 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:55:01.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:55:01.558 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:55:01.828 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:55:01.829 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:55:02.139 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:55:02.139 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:55:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:55:02.140 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:55:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:55:02.464 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:02.090Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:02.464 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:02.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:02.464 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:02.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:02.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='client.94925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.724 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='client.94931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2973803522' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='client.94943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2234576505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='client.94925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='client.94931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2973803522' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:55:02.725 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='client.94943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.726 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:02 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2234576505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:55:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='client.94925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='client.94931 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:02.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2973803522' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:55:02.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='client.94943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:02.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:02 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2234576505' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:55:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:03 smithi146 ceph-mon[103278]: pgmap v13650: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:03 smithi078 ceph-mon[91136]: pgmap v13650: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:03.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:03 smithi078 ceph-mon[99959]: pgmap v13650: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:03.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:05.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:05 smithi146 ceph-mon[103278]: pgmap v13651: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:05 smithi078 ceph-mon[99959]: pgmap v13651: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:05 smithi078 ceph-mon[91136]: pgmap v13651: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:07.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:07 smithi146 ceph-mon[103278]: pgmap v13652: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[99959]: pgmap v13652: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[91136]: pgmap v13652: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:09 smithi146 ceph-mon[103278]: pgmap v13653: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:09 smithi078 ceph-mon[99959]: pgmap v13653: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:09 smithi078 ceph-mon[91136]: pgmap v13653: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:10.847 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:55:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:55:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:55:11.687 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:11 smithi078 ceph-mon[91136]: pgmap v13654: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:11.688 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:11 smithi078 ceph-mon[99959]: pgmap v13654: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:11 smithi146 ceph-mon[103278]: pgmap v13654: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:12.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:12.091Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:12.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:13.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:13 smithi078 ceph-mon[99959]: pgmap v13655: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:13.538 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:13.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:13 smithi078 ceph-mon[91136]: pgmap v13655: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:13.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:13 smithi146 ceph-mon[103278]: pgmap v13655: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:15 smithi146 ceph-mon[103278]: pgmap v13656: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:15 smithi078 ceph-mon[91136]: pgmap v13656: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:15 smithi078 ceph-mon[99959]: pgmap v13656: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:17.676 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:17 smithi146 ceph-mon[103278]: pgmap v13657: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:17 smithi078 ceph-mon[91136]: pgmap v13657: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:17 smithi078 ceph-mon[99959]: pgmap v13657: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:19.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:19 smithi146 ceph-mon[103278]: pgmap v13658: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:19 smithi078 ceph-mon[91136]: pgmap v13658: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:19 smithi078 ceph-mon[99959]: pgmap v13658: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:55:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:55:20] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:55:21.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:21 smithi146 ceph-mon[103278]: pgmap v13659: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:21 smithi078 ceph-mon[99959]: pgmap v13659: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:21 smithi078 ceph-mon[91136]: pgmap v13659: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:22.091Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:22.092Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:22.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:23.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:23.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:23 smithi146 ceph-mon[103278]: pgmap v13660: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:23.918 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:24.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:23 smithi078 ceph-mon[91136]: pgmap v13660: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:24.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:24.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:23 smithi078 ceph-mon[99959]: pgmap v13660: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:24.002 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:26.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:25 smithi146 ceph-mon[103278]: pgmap v13661: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:25 smithi078 ceph-mon[91136]: pgmap v13661: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:25 smithi078 ceph-mon[99959]: pgmap v13661: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:27 smithi146 ceph-mon[103278]: pgmap v13662: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[99959]: pgmap v13662: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[91136]: pgmap v13662: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:29 smithi146 ceph-mon[103278]: pgmap v13663: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:29.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:29 smithi078 ceph-mon[91136]: pgmap v13663: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:29.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:29.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:29 smithi078 ceph-mon[99959]: pgmap v13663: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:29.788 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:55:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:55:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:55:31.640 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:31 smithi078 ceph-mon[99959]: pgmap v13664: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:31.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:31 smithi078 ceph-mon[91136]: pgmap v13664: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:31 smithi146 ceph-mon[103278]: pgmap v13664: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:32.353 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:32.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:32.353 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:32.093Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:32.353 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:32.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:32.437 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:55:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:55:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 11s ago 7h - - 2024-09-21T03:55:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 6s ago 7h - - 2024-09-21T03:55:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 11s ago 7h 29.9M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:55:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 6s ago 7h 91.4M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:55:32.983 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 11s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 6s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 11s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 11s ago 7h 567M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 6s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 11s ago 7h 382M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 11s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 6s ago 7h 14.8M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 11s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 11s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 11s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 11s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 6s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 6s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 6s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:55:32.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 6s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:55:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 6s ago 7h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:55:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 11s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:55:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 6s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:55:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 11s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:55:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 6s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:55:33.312 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:55:33.590 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:55:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:55:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:55:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:55:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:55:33.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:55:33.591 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:55:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[91136]: pgmap v13665: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[91136]: from='client.75039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:33.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[91136]: from='client.94961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/842892951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[99959]: pgmap v13665: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[99959]: from='client.75039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[99959]: from='client.94961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:33.592 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:33 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/842892951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:55:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:33 smithi146 ceph-mon[103278]: pgmap v13665: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:33 smithi146 ceph-mon[103278]: from='client.75039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:33 smithi146 ceph-mon[103278]: from='client.94961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:33 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/842892951' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:55:33.914 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:55:33.915 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:55:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:55:33.915 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:55:33.915 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:55:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:34 smithi146 ceph-mon[103278]: from='client.94967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:34 smithi146 ceph-mon[103278]: from='client.75063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:34 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3657113921' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:55:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:34 smithi078 ceph-mon[99959]: from='client.94967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:34 smithi078 ceph-mon[99959]: from='client.75063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:34.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:34 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3657113921' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:55:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:34 smithi078 ceph-mon[91136]: from='client.94967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:34 smithi078 ceph-mon[91136]: from='client.75063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:55:34.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:34 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3657113921' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:55:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:35 smithi146 ceph-mon[103278]: pgmap v13666: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:35 smithi078 ceph-mon[99959]: pgmap v13666: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:35 smithi078 ceph-mon[91136]: pgmap v13666: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:37 smithi146 ceph-mon[103278]: pgmap v13667: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:37 smithi078 ceph-mon[99959]: pgmap v13667: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:37 smithi078 ceph-mon[91136]: pgmap v13667: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:39.687 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:39 smithi146 ceph-mon[103278]: pgmap v13668: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:39 smithi078 ceph-mon[99959]: pgmap v13668: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:39 smithi078 ceph-mon[91136]: pgmap v13668: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:55:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:55:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:55:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:41 smithi078 ceph-mon[91136]: pgmap v13669: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:41 smithi078 ceph-mon[99959]: pgmap v13669: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:42.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:41 smithi146 ceph-mon[103278]: pgmap v13669: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:42.093Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:42.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:42.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:42 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:43.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:42 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:43 smithi146 ceph-mon[103278]: pgmap v13670: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:44.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:43 smithi078 ceph-mon[99959]: pgmap v13670: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:44.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:43 smithi078 ceph-mon[91136]: pgmap v13670: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:44.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:55:46.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:45 smithi146 ceph-mon[103278]: pgmap v13671: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:45 smithi078 ceph-mon[91136]: pgmap v13671: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:45 smithi078 ceph-mon[99959]: pgmap v13671: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:47 smithi146 ceph-mon[103278]: pgmap v13672: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[91136]: pgmap v13672: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[99959]: pgmap v13672: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:48.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:55:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:49 smithi146 ceph-mon[103278]: pgmap v13673: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:49 smithi078 ceph-mon[91136]: pgmap v13673: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:49 smithi078 ceph-mon[99959]: pgmap v13673: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:55:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:55:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:55:51.916 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:51 smithi078 ceph-mon[91136]: pgmap v13674: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:51.916 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:51 smithi078 ceph-mon[99959]: pgmap v13674: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:51 smithi146 ceph-mon[103278]: pgmap v13674: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:52.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:52.094Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:52.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:52.094Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:55:52.215 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:55:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:55:52.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:55:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:53 smithi146 ceph-mon[103278]: pgmap v13675: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:54.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:53 smithi078 ceph-mon[99959]: pgmap v13675: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:54.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:53 smithi078 ceph-mon[91136]: pgmap v13675: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:54.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:55:56.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:55 smithi146 ceph-mon[103278]: pgmap v13676: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:55 smithi078 ceph-mon[91136]: pgmap v13676: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:55 smithi078 ceph-mon[99959]: pgmap v13676: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:58.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:57 smithi146 ceph-mon[103278]: pgmap v13677: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:58.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:57 smithi078 ceph-mon[91136]: pgmap v13677: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:58.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:57 smithi078 ceph-mon[99959]: pgmap v13677: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:55:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:59 smithi146 ceph-mon[103278]: pgmap v13678: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:55:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:55:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:59 smithi078 ceph-mon[91136]: pgmap v13678: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:55:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:59 smithi078 ceph-mon[99959]: pgmap v13678: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:00.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:55:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:56:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:56:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:56:02.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:01 smithi146 ceph-mon[103278]: pgmap v13679: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:01 smithi078 ceph-mon[91136]: pgmap v13679: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:01 smithi078 ceph-mon[99959]: pgmap v13679: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:02.095Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:02.095Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:02.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:56:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:02 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:03.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:02 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:04.211 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:03 smithi078 ceph-mon[99959]: pgmap v13680: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:04.211 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:04.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:03 smithi078 ceph-mon[91136]: pgmap v13680: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:04.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:04.212 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:03 smithi146 ceph-mon[103278]: pgmap v13680: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 2s ago 7h - - 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 7h - - 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 2s ago 7h 30.0M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 18s ago 7h 91.5M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 2s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:56:04.738 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 18s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 2s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 2s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 18s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 2s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 2s ago 7h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 18s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 2s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 2s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 2s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 2s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 18s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 18s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 18s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:56:04.739 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 18s ago 7h 74.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:56:04.740 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 2s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:56:04.740 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 18s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:56:04.740 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 2s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:56:04.740 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:56:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:56:05.056 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:56:05.325 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:56:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:56:05.325 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:56:05.326 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:56:05.634 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:56:05.634 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:56:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:56:05.634 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:56:05.634 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:56:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[91136]: from='client.75075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[91136]: pgmap v13681: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:06.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[91136]: from='client.75081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[91136]: from='client.95003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1710496948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:56:06.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/474368524' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:56:06.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[99959]: from='client.75075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[99959]: pgmap v13681: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:06.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[99959]: from='client.75081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[99959]: from='client.95003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1710496948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:56:06.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:05 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/474368524' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:56:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:05 smithi146 ceph-mon[103278]: from='client.75075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:05 smithi146 ceph-mon[103278]: pgmap v13681: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:05 smithi146 ceph-mon[103278]: from='client.75081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:05 smithi146 ceph-mon[103278]: from='client.95003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1710496948' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:56:06.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:05 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/474368524' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:56:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:06 smithi146 ceph-mon[103278]: from='client.95015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:06 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[91136]: from='client.95015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[99959]: from='client.95015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:07.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:06 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:07 smithi146 ceph-mon[103278]: pgmap v13682: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:07 smithi078 ceph-mon[91136]: pgmap v13682: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:07 smithi078 ceph-mon[99959]: pgmap v13682: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:10.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:09 smithi146 ceph-mon[103278]: pgmap v13683: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:09 smithi078 ceph-mon[91136]: pgmap v13683: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:09 smithi078 ceph-mon[99959]: pgmap v13683: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:56:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:56:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:56:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:11 smithi146 ceph-mon[103278]: pgmap v13684: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:11 smithi078 ceph-mon[91136]: pgmap v13684: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:11 smithi078 ceph-mon[99959]: pgmap v13684: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:12.095Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:12.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:12.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:56:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:13 smithi078 ceph-mon[91136]: pgmap v13685: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:14.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:13 smithi078 ceph-mon[99959]: pgmap v13685: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:14.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:14.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:13 smithi146 ceph-mon[103278]: pgmap v13685: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:15 smithi146 ceph-mon[103278]: pgmap v13686: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:15 smithi078 ceph-mon[91136]: pgmap v13686: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:15 smithi078 ceph-mon[99959]: pgmap v13686: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:18.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:17 smithi146 ceph-mon[103278]: pgmap v13687: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:17 smithi078 ceph-mon[91136]: pgmap v13687: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:17 smithi078 ceph-mon[99959]: pgmap v13687: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:20.228 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:19 smithi146 ceph-mon[103278]: pgmap v13688: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:20.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:19 smithi078 ceph-mon[91136]: pgmap v13688: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:20.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:19 smithi078 ceph-mon[99959]: pgmap v13688: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:56:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:56:20] "GET /metrics HTTP/1.1" 200 39010 "" "Prometheus/2.43.0" 2024-09-21T03:56:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:21 smithi146 ceph-mon[103278]: pgmap v13689: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:22.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:21 smithi078 ceph-mon[91136]: pgmap v13689: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:22.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:21 smithi078 ceph-mon[99959]: pgmap v13689: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:22.096Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:22.096Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:22.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:22.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:56:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:22 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:22 smithi146 ceph-mon[103278]: pgmap v13690: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:23.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:22 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[91136]: pgmap v13690: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[99959]: pgmap v13690: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:23.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:22 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:25.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:25 smithi078 ceph-mon[91136]: pgmap v13691: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:25.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:25 smithi078 ceph-mon[99959]: pgmap v13691: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:25.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:25 smithi146 ceph-mon[103278]: pgmap v13691: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:27 smithi146 ceph-mon[103278]: pgmap v13692: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[91136]: pgmap v13692: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[99959]: pgmap v13692: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:28.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:29.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:30.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:29 smithi146 ceph-mon[103278]: pgmap v13693: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:30.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:29 smithi078 ceph-mon[91136]: pgmap v13693: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:30.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:29 smithi078 ceph-mon[99959]: pgmap v13693: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:56:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:56:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:56:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:31 smithi146 ceph-mon[103278]: pgmap v13694: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:32.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:31 smithi078 ceph-mon[91136]: pgmap v13694: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:32.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:31 smithi078 ceph-mon[99959]: pgmap v13694: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:32.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:32.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:32.097Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:32.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:32.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:56:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:33 smithi146 ceph-mon[103278]: pgmap v13695: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:33 smithi078 ceph-mon[91136]: pgmap v13695: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:34.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:33 smithi078 ceph-mon[99959]: pgmap v13695: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:34.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:56:36.100 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:35 smithi078 ceph-mon[99959]: pgmap v13696: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:36.101 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:35 smithi078 ceph-mon[91136]: pgmap v13696: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:36.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:35 smithi146 ceph-mon[103278]: pgmap v13696: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 14s ago 7h - - 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 9s ago 7h - - 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 14s ago 7h 30.1M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 9s ago 7h 90.6M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 14s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 9s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 14s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 14s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:56:36.457 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 9s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 14s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 14s ago 7h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 9s ago 7h 15.5M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 14s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 14s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 14s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 14s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 9s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 9s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 9s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 9s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 9s ago 7h 75.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 14s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:56:36.458 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 9s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 14s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:56:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 9s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:36.773 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:56:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:56:37.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:36 smithi078 ceph-mon[99959]: from='client.95027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:37.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:36 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1101146033' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:56:37.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:36 smithi078 ceph-mon[91136]: from='client.95027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:37.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:36 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1101146033' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:56:37.044 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:56:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:56:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:56:37.045 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:56:37.045 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:56:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:36 smithi146 ceph-mon[103278]: from='client.95027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:37.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:36 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1101146033' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:56:37.358 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:56:37.358 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:56:37.358 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:56:37.358 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:56:37.358 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:56:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:37 smithi146 ceph-mon[103278]: from='client.95033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:37 smithi146 ceph-mon[103278]: pgmap v13697: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:37 smithi146 ceph-mon[103278]: from='client.95039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:37 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1994589307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:56:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[91136]: from='client.95033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:38.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[91136]: pgmap v13697: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:38.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[91136]: from='client.95039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:38.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1994589307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:56:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[99959]: from='client.95033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[99959]: pgmap v13697: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[99959]: from='client.95039 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:38.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:37 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1994589307' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:56:39.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:38 smithi146 ceph-mon[103278]: from='client.75135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:39.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:38 smithi078 ceph-mon[91136]: from='client.75135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:39.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:38 smithi078 ceph-mon[99959]: from='client.75135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:56:40.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:39 smithi146 ceph-mon[103278]: pgmap v13698: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:40.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:39 smithi078 ceph-mon[91136]: pgmap v13698: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:40.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:39 smithi078 ceph-mon[99959]: pgmap v13698: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:56:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:56:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:56:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:41 smithi146 ceph-mon[103278]: pgmap v13699: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:42.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:41 smithi078 ceph-mon[91136]: pgmap v13699: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:42.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:41 smithi078 ceph-mon[99959]: pgmap v13699: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:42.097Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:42.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:42.098Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:42.287 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:42.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:56:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:43 smithi146 ceph-mon[103278]: pgmap v13700: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:43.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[91136]: pgmap v13700: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[99959]: pgmap v13700: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:43.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:44.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:44.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:44.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:45.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:45 smithi146 ceph-mon[103278]: pgmap v13701: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:45.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:45 smithi078 ceph-mon[91136]: pgmap v13701: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:45.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:45 smithi078 ceph-mon[99959]: pgmap v13701: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:47 smithi146 ceph-mon[103278]: pgmap v13702: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[91136]: pgmap v13702: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[99959]: pgmap v13702: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:47.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:47.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:56:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:49 smithi146 ceph-mon[103278]: pgmap v13703: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:49 smithi078 ceph-mon[91136]: pgmap v13703: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:49 smithi078 ceph-mon[99959]: pgmap v13703: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:56:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:56:50] "GET /metrics HTTP/1.1" 200 39011 "" "Prometheus/2.43.0" 2024-09-21T03:56:51.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:51 smithi146 ceph-mon[103278]: pgmap v13704: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:51 smithi078 ceph-mon[91136]: pgmap v13704: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:51 smithi078 ceph-mon[99959]: pgmap v13704: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:52.098Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:52.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:56:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:56:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:56:52.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:56:53.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:53 smithi078 ceph-mon[91136]: pgmap v13705: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:53.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:53.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:53 smithi078 ceph-mon[99959]: pgmap v13705: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:53.722 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:53.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:53 smithi146 ceph-mon[103278]: pgmap v13705: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:56:55.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:55 smithi078 ceph-mon[91136]: pgmap v13706: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:55.687 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:55 smithi078 ceph-mon[99959]: pgmap v13706: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:55.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:55 smithi146 ceph-mon[103278]: pgmap v13706: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:57 smithi146 ceph-mon[103278]: pgmap v13707: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:57 smithi078 ceph-mon[91136]: pgmap v13707: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:57 smithi078 ceph-mon[99959]: pgmap v13707: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:56:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:59 smithi146 ceph-mon[103278]: pgmap v13708: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:56:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:59 smithi078 ceph-mon[99959]: pgmap v13708: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:56:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:56:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:59 smithi078 ceph-mon[91136]: pgmap v13708: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:56:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:56:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:57:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:57:00] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:57:01.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:01 smithi146 ceph-mon[103278]: pgmap v13709: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:01.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:01 smithi078 ceph-mon[99959]: pgmap v13709: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:01.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:01 smithi078 ceph-mon[91136]: pgmap v13709: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:02.099Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:02.099Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:02.100Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:03.516 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:03 smithi146 ceph-mon[103278]: pgmap v13710: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[91136]: pgmap v13710: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[99959]: pgmap v13710: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:03.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:05.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:05 smithi146 ceph-mon[103278]: pgmap v13711: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:05.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:05 smithi078 ceph-mon[91136]: pgmap v13711: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:05.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:05 smithi078 ceph-mon[99959]: pgmap v13711: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:07 smithi146 ceph-mon[103278]: pgmap v13712: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[91136]: pgmap v13712: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:07.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.651 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.651 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[99959]: pgmap v13712: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:07.652 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.652 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.652 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:07.652 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:57:08.177 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 5s ago 7h - - 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 7h - - 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 5s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 1s ago 7h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 5s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 1s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 5s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 5s ago 7h 567M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 1s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 5s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 5s ago 7h 15.2M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:57:08.178 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 1s ago 7h 15.1M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 5s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 5s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 5s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 1s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 1s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 1s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 1s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 1s ago 7h 75.1M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 5s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 1s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 5s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:57:08.179 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:57:08.492 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:57:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:57:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:57:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:57:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:57:08.493 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:57:08.493 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:08 smithi078 ceph-mon[99959]: from='client.75147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:08.494 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:08 smithi078 ceph-mon[99959]: from='client.95069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:08.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:08 smithi078 ceph-mon[91136]: from='client.75147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:08.494 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:08 smithi078 ceph-mon[91136]: from='client.95069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:57:08.762 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:57:08.763 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:57:08.763 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:57:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:08 smithi146 ceph-mon[103278]: from='client.75147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:08.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:08 smithi146 ceph-mon[103278]: from='client.95069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.072 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:57:09.072 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:57:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:57:09.073 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:57:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:57:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[91136]: from='client.95075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[91136]: pgmap v13713: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2761815847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:57:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[91136]: from='client.75171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/384236511' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:57:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[99959]: from='client.95075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[99959]: pgmap v13713: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2761815847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:57:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[99959]: from='client.75171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:09 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/384236511' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:57:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:09 smithi146 ceph-mon[103278]: from='client.95075 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:09 smithi146 ceph-mon[103278]: pgmap v13713: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2761815847' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:57:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:09 smithi146 ceph-mon[103278]: from='client.75171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:09 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/384236511' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:57:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:57:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:57:10] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T03:57:11.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:11 smithi146 ceph-mon[103278]: pgmap v13714: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:11 smithi078 ceph-mon[91136]: pgmap v13714: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:11 smithi078 ceph-mon[99959]: pgmap v13714: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:12.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:12.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:12.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:13 smithi146 ceph-mon[103278]: pgmap v13715: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:13 smithi078 ceph-mon[91136]: pgmap v13715: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:13 smithi078 ceph-mon[99959]: pgmap v13715: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:14.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:15 smithi146 ceph-mon[103278]: pgmap v13716: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:15 smithi078 ceph-mon[91136]: pgmap v13716: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:15 smithi078 ceph-mon[99959]: pgmap v13716: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:17.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:17 smithi078 ceph-mon[91136]: pgmap v13717: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:17.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:17 smithi078 ceph-mon[99959]: pgmap v13717: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:17.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:17 smithi146 ceph-mon[103278]: pgmap v13717: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:19.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:19 smithi146 ceph-mon[103278]: pgmap v13718: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:19 smithi078 ceph-mon[99959]: pgmap v13718: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:19 smithi078 ceph-mon[91136]: pgmap v13718: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:21.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:57:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:57:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:57:21.479 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:21 smithi146 ceph-mon[103278]: pgmap v13719: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:21.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:21 smithi078 ceph-mon[99959]: pgmap v13719: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:21.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:21 smithi078 ceph-mon[91136]: pgmap v13719: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:22.397 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:22.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:22.397 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:22.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:22.397 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:22.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:23 smithi146 ceph-mon[103278]: pgmap v13720: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[91136]: pgmap v13720: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:23.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[99959]: pgmap v13720: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:23.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:23.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:25.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:25 smithi146 ceph-mon[103278]: pgmap v13721: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:25.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:25 smithi078 ceph-mon[91136]: pgmap v13721: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:25.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:25 smithi078 ceph-mon[99959]: pgmap v13721: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:27 smithi146 ceph-mon[103278]: pgmap v13722: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[91136]: pgmap v13722: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[99959]: pgmap v13722: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:27.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:29 smithi146 ceph-mon[103278]: pgmap v13723: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:29 smithi078 ceph-mon[91136]: pgmap v13723: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:29 smithi078 ceph-mon[99959]: pgmap v13723: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:29.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:57:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:57:30] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:57:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:31 smithi146 ceph-mon[103278]: pgmap v13724: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:31 smithi078 ceph-mon[99959]: pgmap v13724: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:31 smithi078 ceph-mon[91136]: pgmap v13724: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:32.100Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:32.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:32.101Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:33 smithi146 ceph-mon[103278]: pgmap v13725: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:33 smithi078 ceph-mon[91136]: pgmap v13725: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:33 smithi078 ceph-mon[99959]: pgmap v13725: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:35.676 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:35 smithi078 ceph-mon[91136]: pgmap v13726: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:35.676 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:35 smithi078 ceph-mon[99959]: pgmap v13726: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:35 smithi146 ceph-mon[103278]: pgmap v13726: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:37.599 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:37 smithi078 ceph-mon[91136]: pgmap v13727: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:37.600 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:37 smithi078 ceph-mon[99959]: pgmap v13727: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:37.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:37 smithi146 ceph-mon[103278]: pgmap v13727: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:39.360 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:57:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:39 smithi146 ceph-mon[103278]: pgmap v13728: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:39 smithi078 ceph-mon[99959]: pgmap v13728: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:39 smithi078 ceph-mon[91136]: pgmap v13728: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 17s ago 7h - - 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 12s ago 7h - - 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 17s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 12s ago 7h 90.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 17s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:57:39.890 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 12s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 17s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 17s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 12s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 17s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 17s ago 7h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 12s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 17s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 17s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 17s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 17s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 12s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 12s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 12s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:57:39.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 12s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:57:39.892 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 12s ago 7h 75.2M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:57:39.892 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 17s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:57:39.892 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 12s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:57:39.892 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 17s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:57:39.892 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 12s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:57:40.196 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:57:40.196 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:57:40.196 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:57:40.197 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:57:40.470 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:57:40.471 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[91136]: from='client.75183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[91136]: from='client.95105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[91136]: from='client.75195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3808069962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[99959]: from='client.75183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[99959]: from='client.95105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[99959]: from='client.75195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.472 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3808069962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:57:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:40 smithi146 ceph-mon[103278]: from='client.75183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:40 smithi146 ceph-mon[103278]: from='client.95105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:40 smithi146 ceph-mon[103278]: from='client.75195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:40.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:40 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3808069962' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:57:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:57:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:57:40.782 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:57:40.782 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:57:40.782 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:57:40.782 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:57:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:57:40] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:57:41.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:41 smithi146 ceph-mon[103278]: pgmap v13729: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:41.562 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:41 smithi146 ceph-mon[103278]: from='client.95123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:41.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:41 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2609695261' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:57:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:41 smithi078 ceph-mon[91136]: pgmap v13729: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:41 smithi078 ceph-mon[91136]: from='client.95123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:41 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2609695261' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:57:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:41 smithi078 ceph-mon[99959]: pgmap v13729: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:41 smithi078 ceph-mon[99959]: from='client.95123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:57:41.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:41 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2609695261' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:57:42.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:42.101Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:42.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:42.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:42.527 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:42.102Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:43 smithi146 ceph-mon[103278]: pgmap v13730: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[99959]: pgmap v13730: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[91136]: pgmap v13730: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:43.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:45.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:57:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:45 smithi078 ceph-mon[99959]: pgmap v13731: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:45 smithi078 ceph-mon[91136]: pgmap v13731: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:45 smithi146 ceph-mon[103278]: pgmap v13731: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:47 smithi146 ceph-mon[103278]: pgmap v13732: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[99959]: pgmap v13732: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[91136]: pgmap v13732: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:47.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:57:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:49 smithi078 ceph-mon[91136]: pgmap v13733: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:49 smithi078 ceph-mon[99959]: pgmap v13733: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:49 smithi146 ceph-mon[103278]: pgmap v13733: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:57:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:57:50] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:57:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:51 smithi078 ceph-mon[99959]: pgmap v13734: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:51 smithi078 ceph-mon[91136]: pgmap v13734: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:51 smithi146 ceph-mon[103278]: pgmap v13734: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:52.102Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:52.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:57:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:57:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:57:52.103Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:57:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:53 smithi078 ceph-mon[91136]: pgmap v13735: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:53.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:53 smithi078 ceph-mon[99959]: pgmap v13735: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:53.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:53.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:53 smithi146 ceph-mon[103278]: pgmap v13735: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:53.819 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:57:55.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:55 smithi146 ceph-mon[103278]: pgmap v13736: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:56.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:55 smithi078 ceph-mon[91136]: pgmap v13736: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:56.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:55 smithi078 ceph-mon[99959]: pgmap v13736: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:57:57.645 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:57 smithi078 ceph-mon[91136]: pgmap v13737: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:57.645 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:57 smithi078 ceph-mon[99959]: pgmap v13737: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:57:58.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:57 smithi146 ceph-mon[103278]: pgmap v13737: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:59 smithi146 ceph-mon[103278]: pgmap v13738: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:00.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:57:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:59 smithi078 ceph-mon[91136]: pgmap v13738: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:00.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:57:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:59 smithi078 ceph-mon[99959]: pgmap v13738: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:00.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:57:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:58:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:58:00] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:58:01.915 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:01 smithi146 ceph-mon[103278]: pgmap v13739: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:02.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:01 smithi078 ceph-mon[91136]: pgmap v13739: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:02.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:01 smithi078 ceph-mon[99959]: pgmap v13739: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:02.103Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:02.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:02.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:02.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:03 smithi146 ceph-mon[103278]: pgmap v13740: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:03.768 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[99959]: pgmap v13740: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:04.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:04.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[91136]: pgmap v13740: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:04.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:06.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:05 smithi146 ceph-mon[103278]: pgmap v13741: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:06.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:05 smithi078 ceph-mon[99959]: pgmap v13741: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:06.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:05 smithi078 ceph-mon[91136]: pgmap v13741: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:07 smithi146 ceph-mon[103278]: pgmap v13742: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:07 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[99959]: pgmap v13742: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[91136]: pgmap v13742: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:08.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:07 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:10.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:09 smithi146 ceph-mon[103278]: pgmap v13743: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:10.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:09 smithi078 ceph-mon[91136]: pgmap v13743: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:10.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:09 smithi078 ceph-mon[99959]: pgmap v13743: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:58:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:58:10] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:58:11.076 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 8s ago 7h - - 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 4s ago 7h - - 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 8s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 4s ago 7h 89.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 8s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 4s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 8s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 8s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 4s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 8s ago 7h 380M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:58:11.605 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 8s ago 7h 15.3M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 4s ago 7h 15.2M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 8s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 8s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 8s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 4s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 4s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 4s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 4s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 4s ago 7h 69.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 8s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 4s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 8s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:58:11.606 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 4s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:58:11.915 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:58:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:58:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:58:11.916 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:58:11.917 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:11 smithi078 ceph-mon[99959]: pgmap v13744: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:11.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:11 smithi078 ceph-mon[91136]: pgmap v13744: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:12.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:11 smithi146 ceph-mon[103278]: pgmap v13744: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:12.182 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:58:12.182 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:58:12.182 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:58:12.182 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:58:12.182 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:58:12.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:58:12.183 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:58:12.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:12.104Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:12.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:12.104Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:12.183 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:12.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:12.488 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:58:12.488 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:58:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:58:12.488 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:58:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[99959]: from='client.95135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[99959]: from='client.75225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[99959]: from='client.75231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3088626587' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:58:12.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1387761220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:58:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[91136]: from='client.95135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[91136]: from='client.75225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[91136]: from='client.75231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3088626587' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:58:12.787 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:12 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1387761220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:58:13.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:12 smithi146 ceph-mon[103278]: from='client.95135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:12 smithi146 ceph-mon[103278]: from='client.75225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:12 smithi146 ceph-mon[103278]: from='client.75231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3088626587' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:58:13.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:12 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1387761220' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:58:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:13 smithi146 ceph-mon[103278]: from='client.95159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:13 smithi146 ceph-mon[103278]: pgmap v13745: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:14.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:14.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:13 smithi078 ceph-mon[91136]: from='client.95159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:14.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:13 smithi078 ceph-mon[91136]: pgmap v13745: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:14.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:13 smithi078 ceph-mon[99959]: from='client.95159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:14.038 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:13 smithi078 ceph-mon[99959]: pgmap v13745: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:14.039 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:15.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:15.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:16.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:15 smithi146 ceph-mon[103278]: pgmap v13746: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:16.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:15 smithi078 ceph-mon[91136]: pgmap v13746: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:16.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:15 smithi078 ceph-mon[99959]: pgmap v13746: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:17.950 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:17 smithi078 ceph-mon[99959]: pgmap v13747: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:17.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:17 smithi078 ceph-mon[91136]: pgmap v13747: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:18.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:17 smithi146 ceph-mon[103278]: pgmap v13747: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:19.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:19 smithi078 ceph-mon[91136]: pgmap v13748: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:19.766 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:19 smithi078 ceph-mon[99959]: pgmap v13748: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:20.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:19 smithi146 ceph-mon[103278]: pgmap v13748: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:58:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:58:20] "GET /metrics HTTP/1.1" 200 39006 "" "Prometheus/2.43.0" 2024-09-21T03:58:22.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:21 smithi146 ceph-mon[103278]: pgmap v13749: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:21 smithi078 ceph-mon[91136]: pgmap v13749: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:21 smithi078 ceph-mon[99959]: pgmap v13749: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:22.104Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:22.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:22.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:23.822 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:23 smithi146 ceph-mon[103278]: pgmap v13750: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:23.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:23.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:23.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:23.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[91136]: pgmap v13750: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:24.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[99959]: pgmap v13750: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:24.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:26.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:25 smithi146 ceph-mon[103278]: pgmap v13751: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:25 smithi078 ceph-mon[91136]: pgmap v13751: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:25 smithi078 ceph-mon[99959]: pgmap v13751: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:27 smithi146 ceph-mon[103278]: pgmap v13752: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:27 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[91136]: pgmap v13752: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[99959]: pgmap v13752: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:27 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:29.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:28 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:29.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:28 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:29.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:28 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:30.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:29 smithi146 ceph-mon[103278]: pgmap v13753: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:30.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:29 smithi078 ceph-mon[91136]: pgmap v13753: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:30.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:29 smithi078 ceph-mon[99959]: pgmap v13753: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:58:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:58:30] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:58:32.106 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:31 smithi078 ceph-mon[91136]: pgmap v13754: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:32.107 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:31 smithi078 ceph-mon[99959]: pgmap v13754: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:32.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:31 smithi146 ceph-mon[103278]: pgmap v13754: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:32.105Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:32.105Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:32.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:33 smithi078 ceph-mon[99959]: pgmap v13755: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:34.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:33 smithi078 ceph-mon[91136]: pgmap v13755: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:34.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:33 smithi146 ceph-mon[103278]: pgmap v13755: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:34.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:36.016 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:35 smithi146 ceph-mon[103278]: pgmap v13756: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:36.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:35 smithi078 ceph-mon[99959]: pgmap v13756: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:36.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:35 smithi078 ceph-mon[91136]: pgmap v13756: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:38.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:37 smithi078 ceph-mon[91136]: pgmap v13757: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:38.191 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:37 smithi078 ceph-mon[99959]: pgmap v13757: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:38.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:37 smithi146 ceph-mon[103278]: pgmap v13757: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:58:40.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:39 smithi078 ceph-mon[91136]: pgmap v13758: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:40.013 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:39 smithi078 ceph-mon[99959]: pgmap v13758: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:40.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:39 smithi146 ceph-mon[103278]: pgmap v13758: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:40.739 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:58:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:58:40] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:58:42.107 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:41 smithi078 ceph-mon[91136]: pgmap v13759: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:42.108 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:41 smithi078 ceph-mon[99959]: pgmap v13759: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:42.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:41 smithi146 ceph-mon[103278]: pgmap v13759: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:42.106Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:42.106Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:42.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:42.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:42.785 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:58:43.325 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:58:43.325 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 0s ago 7h - - 2024-09-21T03:58:43.325 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 15s ago 7h - - 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 0s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 15s ago 7h 89.7M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 0s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 15s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 0s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 0s ago 7h 565M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 15s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 0s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 0s ago 7h 14.9M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 15s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 0s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 0s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 0s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:58:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 0s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 15s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 15s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 15s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 15s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 15s ago 7h 69.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 0s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 15s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 0s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:58:43.327 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 15s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:58:43.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:58:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:58:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:58:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:58:43.636 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:58:43.907 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:58:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:58:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:58:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:58:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:58:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:58:43.908 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:58:44.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: pgmap v13760: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='client.75255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2504533080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:58:44.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:58:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:58:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:58:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:58:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: pgmap v13760: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='client.75255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2504533080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: pgmap v13760: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='client.75255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:44.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:44.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:44.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2504533080' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:58:44.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:45.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:44 smithi146 ceph-mon[103278]: from='client.95177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:44 smithi146 ceph-mon[103278]: from='client.95183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:44 smithi146 ceph-mon[103278]: from='client.95195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.268 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:44 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2241208067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[91136]: from='client.95177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[91136]: from='client.95183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[91136]: from='client.95195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2241208067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[99959]: from='client.95177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[99959]: from='client.95183 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[99959]: from='client.95195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:58:45.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:44 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2241208067' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:58:46.234 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:45 smithi146 ceph-mon[103278]: pgmap v13761: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:46.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:45 smithi078 ceph-mon[91136]: pgmap v13761: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:46.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:45 smithi078 ceph-mon[99959]: pgmap v13761: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:47 smithi146 ceph-mon[103278]: pgmap v13762: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:47 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[91136]: pgmap v13762: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[99959]: pgmap v13762: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:48.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:47 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:58:50.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:49 smithi146 ceph-mon[103278]: pgmap v13763: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:50.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:49 smithi078 ceph-mon[91136]: pgmap v13763: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:50.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:49 smithi078 ceph-mon[99959]: pgmap v13763: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:58:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:58:50] "GET /metrics HTTP/1.1" 200 39004 "" "Prometheus/2.43.0" 2024-09-21T03:58:52.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:51 smithi146 ceph-mon[103278]: pgmap v13764: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:52.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:51 smithi078 ceph-mon[99959]: pgmap v13764: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:52.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:51 smithi078 ceph-mon[91136]: pgmap v13764: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:52.106Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:52.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:58:52.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:58:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:58:52.107Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:58:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:53 smithi146 ceph-mon[103278]: pgmap v13765: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:54.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:53 smithi078 ceph-mon[91136]: pgmap v13765: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:54.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:53 smithi078 ceph-mon[99959]: pgmap v13765: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:54.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:58:56.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:55 smithi146 ceph-mon[103278]: pgmap v13766: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:56.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:55 smithi078 ceph-mon[91136]: pgmap v13766: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:56.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:55 smithi078 ceph-mon[99959]: pgmap v13766: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:58:58.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:57 smithi146 ceph-mon[103278]: pgmap v13767: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:58.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:57 smithi078 ceph-mon[91136]: pgmap v13767: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:58.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:57 smithi078 ceph-mon[99959]: pgmap v13767: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:58:59.122 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:58 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:59.122 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:58 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:58:59.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:58 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:00.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:58:59 smithi146 ceph-mon[103278]: pgmap v13768: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:00.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:58:59 smithi078 ceph-mon[91136]: pgmap v13768: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:00.271 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:58:59 smithi078 ceph-mon[99959]: pgmap v13768: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:01.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:59:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:59:00] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:59:02.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:01 smithi146 ceph-mon[103278]: pgmap v13769: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:59:02.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:01 smithi078 ceph-mon[91136]: pgmap v13769: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:59:02.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:01 smithi078 ceph-mon[99959]: pgmap v13769: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:59:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:02.107Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:02.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:02.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:03 smithi146 ceph-mon[103278]: pgmap v13770: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:59:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:03 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[91136]: pgmap v13770: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:59:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[99959]: pgmap v13770: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-21T03:59:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:04.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:06.063 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:05 smithi146 ceph-mon[103278]: pgmap v13771: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:06.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:05 smithi078 ceph-mon[91136]: pgmap v13771: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:06.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:05 smithi078 ceph-mon[99959]: pgmap v13771: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:08.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:07 smithi146 ceph-mon[103278]: pgmap v13772: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:08.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:07 smithi078 ceph-mon[99959]: pgmap v13772: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:08.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:07 smithi078 ceph-mon[91136]: pgmap v13772: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:08 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:08 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:09.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:08 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:10.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:09 smithi146 ceph-mon[103278]: pgmap v13773: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:10.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:09 smithi078 ceph-mon[91136]: pgmap v13773: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:10.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:09 smithi078 ceph-mon[99959]: pgmap v13773: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:59:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:59:10] "GET /metrics HTTP/1.1" 200 39009 "" "Prometheus/2.43.0" 2024-09-21T03:59:12.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:11 smithi146 ceph-mon[103278]: pgmap v13774: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:12.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:11 smithi078 ceph-mon[99959]: pgmap v13774: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:12.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:11 smithi078 ceph-mon[91136]: pgmap v13774: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:12.108Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:12.108Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:12.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:12.109Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:13 smithi146 ceph-mon[103278]: pgmap v13775: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:13 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:14.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:13 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:13 smithi078 ceph-mon[99959]: pgmap v13775: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:13 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:14.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:13 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:13 smithi078 ceph-mon[91136]: pgmap v13775: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:13 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:14.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:13 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:14.510 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:59:15.037 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:59:15.037 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 12s ago 7h - - 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 7s ago 7h - - 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 12s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 7s ago 7h 91.8M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 12s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 7s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 12s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 12s ago 7h 566M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 7s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 12s ago 7h 379M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 12s ago 7h 15.6M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 7s ago 7h 15.3M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 12s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:59:15.038 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 12s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 12s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 12s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 7s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 7s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 7s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 7s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 7s ago 7h 72.6M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 12s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 7s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 12s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:59:15.039 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 7s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:59:15.348 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:59:15.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:59:15.613 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:59:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:59:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:59:15.613 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:59:15.614 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:59:15.923 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:59:15.923 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:59:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:59:15.923 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:59:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:59:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:15 smithi146 ceph-mon[103278]: pgmap v13776: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:15 smithi146 ceph-mon[103278]: from='client.95207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:15 smithi146 ceph-mon[103278]: from='client.95213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2682497602' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:59:16.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:15 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/316774317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[91136]: pgmap v13776: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[91136]: from='client.95207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[91136]: from='client.95213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2682497602' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/316774317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[99959]: pgmap v13776: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:16.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[99959]: from='client.95207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[99959]: from='client.95213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2682497602' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:59:16.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:15 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/316774317' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:59:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:16 smithi146 ceph-mon[103278]: from='client.75303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:17.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:16 smithi146 ceph-mon[103278]: from='client.75315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:16 smithi078 ceph-mon[91136]: from='client.75303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:17.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:16 smithi078 ceph-mon[91136]: from='client.75315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:16 smithi078 ceph-mon[99959]: from='client.75303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:17.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:16 smithi078 ceph-mon[99959]: from='client.75315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:18.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:17 smithi146 ceph-mon[103278]: pgmap v13777: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:18.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:17 smithi078 ceph-mon[91136]: pgmap v13777: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:18.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:17 smithi078 ceph-mon[99959]: pgmap v13777: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:20.117 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:19 smithi078 ceph-mon[99959]: pgmap v13778: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:20.117 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:19 smithi078 ceph-mon[91136]: pgmap v13778: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:20.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:19 smithi146 ceph-mon[103278]: pgmap v13778: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:20.820 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:59:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:59:20] "GET /metrics HTTP/1.1" 200 39002 "" "Prometheus/2.43.0" 2024-09-21T03:59:22.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:21 smithi078 ceph-mon[91136]: pgmap v13779: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:22.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:21 smithi078 ceph-mon[99959]: pgmap v13779: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:22.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:21 smithi146 ceph-mon[103278]: pgmap v13779: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:22.109Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:22.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:22.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:22.110Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:24.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:23 smithi146 ceph-mon[103278]: pgmap v13780: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:24.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:23 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:24.262 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.263 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:23 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[91136]: pgmap v13780: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[99959]: pgmap v13780: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:24.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:24.287 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:23 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:26.267 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:25 smithi146 ceph-mon[103278]: pgmap v13781: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:25 smithi078 ceph-mon[91136]: pgmap v13781: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:26.286 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:25 smithi078 ceph-mon[99959]: pgmap v13781: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:27.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:27 smithi146 ceph-mon[103278]: pgmap v13782: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:27.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:27 smithi078 ceph-mon[91136]: pgmap v13782: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:27.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:27 smithi078 ceph-mon[99959]: pgmap v13782: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:29 smithi146 ceph-mon[103278]: pgmap v13783: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:29.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[91136]: pgmap v13783: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:29.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:29.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[99959]: pgmap v13783: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:29.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:59:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:59:30] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:59:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:31 smithi146 ceph-mon[103278]: pgmap v13784: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:31 smithi078 ceph-mon[91136]: pgmap v13784: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:31 smithi078 ceph-mon[99959]: pgmap v13784: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:32.110Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:32.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:32.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:33.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:33 smithi078 ceph-mon[99959]: pgmap v13785: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:33.736 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:33 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:33.736 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:33 smithi078 ceph-mon[91136]: pgmap v13785: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:33.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:33 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:33 smithi146 ceph-mon[103278]: pgmap v13785: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:33 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:35 smithi146 ceph-mon[103278]: pgmap v13786: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:35 smithi078 ceph-mon[91136]: pgmap v13786: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:35 smithi078 ceph-mon[99959]: pgmap v13786: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:37 smithi146 ceph-mon[103278]: pgmap v13787: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:37 smithi078 ceph-mon[91136]: pgmap v13787: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:37 smithi078 ceph-mon[99959]: pgmap v13787: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:39 smithi146 ceph-mon[103278]: pgmap v13788: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:39 smithi078 ceph-mon[99959]: pgmap v13788: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:39 smithi078 ceph-mon[91136]: pgmap v13788: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:40.988 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:59:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:59:40] "GET /metrics HTTP/1.1" 200 39008 "" "Prometheus/2.43.0" 2024-09-21T03:59:41.690 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:41 smithi078 ceph-mon[99959]: pgmap v13789: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:41.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:41 smithi078 ceph-mon[91136]: pgmap v13789: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:41 smithi146 ceph-mon[103278]: pgmap v13789: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:42.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:42.111Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:42.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:42.111Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:42.198 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:42.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:43 smithi146 ceph-mon[103278]: pgmap v13790: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:43 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:43 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[91136]: pgmap v13790: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[99959]: pgmap v13790: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:43.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:43.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:43 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:44.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:44.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:45.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:45 smithi146 ceph-mon[103278]: pgmap v13791: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:45.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:45 smithi078 ceph-mon[91136]: pgmap v13791: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:45.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:45 smithi078 ceph-mon[99959]: pgmap v13791: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:46.218 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 3s ago 7h - - 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 18s ago 7h - - 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 3s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 18s ago 7h 89.3M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 3s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 18s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 3s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 3s ago 7h 567M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 18s ago 7h 376M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T03:59:46.748 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 3s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 3s ago 7h 15.8M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 18s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 3s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 3s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 3s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 3s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 18s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 18s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 18s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 18s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 18s ago 7h 72.7M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 3s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 18s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 3s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T03:59:46.749 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 18s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T03:59:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T03:59:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T03:59:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T03:59:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T03:59:47.062 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T03:59:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T03:59:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T03:59:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T03:59:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T03:59:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T03:59:47.333 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T03:59:47.646 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T03:59:47.646 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:59:47.646 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T03:59:47.646 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T03:59:47.646 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T03:59:47.646 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[99959]: from='client.75327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[99959]: pgmap v13792: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[99959]: from='client.95249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[99959]: from='client.75339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/3143494493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[91136]: from='client.75327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[91136]: pgmap v13792: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[91136]: from='client.95249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[91136]: from='client.75339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.647 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:47 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/3143494493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:59:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:47 smithi146 ceph-mon[103278]: from='client.75327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:47 smithi146 ceph-mon[103278]: pgmap v13792: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:47 smithi146 ceph-mon[103278]: from='client.95249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:47 smithi146 ceph-mon[103278]: from='client.75339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:47.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:47 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/3143494493' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T03:59:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:48 smithi146 ceph-mon[103278]: from='client.75351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:48 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2078087225' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:59:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.517 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:48 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[91136]: from='client.75351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2078087225' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:59:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[99959]: from='client.75351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T03:59:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2078087225' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T03:59:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:48.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:48 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T03:59:49.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:49 smithi146 ceph-mon[103278]: pgmap v13793: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:49.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:49 smithi078 ceph-mon[91136]: pgmap v13793: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:49.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:49 smithi078 ceph-mon[99959]: pgmap v13793: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:51.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 03:59:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:03:59:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T03:59:51.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:51 smithi146 ceph-mon[103278]: pgmap v13794: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:51.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:51 smithi078 ceph-mon[99959]: pgmap v13794: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:51.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:51 smithi078 ceph-mon[91136]: pgmap v13794: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:52.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:52.112Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T03:59:52.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 03:59:52 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T03:59:52.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T03:59:53.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:53 smithi078 ceph-mon[91136]: pgmap v13795: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:53.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:53 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:53.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:53 smithi078 ceph-mon[99959]: pgmap v13795: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:53.661 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:53 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:53 smithi146 ceph-mon[103278]: pgmap v13795: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T03:59:53.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:53 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T03:59:55.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:55 smithi146 ceph-mon[103278]: pgmap v13796: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:55.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:55 smithi078 ceph-mon[91136]: pgmap v13796: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:55.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:55 smithi078 ceph-mon[99959]: pgmap v13796: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:57.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:57 smithi146 ceph-mon[103278]: pgmap v13797: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:59:57.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:57 smithi078 ceph-mon[91136]: pgmap v13797: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:59:57.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:57 smithi078 ceph-mon[99959]: pgmap v13797: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T03:59:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:59 smithi146 ceph-mon[103278]: pgmap v13798: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:59.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 03:59:59 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:59 smithi078 ceph-mon[91136]: pgmap v13798: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:59.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 03:59:59 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T03:59:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:59 smithi078 ceph-mon[99959]: pgmap v13798: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T03:59:59.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 03:59:59 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:00.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mon-a[91130]: 2024-09-21T04:00:00.000+0000 7f14942a4640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T04:00:00.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:00 smithi146 ceph-mon[103278]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T04:00:00.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:00 smithi078 ceph-mon[99959]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T04:00:00.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:00 smithi078 ceph-mon[91136]: overall HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T04:00:00.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 04:00:00 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:04:00:00] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T04:00:01.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:01 smithi078 ceph-mon[91136]: pgmap v13799: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T04:00:01.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:01 smithi078 ceph-mon[99959]: pgmap v13799: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T04:00:01.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:01 smithi146 ceph-mon[103278]: pgmap v13799: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T04:00:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:02.112Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:02.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:02.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:02 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:02.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:03 smithi078 ceph-mon[91136]: pgmap v13800: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T04:00:03.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:03 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:03 smithi078 ceph-mon[99959]: pgmap v13800: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T04:00:03.536 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:03 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:03 smithi146 ceph-mon[103278]: pgmap v13800: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T04:00:03.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:03 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:04 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:04.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:04 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:04 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:04.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:04 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:04 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:04.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:04 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:05.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:05 smithi146 ceph-mon[103278]: pgmap v13801: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:05.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:05 smithi078 ceph-mon[91136]: pgmap v13801: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:05.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:05 smithi078 ceph-mon[99959]: pgmap v13801: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:07.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:07 smithi146 ceph-mon[103278]: pgmap v13802: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:07.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:07 smithi078 ceph-mon[91136]: pgmap v13802: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:07.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:07 smithi078 ceph-mon[99959]: pgmap v13802: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:09 smithi146 ceph-mon[103278]: pgmap v13803: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:09 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[91136]: pgmap v13803: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[99959]: pgmap v13803: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:09.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:09.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:09 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:11.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 04:00:10 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:04:00:10] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T04:00:11.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:11 smithi146 ceph-mon[103278]: pgmap v13804: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:11.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:11 smithi078 ceph-mon[91136]: pgmap v13804: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:11.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:11 smithi078 ceph-mon[99959]: pgmap v13804: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:12.113Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:12.113Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:12.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:12 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:12.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:13.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:13 smithi146 ceph-mon[103278]: pgmap v13805: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:13.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:13 smithi078 ceph-mon[91136]: pgmap v13805: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:13.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:13 smithi078 ceph-mon[99959]: pgmap v13805: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:14 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:14.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:14 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:14 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:14.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:14 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:14 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:14.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:14 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:15.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:15 smithi146 ceph-mon[103278]: pgmap v13806: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:15.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:15 smithi078 ceph-mon[91136]: pgmap v13806: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:15.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:15 smithi078 ceph-mon[99959]: pgmap v13806: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:17.703 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:17 smithi146 ceph-mon[103278]: pgmap v13807: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:17.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:17 smithi078 ceph-mon[99959]: pgmap v13807: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:17.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:17 smithi078 ceph-mon[91136]: pgmap v13807: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:17.939 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T04:00:18.470 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 15s ago 7h - - 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 10s ago 7h - - 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 15s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 10s ago 7h 90.1M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 15s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 10s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 15s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 15s ago 7h 567M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 10s ago 7h 377M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 15s ago 7h 381M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 15s ago 7h 15.7M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 10s ago 7h 14.9M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T04:00:18.471 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 15s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 15s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 15s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 15s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 10s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 10s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 10s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 10s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 10s ago 7h 74.0M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 15s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 10s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 15s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T04:00:18.472 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 10s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T04:00:18.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:18 smithi146 ceph-mon[103278]: from='client.75363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T04:00:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T04:00:18.778 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T04:00:18.778 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:18 smithi078 ceph-mon[99959]: from='client.75363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:18.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:18 smithi078 ceph-mon[91136]: from='client.75363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T04:00:19.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T04:00:19.047 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T04:00:19.359 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T04:00:19.360 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T04:00:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T04:00:19.360 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T04:00:19.360 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T04:00:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:19 smithi146 ceph-mon[103278]: from='client.75369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:19 smithi146 ceph-mon[103278]: pgmap v13808: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:19 smithi146 ceph-mon[103278]: from='client.75375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:19 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2949896719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[91136]: from='client.75369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[91136]: pgmap v13808: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[91136]: from='client.75375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2949896719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[99959]: from='client.75369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[99959]: pgmap v13808: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[99959]: from='client.75375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:19.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:19 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2949896719' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T04:00:20.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:20 smithi146 ceph-mon[103278]: from='client.95303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:20.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:20 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2770872207' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T04:00:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:20 smithi078 ceph-mon[91136]: from='client.95303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:20.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:20 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2770872207' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T04:00:20.786 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 04:00:20 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:04:00:20] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T04:00:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:20 smithi078 ceph-mon[99959]: from='client.95303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:20.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:20 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2770872207' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T04:00:21.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:21 smithi146 ceph-mon[103278]: pgmap v13809: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:21.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:21 smithi078 ceph-mon[91136]: pgmap v13809: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:21.769 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:21 smithi078 ceph-mon[99959]: pgmap v13809: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:22.113Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:22.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:22.286 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:22 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:22.114Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:23.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:23 smithi078 ceph-mon[91136]: pgmap v13810: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:23.537 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:23 smithi078 ceph-mon[99959]: pgmap v13810: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:23.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:23 smithi146 ceph-mon[103278]: pgmap v13810: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:24.693 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:24 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:24 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:25.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:24 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:25.917 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:25 smithi146 ceph-mon[103278]: pgmap v13811: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:26.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:25 smithi078 ceph-mon[91136]: pgmap v13811: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:26.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:25 smithi078 ceph-mon[99959]: pgmap v13811: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:27.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:27 smithi146 ceph-mon[103278]: pgmap v13812: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:28.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:27 smithi078 ceph-mon[91136]: pgmap v13812: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:28.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:27 smithi078 ceph-mon[99959]: pgmap v13812: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:29 smithi146 ceph-mon[103278]: pgmap v13813: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:29 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[91136]: pgmap v13813: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:29.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[99959]: pgmap v13813: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:29.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:29.787 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:29 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:31.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 04:00:30 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:04:00:30] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T04:00:31.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:31 smithi146 ceph-mon[103278]: pgmap v13814: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:31.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:31 smithi078 ceph-mon[91136]: pgmap v13814: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:31.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:31 smithi078 ceph-mon[99959]: pgmap v13814: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:32.115Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:32.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:32.536 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:32 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:32.115Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:33.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:33 smithi146 ceph-mon[103278]: pgmap v13815: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:33.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:33 smithi078 ceph-mon[91136]: pgmap v13815: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:33.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:33 smithi078 ceph-mon[99959]: pgmap v13815: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:34.734 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:34 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:34.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:34 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:34.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:34 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:35.766 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:35 smithi146 ceph-mon[103278]: pgmap v13816: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:35.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:35 smithi078 ceph-mon[91136]: pgmap v13816: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:35.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:35 smithi078 ceph-mon[99959]: pgmap v13816: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:37.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:37 smithi146 ceph-mon[103278]: pgmap v13817: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:37.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:37 smithi078 ceph-mon[91136]: pgmap v13817: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:37.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:37 smithi078 ceph-mon[99959]: pgmap v13817: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:39.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:39 smithi146 ceph-mon[103278]: pgmap v13818: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:39.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:39 smithi078 ceph-mon[91136]: pgmap v13818: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:39.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:39 smithi078 ceph-mon[99959]: pgmap v13818: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:41.036 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 04:00:40 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:04:00:40] "GET /metrics HTTP/1.1" 200 39005 "" "Prometheus/2.43.0" 2024-09-21T04:00:41.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:41 smithi146 ceph-mon[103278]: pgmap v13819: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:41.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:41 smithi078 ceph-mon[91136]: pgmap v13819: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:41.786 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:41 smithi078 ceph-mon[99959]: pgmap v13819: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:42.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:42.115Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:42.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:42.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi078.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.78:8443: connect: connection refused" 2024-09-21T04:00:42.483 INFO:journalctl@ceph.alertmanager.a.smithi078.stdout:Sep 21 04:00:42 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-alertmanager-a[56026]: ts=2024-09-21T04:00:42.116Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-09-21T04:00:43.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:43 smithi146 ceph-mon[103278]: pgmap v13820: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:43.783 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:43 smithi078 ceph-mon[91136]: pgmap v13820: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:43.783 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:43 smithi078 ceph-mon[99959]: pgmap v13820: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:44 smithi146 ceph-mon[103278]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.767 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:44 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[91136]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:44.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:44.789 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[99959]: from='client.15519 -' entity='client.iscsi.foo.smithi078.bwbwwj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:44.790 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:44 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:46.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:45 smithi146 ceph-mon[103278]: pgmap v13821: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:46.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:45 smithi078 ceph-mon[91136]: pgmap v13821: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:46.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:45 smithi078 ceph-mon[99959]: pgmap v13821: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:47.856 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:47 smithi146 ceph-mon[103278]: pgmap v13822: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:48.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:47 smithi078 ceph-mon[91136]: pgmap v13822: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:48.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:47 smithi078 ceph-mon[99959]: pgmap v13822: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:49.659 INFO:teuthology.orchestra.run.smithi078.stdout:true 2024-09-21T04:00:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:49 smithi146 ceph-mon[103278]: pgmap v13823: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:49 smithi146 ceph-mon[103278]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[99959]: pgmap v13823: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[99959]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[91136]: pgmap v13823: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:49 smithi078 ceph-mon[91136]: from='mgr.25463 172.21.15.78:0/2911614351' entity='mgr.y' 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi078 smithi078 running 6s ago 7h - - 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:agent.smithi146 smithi146 running 1s ago 7h - - 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:alertmanager.a smithi078 *:9093,9094 running (7h) 6s ago 7h 29.5M - 0.25.0 c8568f914cd2 df977df631df 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:grafana.a smithi146 *:3000 running (7h) 1s ago 7h 87.9M - 9.4.7 954c08fa6188 f0f8f5c8c22f 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:iscsi.foo.smithi078.bwbwwj smithi078 *:5000 running (7h) 6s ago 7h 62.6M - 3.8 4404209dfb71 5603af2a3f10 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi146 *:9283,8765 running (7h) 1s ago 7h 458M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38d4d95ffa4e 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi078 *:9283,8765 running (7h) 6s ago 7h 585M - 19.3.0-5127-ge8baca1e 82f5f45939a4 38852dc0bdd7 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7h) 6s ago 7h 570M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ddc9330d21c8 2024-09-21T04:00:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi146 running (7h) 1s ago 7h 375M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 9edca87d335c 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (7h) 6s ago 7h 385M 2048M 19.3.0-5127-ge8baca1e 82f5f45939a4 ffd6b0f668a7 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.a smithi078 *:9100 running (7h) 6s ago 7h 15.5M - 1.5.0 0da6a335fe13 e38df3435cd8 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:node-exporter.b smithi146 *:9100 running (7h) 1s ago 7h 15.0M - 1.5.0 0da6a335fe13 235fece1738e 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7h) 6s ago 7h 100M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 d07702be1687 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi078 running (7h) 6s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0520846b86c3 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi078 running (7h) 6s ago 7h 88.1M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 ea01eae519aa 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (7h) 6s ago 7h 109M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 b2d0b8b0eb4e 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi146 running (7h) 1s ago 7h 125M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 f18ce9968a7d 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi146 running (7h) 1s ago 7h 96.3M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 0678af9b109f 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi146 running (7h) 1s ago 7h 104M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 fead6e0b8b60 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi146 running (7h) 1s ago 7h 99.4M 4096M 19.3.0-5127-ge8baca1e 82f5f45939a4 515bbfa3c73c 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:prometheus.a smithi146 *:9095 running (7h) 1s ago 7h 74.4M - 2.43.0 a07b618ecd1d 843675fd120f 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi078.fewvbr smithi078 *:8000 running (7h) 6s ago 7h 89.5M - 19.3.0-5127-ge8baca1e 82f5f45939a4 a6d1dade8bf4 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.foo.smithi146.dulerg smithi146 *:8000 running (7h) 1s ago 7h 90.1M - 19.3.0-5127-ge8baca1e 82f5f45939a4 be60cf1ca326 2024-09-21T04:00:50.185 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi078.qphqws smithi078 *:80 running (7h) 6s ago 7h 90.8M - 19.3.0-5127-ge8baca1e 82f5f45939a4 acb7e73b7dfa 2024-09-21T04:00:50.186 INFO:teuthology.orchestra.run.smithi078.stdout:rgw.smpl.smithi146.gfhcht smithi146 *:80 running (7h) 1s ago 7h 90.0M - 19.3.0-5127-ge8baca1e 82f5f45939a4 b864cf2d242c 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": { 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 3 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": { 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 2 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": { 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 8 2024-09-21T04:00:50.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw": { 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 4 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "overall": { 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph version 19.3.0-5127-ge8baca1e (e8baca1eac6c3c9c88bf2ea4493e448bd54936a7) squid (dev)": 17 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-21T04:00:50.494 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T04:00:50.762 INFO:journalctl@ceph.mgr.y.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-f2df6e16-778b-11ef-baf5-efdc52797490-mgr-y[76546]: ::ffff:172.21.15.146 - - [21/Sep/2024:04:00:50] "GET /metrics HTTP/1.1" 200 39007 "" "Prometheus/2.43.0" 2024-09-21T04:00:50.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-mon[99959]: from='client.95315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:50.762 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-mon[99959]: from='client.95321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:50.763 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/1543393965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T04:00:50.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-mon[91136]: from='client.95315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:50.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-mon[91136]: from='client.95321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:50.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:50 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/1543393965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T04:00:50.763 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-21T04:00:50.763 INFO:teuthology.orchestra.run.smithi078.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:dc9dc647e3e3360d1adc5cf221a5bec436cbf643c6188f083c790ea05138208d", 2024-09-21T04:00:50.763 INFO:teuthology.orchestra.run.smithi078.stdout: "in_progress": true, 2024-09-21T04:00:50.763 INFO:teuthology.orchestra.run.smithi078.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T04:00:50.763 INFO:teuthology.orchestra.run.smithi078.stdout: "services_complete": [ 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mon", 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr", 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw", 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "osd" 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "progress": "17/23 daemons upgraded", 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "message": "Currently upgrading rgw daemons", 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout: "is_paused": false 2024-09-21T04:00:50.764 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-21T04:00:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:50 smithi146 ceph-mon[103278]: from='client.95315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:50 smithi146 ceph-mon[103278]: from='client.95321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:51.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:50 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/1543393965' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T04:00:51.072 INFO:teuthology.orchestra.run.smithi078.stdout:HEALTH_ERR Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251); Module 'cephadm' has failed: Module not found 2024-09-21T04:00:51.072 INFO:teuthology.orchestra.run.smithi078.stdout:[WRN] MGR_MODULE_DEPENDENCY: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T04:00:51.072 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'dashboard' has failed dependency: EOL while scanning string literal (rgw_client.py, line 251) 2024-09-21T04:00:51.073 INFO:teuthology.orchestra.run.smithi078.stdout:[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Module not found 2024-09-21T04:00:51.073 INFO:teuthology.orchestra.run.smithi078.stdout: Module 'cephadm' has failed: Module not found 2024-09-21T04:00:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:51 smithi146 ceph-mon[103278]: from='client.75411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:51 smithi146 ceph-mon[103278]: pgmap v13824: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:51 smithi146 ceph-mon[103278]: from='client.75423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:52.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Sep 21 04:00:51 smithi146 ceph-mon[103278]: from='client.? 172.21.15.78:0/2571656272' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T04:00:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[91136]: from='client.75411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[91136]: pgmap v13824: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[91136]: from='client.75423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:52.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[91136]: from='client.? 172.21.15.78:0/2571656272' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T04:00:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[99959]: from='client.75411 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[99959]: pgmap v13824: 161 pgs: 161 active+clean; 585 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T04:00:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[99959]: from='client.75423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T04:00:52.037 INFO:journalctl@ceph.mon.c.smithi078.stdout:Sep 21 04:00:51 smithi078 ceph-mon[99959]: from='client.? 172.21.15.78:0/2571656272' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-21T04:00:52.097 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-09-21T04:00:52.098 DEBUG:teuthology.task.console_log:Killing console logger for smithi078 2024-09-21T04:00:52.098 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2024-09-21T04:00:52.099 DEBUG:teuthology.exit:Finished running handlers